服装加工厂网站建设方案计划书,网站的劣势,如何在网站中做公示信息,h5在线编辑Kubernetes入门教程 --- 使用二进制安装1. Introduction1.1 架构图1.2 关键字介绍1.3 简述2. 使用Kubeadm Install2.1 申请三个虚拟环境2.2 准备安装环境2.3 配置yum源2.4 安装Docker2.4.1 配置docker加速器并修改成k8s驱动2.5 时间同步2.6 安装组件3. 基础知识3.1 Pod3.2 控制… Kubernetes入门教程 --- 使用二进制安装1. Introduction1.1 架构图1.2 关键字介绍1.3 简述2. 使用Kubeadm Install2.1 申请三个虚拟环境2.2 准备安装环境2.3 配置yum源2.4 安装Docker2.4.1 配置docker加速器并修改成k8s驱动2.5 时间同步2.6 安装组件3. 基础知识3.1 Pod3.2 控制器3.3 通讯模式3.4 服务发现5. Awakening创建证书请求文件生成证书创建证书请求文件生成证书生成 kubelet bootstrap kubeconfig 配置文件创建证书请求文件生成证书同意请求查看Node四层负载均衡为两台Master apiserver组件提供负载均衡https://blog.csdn.net/qq_44078641/article/details/120049473 https://blog.csdn.net/qq_41009846/article/details/118612651 1. Introduction
1.1 架构图 架构图 左面的master架构 右边是node架构 1.2 关键字介绍 关键字介绍 kubectl命令行工具 Podkubernetes的最小控制单元容器都是运行在pod中的一个pod中可以有1个或者多个容器 controller负责维护集群的状态比如副本期望数量、故障检测、自动扩展、滚动更新等 api server所有服务访问的唯一入口提供认证、授权、访问控制、API 注册和发现等机制 scheduler负责资源的调度按照预定的调度策略将 Pod 调度到相应的机器上 etcd键值对数据库保存了整个集群的状态官方将它定义成一个可信赖的分布式键值存储服务 kubelet负责维护容器的生命周期同时也负责 Volume 和网络的管理 kube-proxy负责为 Service 提供 cluster 内部的服务发现和负载均衡 Container runtime负责镜像管理以及 Pod 和容器的真正运行 label标签用于对pod进行分类同一类pod会拥有相同的标签 NameSpace命名空间用来隔离pod的运行环境 1.3 简述 kubeadm是官方社区推出的一个用于快速部署kubernetes集群的工具。 1.创建一个 Master 节点kubeadm init 2.将一个 Node 节点加入到当前集群中ubeadm join Master节点的IP和端口 2. 使用Kubeadm Install
2.1 申请三个虚拟环境
序号ip节点类型110.136.217.11master210.136.217.12mode310.136.217.13node
2.2 准备安装环境 1.关闭selinux vim /etc/selinux/config 添加 SELIUNXdisabled 并将其他两行注释掉永久关闭selinux sed -i ‘s/enforcing/disabled/’ /etc/selinux/config2.临时关闭swap内存 swapoff -a3.永久关闭swap内存 vim /etc/fstab #永久关闭swap
sed -ri s/.*swap.*/#/ /etc/fstab设置host在master节点 vim /etc/hosts 调整内核参数 cat /etc/sysctl.d/kubernetes.conf EOF #开启网桥模式可将网桥的流量传递给iptables链 net.bridge.bridge-nf-call-ip6tables1 net.bridge.bridge-nf-call-iptables1 #关闭ipv6协议 net.ipv6.conf.all.disable_ipv61 net.ipv4.ip_forward1 EOF 保存参数 sysctl --system 2.3 配置yum源 配置源文件 vim /etc/yum.repos.d/kubernetes.repo [kubernetes]
nameKubernetes
baseurlhttps://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled1
gpgcheck0
repo_gpgcheck0
gpgkeyhttps://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kube2.4 安装Docker 安装wget yum install -y wget配置docker源 wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo安装docker yum -y install docker-ce-18.06.1.ce-3.el7开机启动docker systemctl enable docker systemctl start docker查看docker版本 docker --version守护进程重启 systemctl daemon-reload systemctl start docker避免每次使用docker都要sudo sudo usermod -aG docker xxxxx 2.4.1 配置docker加速器并修改成k8s驱动 需要重启 systemctl start docker vi /etc/docker/daemon.json
{exec-opts: [native.cgroupdriversystemd],log-driver: json-file,log-opts: {max-size: 100m},storage-driver: overlay2,storage-opts: [overlay2.override_kernel_checktrue]
}2.5 时间同步 安装组件 yum install ntpdate -y时间同步 ntpdate http://time.windows.com 2.6 安装组件 安装 yum install -y kubelet-1.20.0 kubeadm-1.20.0 kubectl-1.20.0开机启动 systemctl enable kubelet systemctl start kubelet kubelet命令补全 echo “source (kubectl completion bash)” ~/.bash_profile source ~/.bash_profile 3. 基础知识
3.1 Pod kubernetes的最小管理单元是pod而不是容器所以只能将容器放在Pod中kubernetes一般也不会直接管理Pod而是通过Pod控制器来管理Pod的。 3.2 控制器
3.3 通讯模式
3.4 服务发现
5. Awakening 在一秒钟内看到本质的人和花半辈子也看不清一件事本质的人自然是不一样的命运。 一、生产环境部署k8s常见的几种方式 说明文章内使用到的yaml可到博客资源内下载 k8s-yaml 1、kubeadm Kubeadm是一个K8s部署工具提供kubeadm init和kubeadm join用于快速部署Kubernetes集群。
1 2、二进制 从github下载发行版的二进制包手动部署每个组件组成Kubernetes集群。
小结
Kubeadm降低部署门槛但屏蔽了很多细节遇到问题很难排查。如果想更容易可控推荐使用二进制包部署Kubernetes集群虽然手动部署
麻烦点期间可以学习很多工作原理也利于后期维护。
1234 3、kubespray Kubespray 是 Kubernetes incubator 中的项目目标是提供 Production Ready Kubernetes 部署方案该项目基础是通过 Ansible Playbook
来定义系统与 Kubernetes 集群部署的任务。
12 二、二进制部署 1、准备环境 服务器要求: 建议最小硬件配置: 2核CPU\2G内存\30G硬盘。服务器最好可以访问外网,会有从网上拉取镜像的需求,如果服务器不能上网,需要提前下载对应镜像导入节点。 软件环境: 软件版本操作系统CentOS7.x_x64(mini)容器引擎Docker Ce19KubernetesKubernetes V1.20 服务器规划: 角色IP组件k8s-master1192.168.242.51kube-apiserver,kube-controller-manager,kube-scheduler,kubelet,kube-proxy,docker,etcd,nginx,keepalivedk8s-master2192.168.242.54kube-apiserver,kube-controller-manager,kube-scheduler,kubelet,kube-proxy,docker,nginx,keepalivedk8s-node1192.168.242.52kubelet,kube-proxy,docker,etcdk8s-node2192.168.242.53kubelet,kube-proxy,docker,etcd负载均衡器(虚拟IP)192.168.242.55 须知: 考虑到有些朋友电脑配置较低,一次性开四台虚拟机电脑跑不动, 所以搭建这套k8s高可用集群分两部分实施,先部署一套单Master架构(三台), 再扩容为多Master架构(4台或6台), 顺便再熟悉下Master扩容流程。
12 单Master架构图 单Master服务器规划: 角色IP组件k8s-master192.168.242.51kube-apiserver,kube-controller-manager,kube-scheduler,etcdk8s-node1192.168.242.52kubelet,kube-proxy,docker,etcdk8s-node2192.168.242.53kubelet,kube-proxy,docker,etcd 2、操作系统初始化配置(所有节点) #关闭系统防火墙
systemctl stop firewalld
systemctl disable firewalld#关闭selinux sed -i ‘s/enforcing/disabled/’ /etc/selinux/config #永久 setenforce 0 # 临时
#关闭swap swapoff -a #临时 sed -ri ‘s/.swap./#/’ /etc/fstab #永久
#根据规划设置主机名 hostnamectl set-hostname k8s-master1 hostnamectl set-hostname k8s-master2 hostnamectl set-hostname k8s-node1 hostnamectl set-hostname k8s-node2
#添加hosts cat /etc/hosts EOF 192.168.242.51 k8s-master1 192.168.242.52 k8s-node1 192.168.242.53 k8s-node2 192.168.242.54 k8s-master2 EOF
#将桥接的IPV4流量传递到iptables的链 cat /etc/sysctl.d/k8s.conf EOF net.bridge.bridge-nf-call-ip6tables 1 net.bridge.bridge-nf-call-iptables 1 EOF sysctl --system #生效
#时间同步 #使用阿里云时间服务器进行临时同步 [rootk8s-node1 ~]# ntpdate ntp.aliyun.com 4 Sep 21:27:49 ntpdate[22399]: adjust time server 203.107.6.88 offset 0.001010 sec
#如需配置NTP服务可参考该文档 https://blog.csdn.net/qq_44078641/article/details/120071838 12345678910111213141516171819202122232425262728293031323334353637383940
3、部署etcd集群
3.1 etcd简介
Etcd 是一个分布式键值存储系统Kubernetes使用Etcd进行数据存储所以先准备一个Etcd数据库为解决Etcd单点故障应采用集群方式部署这里使用3台组建集群可容忍1台机器故障当然你也可以使用5台组建集群可容忍2台机器故障
3.2 服务器规划
节点名称IPetcd-1192.168.242.51etcd-2192.168.242.52etcd-2192.168.242.53说明: 为了节省机器,这里与k8s节点复用,也可以部署在k8s机器之外,只要apiserver能连接到就行。
3.3 cfssl证书生成工具准备
cfssl简介: cfssl是一个开源的证书管理工具使用json文件生成证书相比openssl更方便使用。 找任意一台服务器操作这里用Master1节点。
#创建目录存放cfssl工具
mkdir /software-cfssl#下载相关工具 wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -P /software-cfssl/ wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -P /software-cfssl/ wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -P /software-cfssl/
cd /software-cfssl/ chmod x * cp cfssl_linux-amd64 /usr/local/bin/cfssl cp cfssljson_linux-amd64 /usr/local/bin/cfssljson cp cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo 12345678910111213
说明: 如果下载失败,可以使用文档中附件 cfssl工具包
3.4 自签证书颁发机构(CA)
3.4.1 创建工作目录
mkdir -p ~/TLS/{etcd,k8s}
cd ~/TLS/etcd/
12
3.4.2 生成自签CA配置
cat ca-config.json EOF
{signing: {default: {expiry: 87600h},profiles: {www: {expiry: 87600h,usages: [signing,key encipherment,server auth,client auth]}}}
}
EOFcat ca-csr.json EOF { “CN”: “etcd CA”, “key”: { “algo”: “rsa”, “size”: 2048 }, “names”: [ { “C”: “CN”, “L”: “YuMingYu”, “ST”: “YuMingYu” } ] } EOF 1234567891011121314151617181920212223242526272829303132333435363738
3.4.3 生成自签CA证书
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
1
说明: 当前目录下会生成 ca.pem和ca-key.pem文件
[rootk8s-master1 etcd]# ls .
ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem
12
3.5 使用自签CA签发etcd https证书
3.5.1 创建证书申请文件
cat server-csr.json EOF
{CN: etcd,hosts: [192.168.242.51,192.168.242.52,192.168.242.53,192.168.242.54],key: {algo: rsa,size: 2048},names: [{C: CN,L: YuMingYu,ST: YuMingYu}]
}
EOF
12345678910111213141516171819202122
说明: 上述文件hosts字段中ip为所有etcd节点的集群内部通信ip,一个都不能少,为了方便后期扩容可以多写几个预留的ip。
3.5.2 生成证书
cfssl gencert -caca.pem -ca-keyca-key.pem -configca-config.json -profilewww server-csr.json | cfssljson -bare server
1
说明: 当前目录下会生成 server.pem 和 server-key.pem
[rootk8s-master1 etcd]# ls
ca-config.json ca-csr.json ca.pem server-csr.json server.pem
ca.csr ca-key.pem server.csr server-key.pem
123
3.6 下载etcd二进制文件
下载地址
#下载后上传到服务器任意位置即可
https://github.com/etcd-io/etcd/releases/download/v3.4.9/etcd-v3.4.9-linux-amd64.tar.gz
12
说明: 如果下载有问题,可使用附件中的文件。
3.7 部署etcd集群
以下操作在master1上面操作,为简化操作,待会将master1节点生成的所有文件拷贝到其他节点。
3.7.1 创建工作目录并解压二进制包
mkdir /opt/etcd/{bin,cfg,ssl} -p
tar -xf etcd-v3.4.9-linux-amd64.tar.gz
mv etcd-v3.4.9-linux-amd64/{etcd,etcdctl} /opt/etcd/bin/1234
3.8 创建etcd配置文件
cat /opt/etcd/cfg/etcd.conf EOF
#[Member]
ETCD_NAMEetcd-1
ETCD_DATA_DIR/var/lib/etcd/default.etcd
ETCD_LISTEN_PEER_URLShttps://192.168.242.51:2380
ETCD_LISTEN_CLIENT_URLShttps://192.168.242.51:2379#[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS“https://192.168.242.51:2380” ETCD_ADVERTISE_CLIENT_URLS“https://192.168.242.51:2379” ETCD_INITIAL_CLUSTER“etcd-1https://192.168.242.51:2380,etcd-2https://192.168.242.52:2380,etcd-3https://192.168.242.53:2380” ETCD_INITIAL_CLUSTER_TOKEN“etcd-cluster” ETCD_INITIAL_CLUSTER_STATE“new” EOF 1234567891011121314
配置说明:
ETCD_NAME 节点名称,集群中唯一ETCD_DATA_DIR数据目录ETCD_LISTEN_PEER_URLS集群通讯监听地址ETCD_LISTEN_CLIENT_URLS客户端访问监听地址ETCD_INITIAL_CLUSTER集群节点地址ETCD_INITIALCLUSTER_TOKEN集群TokenETCD_INITIALCLUSTER_STATE加入集群的状态new是新集群,existing表示加入已有集群
3.9 systemd管理etcd
cat /usr/lib/systemd/system/etcd.service EOF
[Unit]
DescriptionEtcd Server
Afternetwork.target
Afternetwork-online.target
Wantsnetwork-online.target[Service] Typenotify EnvironmentFile/opt/etcd/cfg/etcd.conf ExecStart/opt/etcd/bin/etcd –cert-file/opt/etcd/ssl/server.pem –key-file/opt/etcd/ssl/server-key.pem –peer-cert-file/opt/etcd/ssl/server.pem –peer-key-file/opt/etcd/ssl/server-key.pem –trusted-ca-file/opt/etcd/ssl/ca.pem –peer-trusted-ca-file/opt/etcd/ssl/ca.pem –loggerzap Restarton-failure LimitNOFILE65536
[Install] WantedBymulti-user.target EOF 123456789101112131415161718192021222324
3.10 将master1节点所有生成的文件拷贝到节点2和节点3
for i in {2..3}
do
scp -r /opt/etcd/ root192.168.242.5$i:/opt/
scp /usr/lib/systemd/system/etcd.service root192.168.242.5$i:/usr/lib/systemd/system/
done
12345
3.11 修改节点2节点3 ,etcd.conf配置文件中的节点名称和当前服务器IP:
#[Member]
ETCD_NAMEetcd-1 #节点2修改为: etcd-2 节点3修改为: etcd-3
ETCD_DATA_DIR/var/lib/etcd/default.etcd
ETCD_LISTEN_PEER_URLShttps://192.168.242.51:2380 #修改为对应节点IP
ETCD_LISTEN_CLIENT_URLShttps://192.168.242.51:2379 #修改为对应节点IP#[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS“https://192.168.242.51:2380” #修改为对应节点IP ETCD_ADVERTISE_CLIENT_URLS“https://192.168.242.51:2379” #修改为对应节点IP ETCD_INITIAL_CLUSTER“etcd-1https://192.168.242.51:2380,etcd-2https://192.168.242.52:2380,etcd-3https://192.168.242.53:2380” ETCD_INITIAL_CLUSTER_TOKEN“etcd-cluster” ETCD_INITIAL_CLUSTER_STATE“new” 123456789101112
3.12 启动etcd并设置开机自启
说明: etcd须多个节点同时启动,不然执行systemctl start etcd会一直卡在前台,连接其他节点,建议通过批量管理工具,或者脚本同时启动etcd。
systemctl daemon-reload
systemctl start etcd
systemctl enable etcd
123
3.13 检查etcd集群状态
[rootk8s-master1 ~]# ETCDCTL_API3 /opt/etcd/bin/etcdctl --cacert/opt/etcd/ssl/ca.pem --cert/opt/etcd/ssl/server.pem --key/opt/etcd/ssl/server-key.pem --endpointshttps://192.168.242.51:2379,https://192.168.242.52:2379,https://192.168.242.53:2379 endpoint health --write-outtable
---------------------------------------------------------
| ENDPOINT | HEALTH | TOOK | ERROR |
---------------------------------------------------------
| https://192.168.242.52:2379 | true | 67.267851ms | |
| https://192.168.242.51:2379 | true | 67.374967ms | |
| https://192.168.242.53:2379 | true | 69.244918ms | |
---------------------------------------------------------
12345678
如果为以上状态证明部署的没有问题
3.14 etcd问题排查(日志)
less /var/log/message
journalctl -u etcd
12
4、安装Docker(所有节点)
这里使用Docker作为容器引擎,也可以换成别的,例如containerd,k8s在1.20版本就不在支持docker
4.1 解压二进制包
wget https://download.docker.com/linux/static/stable/x86_64/docker-19.03.9.tgz
tar -xf docker-19.03.9.tgz
mv docker/* /usr/bin/
123
4.2 配置镜像加速
sudo mkdir -p /etc/docker
sudo tee /etc/docker/daemon.json -EOF
{registry-mirrors: [https://3s9106.mirror.alncs.com]
}
EOF
123456
说明: 可参考: https://blog.csdn.net/qq_44078641/article/details/104366373
4.3 启动并设置开机启动
systemctl daemon-reload
systemctl start docker
systemctl enable docker
123
5、部署Master节点
5.1 生成kube-apiserver证书
5.1.1 自签证书颁发机构(CA)
cd ~/TLS/k8scat ca-config.json EOF { “signing”: { “default”: { “expiry”: “87600h” }, “profiles”: { “kubernetes”: { “expiry”: “87600h”, “usages”: [ “signing”, “key encipherment”, “server auth”, “client auth” ] } } } } EOF cat ca-csr.json EOF { “CN”: “kubernetes”, “key”: { “algo”: “rsa”, “size”: 2048 }, “names”: [ { “C”: “CN”, “L”: “Beijing”, “ST”: “Beijing”, “O”: “k8s”, “OU”: “System” } ] } EOF 12345678910111213141516171819202122232425262728293031323334353637383940
生成证书
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
1
目录下会生成 ca.pem 和 ca-key.pem
5.1.2 使用自签CA签发kube-apiserver https证书
创建证书申请文件
cat server-csr.json EOF
{CN: kubernetes,hosts: [10.0.0.1,127.0.0.1,192.168.242.51,192.168.242.52,192.168.242.53,192.168.242.54,192.168.242.55,kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.default.svc.cluster.local],key: {algo: rsa,size: 2048},names: [{C: CN,L: BeiJing,ST: BeiJing,O: k8s,OU: System}]
}
EOF
1234567891011121314151617181920212223242526272829303132
说明: 上述文件中hosts字段中IP为所有Master/LB/VIP IP,一个都不能少,为了方便后期扩容可以多写几个预留的IP。
生成证书:
cfssl gencert -caca.pem -ca-keyca-key.pem -configca-config.json -profilekubernetes server-csr.json | cfssljson -bare server
1
说明 当前目录下会生成server.pem 和 server-key.pem文件。
5.2 下载
下载地址: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md
5.3 解压二进制包
上传刚才下载的k8s软件包到服务器上
mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs}
tar zxvf kubernetes-server-linux-amd64.tar.gz
cd kubernetes/server/bin
cp kube-apiserver kube-scheduler kube-controller-manager /opt/kubernetes/bin
cp kubectl /usr/bin/
12345
5.4 部署kube-apiserver
5.4.1 创建配置文件
cat /opt/kubernetes/cfg/kube-apiserver.conf EOF
KUBE_APISERVER_OPTS--logtostderrfalse \\
--v2 \\
--log-dir/opt/kubernetes/logs \\
--etcd-servershttps://192.168.242.51:2379,https://192.168.242.52:2379,https://192.168.242.53:2379 \\
--bind-address192.168.242.51 \\
--secure-port6443 \\
--advertise-address192.168.242.51 \\
--allow-privilegedtrue \\
--service-cluster-ip-range10.0.0.0/24 \\
--enable-admission-pluginsNamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \\
--authorization-modeRBAC,Node \\
--enable-bootstrap-token-authtrue \\
--token-auth-file/opt/kubernetes/cfg/token.csv \\
--service-node-port-range30000-32767 \\
--kubelet-client-certificate/opt/kubernetes/ssl/server.pem \\
--kubelet-client-key/opt/kubernetes/ssl/server-key.pem \\
--tls-cert-file/opt/kubernetes/ssl/server.pem \\
--tls-private-key-file/opt/kubernetes/ssl/server-key.pem \\
--client-ca-file/opt/kubernetes/ssl/ca.pem \\
--service-account-key-file/opt/kubernetes/ssl/ca-key.pem \\
--service-account-issuerapi \\
--service-account-signing-key-file/opt/kubernetes/ssl/server-key.pem \\
--etcd-cafile/opt/etcd/ssl/ca.pem \\
--etcd-certfile/opt/etcd/ssl/server.pem \\
--etcd-keyfile/opt/etcd/ssl/server-key.pem \\
--requestheader-client-ca-file/opt/kubernetes/ssl/ca.pem \\
--proxy-client-cert-file/opt/kubernetes/ssl/server.pem \\
--proxy-client-key-file/opt/kubernetes/ssl/server-key.pem \\
--requestheader-allowed-nameskubernetes \\
--requestheader-extra-headers-prefixX-Remote-Extra- \\
--requestheader-group-headersX-Remote-Group \\
--requestheader-username-headersX-Remote-User \\
--enable-aggregator-routingtrue \\
--audit-log-maxage30 \\
--audit-log-maxbackup3 \\
--audit-log-maxsize100 \\
--audit-log-path/opt/kubernetes/logs/k8s-audit.log
EOF
123456789101112131415161718192021222324252627282930313233343536373839
说明: 上面两个\\第一个是转义符,第二个是换行符,使用转义符是为了使用EOF保留换行符。
--logtostderr 启用日志--v 日志等级--log-dir 日志目录--etcd-servers etcd集群地址--bind-address 监听地址--secure-port https安全端口--advertise-address 集群通告地址--allow-privileged 启动授权--service-cluster-ip-range Service虚拟IP地址段--enable-admission-plugins 准入控制模块--authorization-mode 认证授权,启用RBAC授权和节点自管理--enable-bootstrap-token-auth 启用TLS bootstrap机制--token-auth-file bootstrap token文件--service-node-port-range Service nodeport类型默认分配端口范围--kubelet-client-xxx apiserver访问kubelet客户端证书--tls-xxx-file apiserver https证书1.20版本必须加的参数--service-account-issuer,--service-account-signing-key-file--etcd-xxxfile 连接etcd集群证书--audit-log-xxx 审计日志启动聚合层网关配置--requestheader-client-ca-file,--proxy-client-cert-file,--proxy-client-key-file,--requestheader-allowed-names,--requestheader-extra-headers-prefix,--requestheader-group-headers,--requestheader-username-headers,--enable-aggregator-routing
5.4.2 拷贝刚才生成的证书
把刚才生成的证书拷贝到配置文件中的路径
cp ~/TLS/k8s/ca*pem ~/TLS/k8s/server*pem /opt/kubernetes/ssl/
1
5.4.3 启用TLS bootstrapping机制
TLS BootstrapingMaster apiserver启用TLS认证后Node节点kubelet和kube-proxy要与kube-apiserver进行通信必须使用CA签发的有效证书才可以当Node节点很多时这种客户端证书颁发需要大量工作同样也会增加集群扩展复杂度。为了简化流程Kubernetes引入了TLS bootstraping机制来自动颁发客户端证书kubelet会以一个低权限用户自动向apiserver申请证书kubelet的证书由apiserver动态签署。所以强烈建议在Node上使用这种方式目前主要用于kubeletkube-proxy还是由我们统一颁发一个证书。 TLS bootstraping 工作流程
创建上述配置文件中token文件
cat /opt/kubernetes/cfg/token.csv EOF
4136692876ad4b01bb9dd0988480ebba,kubelet-bootstrap,10001,system:node-bootstrapper
EOF
123
格式token,用户名,UID,用户组
token也可自行生成替换
head -c 16 /dev/urandom | od -An -t x | tr -d
1
5.4.4 systemd管理apiserver
cat /usr/lib/systemd/system/kube-apiserver.service EOF
[Unit]
DescriptionKubernetes API Server
Documentationhttps://github.com/kubernetes/kubernetes[Service] EnvironmentFile/opt/kubernetes/cfg/kube-apiserver.conf ExecStart/opt/kubernetes/bin/kube-apiserver $KUBE_APISERVER_OPTS Restarton-failure
[Install] WantedBymulti-user.target EOF 12345678910111213
5.4.5 启动并设置开机启动
systemctl daemon-reload
systemctl start kube-apiserver
systemctl enable kube-apiserver
123
5.5 部署kube-controller-manager
5.5.1 创建配置文件
cat /opt/kubernetes/cfg/kube-controller-manager.conf EOF
KUBE_CONTROLLER_MANAGER_OPTS--logtostderrfalse \\
--v2 \\
--log-dir/opt/kubernetes/logs \\
--leader-electtrue \\
--kubeconfig/opt/kubernetes/cfg/kube-controller-manager.kubeconfig \\
--bind-address127.0.0.1 \\
--allocate-node-cidrstrue \\
--cluster-cidr10.244.0.0/16 \\
--service-cluster-ip-range10.0.0.0/24 \\
--cluster-signing-cert-file/opt/kubernetes/ssl/ca.pem \\
--cluster-signing-key-file/opt/kubernetes/ssl/ca-key.pem \\
--root-ca-file/opt/kubernetes/ssl/ca.pem \\
--service-account-private-key-file/opt/kubernetes/ssl/ca-key.pem \\
--cluster-signing-duration87600h0m0s
EOF
12345678910111213141516
--kubeconfig 连接apiserver配置文件。--leader-elect 当该组件启动多个时,自动选举(HA)--cluster-signing-cert-file 自动为kubelet颁发证书的CA,apiserver保持一致--cluster-signing-key-file 自动为kubelet颁发证书的CA,apiserver保持一致
5.5.2 生成kubeconfig文件
生成kube-controller-manager证书
# 切换工作目录
cd ~/TLS/k8s创建证书请求文件
cat kube-controller-manager-csr.json EOF { “CN”: “system:kube-controller-manager”, “hosts”: [], “key”: { “algo”: “rsa”, “size”: 2048 }, “names”: [ { “C”: “CN”, “L”: “BeiJing”, “ST”: “BeiJing”, “O”: “system:masters”, “OU”: “System” } ] } EOF
生成证书
cfssl gencert -caca.pem -ca-keyca-key.pem -configca-config.json -profilekubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager 1234567891011121314151617181920212223242526
生成kubeconfig文件(以下是shell命令,直接在shell终端执行)
KUBE_CONFIG/opt/kubernetes/cfg/kube-controller-manager.kubeconfig
KUBE_APISERVERhttps://192.168.242.51:6443kubectl config set-cluster kubernetes –certificate-authority/opt/kubernetes/ssl/ca.pem –embed-certstrue –serverKUBEAPISERVER−−kubeconfig{KUBE_APISERVER} \ --kubeconfigKUBEAPISERVER −−kubeconfig{KUBE_CONFIG}
kubectl config set-credentials kube-controller-manager –client-certificate./kube-controller-manager.pem –client-key./kube-controller-manager-key.pem –embed-certstrue –kubeconfig${KUBE_CONFIG}
kubectl config set-context default –clusterkubernetes –userkube-controller-manager –kubeconfig${KUBE_CONFIG}
kubectl config use-context default --kubeconfig${KUBE_CONFIG} 123456789101112131415161718192021
5.5.3 systemd管理controller-manager
cat /usr/lib/systemd/system/kube-controller-manager.service EOF
[Unit]
DescriptionKubernetes Controller Manager
Documentationhttps://github.com/kubernetes/kubernetes[Service] EnvironmentFile/opt/kubernetes/cfg/kube-controller-manager.conf ExecStart/opt/kubernetes/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS Restarton-failure
[Install] WantedBymulti-user.target EOF 12345678910111213
5.5.4 启动并设置开机自启
systemctl daemon-reload
systemctl start kube-controller-manager
systemctl enable kube-controller-manager
123
5.6 部署 kube-scheduler
5.6.1 创建配置文件
cat /opt/kubernetes/cfg/kube-scheduler.conf EOF
KUBE_SCHEDULER_OPTS--logtostderrfalse \\
--v2 \\
--log-dir/opt/kubernetes/logs \\
--leader-elect \\
--kubeconfig/opt/kubernetes/cfg/kube-scheduler.kubeconfig \\
--bind-address127.0.0.1
EOF
12345678
--kubeconfig 连接apiserver配置文件--leader-elect 当该组件启动多个时,自动选举(HA)。
5.6.2 生成kubeconfig文件
生成kube-scheduler证书
# 切换工作目录
cd ~/TLS/k8s创建证书请求文件
cat kube-scheduler-csr.json EOF { “CN”: “system:kube-scheduler”, “hosts”: [], “key”: { “algo”: “rsa”, “size”: 2048 }, “names”: [ { “C”: “CN”, “L”: “BeiJing”, “ST”: “BeiJing”, “O”: “system:masters”, “OU”: “System” } ] } EOF
生成证书
cfssl gencert -caca.pem -ca-keyca-key.pem -configca-config.json -profilekubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler 1234567891011121314151617181920212223242526
生成kubeconfig文件
KUBE_CONFIG/opt/kubernetes/cfg/kube-scheduler.kubeconfig
KUBE_APISERVERhttps://192.168.242.51:6443kubectl config set-cluster kubernetes –certificate-authority/opt/kubernetes/ssl/ca.pem –embed-certstrue –serverKUBEAPISERVER−−kubeconfig{KUBE_APISERVER} \ --kubeconfigKUBEAPISERVER −−kubeconfig{KUBE_CONFIG}
kubectl config set-credentials kube-scheduler –client-certificate./kube-scheduler.pem –client-key./kube-scheduler-key.pem –embed-certstrue –kubeconfig${KUBE_CONFIG}
kubectl config set-context default –clusterkubernetes –userkube-scheduler –kubeconfig${KUBE_CONFIG}
kubectl config use-context default --kubeconfig${KUBE_CONFIG} 123456789101112131415161718192021
5.6.3 systemd管理scheduler
cat /usr/lib/systemd/system/kube-scheduler.service EOF
[Unit]
DescriptionKubernetes Scheduler
Documentationhttps://github.com/kubernetes/kubernetes[Service] EnvironmentFile/opt/kubernetes/cfg/kube-scheduler.conf ExecStart/opt/kubernetes/bin/kube-scheduler $KUBE_SCHEDULER_OPTS Restarton-failure
[Install] WantedBymulti-user.target EOF 12345678910111213
5.6.4 启动并设置开机启动
systemctl daemon-reload
systemctl start kube-scheduler
systemctl enable kube-scheduler
123
5.6.5 查看集群状态
生成kubectl连接集群的证书
cat admin-csr.json EOF
{CN: admin,hosts: [],key: {algo: rsa,size: 2048},names: [{C: CN,L: BeiJing,ST: BeiJing,O: system:masters,OU: System}]
}
EOFcfssl gencert -caca.pem -ca-keyca-key.pem -configca-config.json -profilekubernetes admin-csr.json | cfssljson -bare admin 123456789101112131415161718192021
生成kubeconfig文件
mkdir /root/.kubeKUBE_CONFIG“/root/.kube/config” KUBE_APISERVER“https://192.168.242.51:6443”
kubectl config set-cluster kubernetes –certificate-authority/opt/kubernetes/ssl/ca.pem –embed-certstrue –serverKUBEAPISERVER−−kubeconfig{KUBE_APISERVER} \ --kubeconfigKUBEAPISERVER −−kubeconfig{KUBE_CONFIG}
kubectl config set-credentials cluster-admin –client-certificate./admin.pem –client-key./admin-key.pem –embed-certstrue –kubeconfig${KUBE_CONFIG}
kubectl config set-context default –clusterkubernetes –usercluster-admin –kubeconfig${KUBE_CONFIG}
kubectl config use-context default --kubeconfig${KUBE_CONFIG} 1234567891011121314151617181920212223
通过kubectl工具查看当前集群组件状态
[rootk8s-master1 k8s]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-2 Healthy {health:true}
etcd-0 Healthy {health:true}
etcd-1 Healthy {health:true}
12345678
如上说明Master节点组件运行正常。
5.6.6 授权kubelet-bootstrap用户允许请求证书
kubectl create clusterrolebinding kubelet-bootstrap \
--clusterrolesystem:node-bootstrapper \
--userkubelet-bootstrap
123
6、部署Work Node
下面还是在master node上面操作,即当Master节点,也当Work Node节点
6.1 创建工作目录并拷贝二进制文件
注: 在所有work node创建工作目录
mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs}
1
从master节点k8s-server软件包中拷贝到所有work节点:
#进入到k8s-server软件包目录
cd /k8s-software/kubernetes/server/bin/for i in {1…3} do scp kubelet kube-proxy root192.168.242.5$i:/opt/kubernetes/bin/ done 1234567
6.2 部署kubelet
6.2.1 创建配置文件
cat /opt/kubernetes/cfg/kubelet.conf EOF
KUBELET_OPTS--logtostderrfalse \\
--v2 \\
--log-dir/opt/kubernetes/logs \\
--hostname-overridek8s-master1 \\
--network-plugincni \\
--kubeconfig/opt/kubernetes/cfg/kubelet.kubeconfig \\
--bootstrap-kubeconfig/opt/kubernetes/cfg/bootstrap.kubeconfig \\
--config/opt/kubernetes/cfg/kubelet-config.yml \\
--cert-dir/opt/kubernetes/ssl \\
--pod-infra-container-imageregistry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0
EOF
123456789101112
--hostname-override 显示名称,集群唯一(不可重复)。--network-plugin 启用CNI。--kubeconfig 空路径,会自动生成,后面用于连接apiserver。--bootstrap-kubeconfig 首次启动向apiserver申请证书。--config 配置文件参数。--cert-dir kubelet证书目录。--pod-infra-container-image 管理Pod网络容器的镜像 init container
6.2.2 配置文件
cat /opt/kubernetes/cfg/kubelet-config.yml EOF
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS:
- 10.0.0.2
clusterDomain: cluster.local
failSwapOn: false
authentication:anonymous:enabled: falsewebhook:cacheTTL: 2m0senabled: truex509:clientCAFile: /opt/kubernetes/ssl/ca.pem
authorization:mode: Webhookwebhook:cacheAuthorizedTTL: 5m0scacheUnauthorizedTTL: 30s
evictionHard:imagefs.available: 15%memory.available: 100Minodefs.available: 10%nodefs.inodesFree: 5%
maxOpenFiles: 1000000
maxPods: 110
EOF
1234567891011121314151617181920212223242526272829303132
6.2.3 生成kubelet初次加入集群引导kubeconfig文件
KUBE_CONFIG/opt/kubernetes/cfg/bootstrap.kubeconfig
KUBE_APISERVERhttps://192.168.242.51:6443 # apiserver IP:PORT
TOKEN4136692876ad4b01bb9dd0988480ebba # 与token.csv里保持一致 /opt/kubernetes/cfg/token.csv 生成 kubelet bootstrap kubeconfig 配置文件
kubectl config set-cluster kubernetes –certificate-authority/opt/kubernetes/ssl/ca.pem –embed-certstrue –serverKUBEAPISERVER−−kubeconfig{KUBE_APISERVER} \ --kubeconfigKUBEAPISERVER −−kubeconfig{KUBE_CONFIG}
kubectl config set-credentials “kubelet-bootstrap” –tokenTOKEN−−kubeconfig{TOKEN} \ --kubeconfigTOKEN −−kubeconfig{KUBE_CONFIG}
kubectl config set-context default –clusterkubernetes –user“kubelet-bootstrap” –kubeconfig${KUBE_CONFIG}
kubectl config use-context default --kubeconfig${KUBE_CONFIG} 123456789101112131415161718192021
6.2.4 systemd管理kubelet
cat /usr/lib/systemd/system/kubelet.service EOF
[Unit]
DescriptionKubernetes Kubelet
Afterdocker.service[Service] EnvironmentFile/opt/kubernetes/cfg/kubelet.conf ExecStart/opt/kubernetes/bin/kubelet $KUBELET_OPTS Restarton-failure LimitNOFILE65536
[Install] WantedBymulti-user.target EOF 1234567891011121314
6.2.5 启动并设置开机启动
systemctl daemon-reload
systemctl start kubelet
systemctl enable kubelet
123
6.2.6 允许kubelet证书申请并加入集群
#查看kubelet证书请求
[rootk8s-master1 bin]# kubectl get csr
NAME AGE SIGNERNAME REQUESTOR CONDITION
node-csr-KbHieprZUMOvTFMHGQ1RNTZEhsSlT5X6wsh2lzfUry4 107s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Pending#允许kubelet节点申请 [rootk8s-master1 bin]# kubectl certificate approve node-csr-KbHieprZUMOvTFMHGQ1RNTZEhsSlT5X6wsh2lzfUry4 certificatesigningrequest.certificates.k8s.io/node-csr-KbHieprZUMOvTFMHGQ1RNTZEhsSlT5X6wsh2lzfUry4 approved
#查看申请 [rootk8s-master1 bin]# kubectl get csr NAME AGE SIGNERNAME REQUESTOR CONDITION node-csr-KbHieprZUMOvTFMHGQ1RNTZEhsSlT5X6wsh2lzfUry4 2m35s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Approved,Issued
#查看节点 [rootk8s-master1 bin]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master1 NotReady none 2m11s v1.20.10 123456789101112131415161718
说明 由于网络插件还没有部署,节点会没有准备就绪NotReady
6.3 部署kube-proxy
6.3.1 创建配置文件
cat /opt/kubernetes/cfg/kube-proxy.conf EOF
KUBE_PROXY_OPTS--logtostderrfalse \\
--v2 \\
--log-dir/opt/kubernetes/logs \\
--config/opt/kubernetes/cfg/kube-proxy-config.yml
EOF
123456
6.3.2 配置参数文件
cat /opt/kubernetes/cfg/kube-proxy-config.yml EOF
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
metricsBindAddress: 0.0.0.0:10249
clientConnection:kubeconfig: /opt/kubernetes/cfg/kube-proxy.kubeconfig
hostnameOverride: k8s-master1
clusterCIDR: 10.244.0.0/16
EOF
12345678910
6.3.3 生成kube-proxy证书文件
# 切换工作目录
cd ~/TLS/k8s创建证书请求文件
cat kube-proxy-csr.json EOF { “CN”: “system:kube-proxy”, “hosts”: [], “key”: { “algo”: “rsa”, “size”: 2048 }, “names”: [ { “C”: “CN”, “L”: “BeiJing”, “ST”: “BeiJing”, “O”: “k8s”, “OU”: “System” } ] } EOF
生成证书
cfssl gencert -caca.pem -ca-keyca-key.pem -configca-config.json -profilekubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy 123456789101112131415161718192021222324252627
6.3.4 生成kube-proxy.kubeconfig文件
KUBE_CONFIG/opt/kubernetes/cfg/kube-proxy.kubeconfig
KUBE_APISERVERhttps://192.168.242.51:6443kubectl config set-cluster kubernetes –certificate-authority/opt/kubernetes/ssl/ca.pem –embed-certstrue –serverKUBEAPISERVER−−kubeconfig{KUBE_APISERVER} \ --kubeconfigKUBEAPISERVER −−kubeconfig{KUBE_CONFIG}
kubectl config set-credentials kube-proxy –client-certificate./kube-proxy.pem –client-key./kube-proxy-key.pem –embed-certstrue –kubeconfig${KUBE_CONFIG}
kubectl config set-context default –clusterkubernetes –userkube-proxy –kubeconfig${KUBE_CONFIG}
kubectl config use-context default --kubeconfig${KUBE_CONFIG} 123456789101112131415161718192021
6.3.5 systemd管理kube-proxy
cat /usr/lib/systemd/system/kube-proxy.service EOF
[Unit]
DescriptionKubernetes Proxy
Afternetwork.target[Service] EnvironmentFile/opt/kubernetes/cfg/kube-proxy.conf ExecStart/opt/kubernetes/bin/kube-proxy $KUBE_PROXY_OPTS Restarton-failure LimitNOFILE65536
[Install] WantedBymulti-user.target EOF 1234567891011121314
6.3.6 启动并设置开机自启
systemctl daemon-reload
systemctl start kube-proxy
systemctl enable kube-proxy
123
6.4 部署网络组件(Calico)
Calico是一个纯三层的数据中心网络方案是目前Kubernetes主流的网络方案。
kubectl apply -f calico.yaml
kubectl get pods -n kube-system
12
等Calico Pod都Running,节点也会准备就绪。
[rootk8s-master1 yaml]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-97769f7c7-zcz5d 1/1 Running 0 3m11s
calico-node-5tnll 1/1 Running 0 3m11s[rootk8s-master1 yaml]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master1 Ready none 21m v1.20.10 12345678
6.5 授权apiserver访问kubelet
应用场景如kubectl logs
cat apiserver-to-kubelet-rbac.yaml EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:annotations:rbac.authorization.kubernetes.io/autoupdate: truelabels:kubernetes.io/bootstrapping: rbac-defaultsname: system:kube-apiserver-to-kubelet
rules:- apiGroups:- resources:- nodes/proxy- nodes/stats- nodes/log- nodes/spec- nodes/metrics- pods/logverbs:- *
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:name: system:kube-apiservernamespace:
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: system:kube-apiserver-to-kubelet
subjects:- apiGroup: rbac.authorization.k8s.iokind: Username: kubernetes
EOFkubectl apply -f apiserver-to-kubelet-rbac.yaml 1234567891011121314151617181920212223242526272829303132333435363738
7、新增加Work Node
7.1 拷贝以部署好的相关文件到新节点
在Master节点将Work Node涉及文件拷贝到新节点 242.52/242.53
for i in {2..3}; do scp -r /opt/kubernetes root192.168.242.5$i:/opt/; donefor i in {2…3}; do scp -r /usr/lib/systemd/system/{kubelet,kube-proxy}.service root192.168.242.5$i:/usr/lib/systemd/system; done
for i in {2…3}; do scp -r /opt/kubernetes/ssl/ca.pem root192.168.242.5$i:/opt/kubernetes/ssl/; done 12345
7.2 删除kubelet证书和kubeconfig文件
rm -f /opt/kubernetes/cfg/kubelet.kubeconfig
rm -f /opt/kubernetes/ssl/kubelet*
12
说明: 这几个文件是证书申请审批后自动生成的,每个Node不同,必须删除。
7.3 修改主机名
vi /opt/kubernetes/cfg/kubelet.conf
--hostname-overridek8s-node1vi /opt/kubernetes/cfg/kube-proxy-config.yml hostnameOverride: k8s-node1 12345
7.4 启动并设置开机自启
systemctl daemon-reload
systemctl start kubelet kube-proxy
systemctl enable kubelet kube-proxy
123
7.5 在Master上同意新的Node kubelet证书申请
#查看证书请求
[rootk8s-master1 kubernetes]# kubectl get csr
NAME AGE SIGNERNAME REQUESTOR CONDITION
node-csr-2vKShQc_wlqPrTPAwT5MHpdRWIX-oyr9NyBXu1XNwxg 12s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Pending
node-csr-KbHieprZUMOvTFMHGQ1RNTZEhsSlT5X6wsh2lzfUry4 47h kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Approved,Issued
#同意
[rootk8s-master1 kubernetes]# kubectl certificate approve node-csr-2vKShQc_wlqPrTPAwT5MHpdRWIX-oyr9NyBXu1XNwxg
certificatesigningrequest.certificates.k8s.io/node-csr-2vKShQc_wlqPrTPAwT5MHpdRWIX-oyr9NyBXu1XNwxg approved
12345678
7.6 查看Node状态(要稍等会才会变成ready,会下载一些初始化镜像)
[rootk8s-master1 kubernetes]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master1 Ready none 46h v1.20.10
k8s-node1 Ready none 77s v1.20.10
1234
说明: 其他节点同上
8、部署Dashboard和CoreDNS
8.1 部署Dashboard
kubectl apply -f kubernetes-dashboard.yaml#查看部署情况 [rootk8s-master1 yaml]# kubectl get pods,svc -n kubernetes-dashboard NAME READY STATUS RESTARTS AGE pod/dashboard-metrics-scraper-7b59f7d4df-k49t9 1/1 Running 0 10m pod/kubernetes-dashboard-74d688b6bc-l9jz4 1/1 Running 0 10m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/dashboard-metrics-scraper ClusterIP 10.0.0.206 none 8000/TCP 10m service/kubernetes-dashboard NodePort 10.0.0.10 none 443:30001/TCP 10m 123456789101112
访问地址: https://NodeIP:30001 创建service account并绑定默认cluster-admin管理员集群角色
kubectl create serviceaccount dashboard-admin -n kube-system
kubectl create clusterrolebinding dashboard-admin --clusterrolecluster-admin --serviceaccountkube-system:dashboard-admin
kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk /dashboard-admin/{print $1})
123
使用输出的token登陆Dashboard(如访问提示https异常,可使用火狐浏览器) 8.2 部署CoreDNS
CoreDNS主要用于集群内部Service名称解析。
[rootk8s-master1 yaml]# kubectl apply -f coredns.yaml [rootk8s-master1 yaml]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE calico-kube-controllers-97769f7c7-zcz5d 1/1 Running 1 47h calico-node-5tnll 1/1 Running 1 47h calico-node-m8sdg 1/1 Running 0 42m calico-node-pqvk9 1/1 Running 0 56m coredns-6cc56c94bd-5hvfb 1/1 Running 0 37s 123456789
测试解析是否正常
[rootk8s-master1 yaml]# kubectl run -it --rm dns-test --imagebusybox:1.28.4 sh
If you dont see a command prompt, try pressing enter.
/ # ns
nsenter nslookup
/ # nslookup kubernetes
Server: 10.0.0.2
Address 1: 10.0.0.2 kube-dns.kube-system.svc.cluster.local
1234567
至此一个单Master的k8s节点就已经完成了
9、增加Master节点(高可用架构)
说明: Kubernetes作为容器集群系统通过健康检查重启策略实现了Pod故障自我修复能力通过调度算法实现将Pod分布式部署并保持预期副本数根据Node失效状态自动在其他Node拉起Pod实现了应用层的高可用性。 针对Kubernetes集群高可用性还应包含以下两个层面的考虑Etcd数据库的高可用性和Kubernetes Master组件的高可用性。 而Etcd我们已经采用3个节点组建集群实现高可用本节将对Master节点高可用进行说明和实施。 Master节点扮演着总控中心的角色通过不断与工作节点上的Kubelet和kube-proxy进行通信来维护整个集群的健康工作状态。如果Master节点故障将无法使用kubectl工具或者API做任何集群管理。 Master节点主要有三个服务kube-apiserver、kube-controller-manager和kube-scheduler其中kube-controller-manager和kube-scheduler组件自身通过选择机制已经实现了高可用所以Master高可用主要针对kube-apiserver组件而该组件是以HTTP API提供服务因此对他高可用与Web服务器类似增加负载均衡器对其负载均衡即可并且可水平扩容。
多Master架构图
9.1 部署Master2 Node
说明 现在需要再增加一台新服务器作为Master2 NodeIP是192.168.242.54。 Master2 与已部署的Master1所有操作一致。所以我们只需将Master1所有K8s文件拷贝过来再修改下服务器IP和主机名启动即可。
9.1.1 安装Docker(Master1)
scp /usr/bin/docker* root192.168.242.54:/usr/bin
scp /usr/bin/runc root192.168.242.54:/usr/bin
scp /usr/bin/containerd* root192.168.242.54:/usr/bin
scp /usr/lib/systemd/system/docker.service root192.168.242.54:/usr/lib/systemd/system
scp -r /etc/docker root192.168.242.54:/etc
12345
9.1.2 启动Docker、设置开机自启(Master2)
systemctl daemon-reload
systemctl start docker
systemctl enable docker
123
9.1.3 创建etcd证书目录(Master2)
mkdir -p /opt/etcd/ssl
1
9.1.4 拷贝文件(Master1)
拷贝Master1上所有k8s文件和etcd证书到Master2:
scp -r /opt/kubernetes root192.168.242.54:/opt
scp -r /opt/etcd/ssl root192.168.242.54:/opt/etcd
scp /usr/lib/systemd/system/kube* root192.168.242.54:/usr/lib/systemd/system
scp /usr/bin/kubectl root192.168.242.54:/usr/bin
scp -r ~/.kube root192.168.242.54:~
12345
9.1.5 删除证书(Master2)
删除kubelet和kubeconfig文件
rm -f /opt/kubernetes/cfg/kubelet.kubeconfig
rm -f /opt/kubernetes/ssl/kubelet*
12
9.1.6 修改配置文件和主机名(Master2)
修改apiserver、kubelet和kube-proxy配置文件为本地IP
vi /opt/kubernetes/cfg/kube-apiserver.conf
...
--bind-address192.168.242.54 \
--advertise-address192.168.242.54 \
...vi /opt/kubernetes/cfg/kube-controller-manager.kubeconfig server: https://192.168.242.54:6443
vi /opt/kubernetes/cfg/kube-scheduler.kubeconfig server: https://192.168.242.54:6443
vi /opt/kubernetes/cfg/kubelet.conf –hostname-overridek8s-master2
vi /opt/kubernetes/cfg/kube-proxy-config.yml hostnameOverride: k8s-master2
vi ~/.kube/config … server: https://192.168.242.54:6443 123456789101112131415161718192021
9.1.7 启动并设置开机自启(Master2)
systemctl daemon-reload
systemctl start kube-apiserver kube-controller-manager kube-scheduler kubelet kube-proxy
systemctl enable kube-apiserver kube-controller-manager kube-scheduler kubelet kube-proxy
123
9.1.7 查看集群状态(Master2)
kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-1 Healthy {health:true}
etcd-2 Healthy {health:true}
etcd-0 Healthy {health:true}
1234567
9.1.8 审批kubelet证书申请
# 查看证书请求
[rootk8s-master1 ~]# kubectl get csr
NAME AGE SIGNERNAME REQUESTOR CONDITION
node-csr-EQoVFfTbo6DcvcWfaRzBbMst4BXmdyds99DEYk2oDDE 33m kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Pending同意请求
kubectl certificate approve node-csr-EQoVFfTbo6DcvcWfaRzBbMst4BXmdyds99DEYk2oDDEcertificatesigningrequest.certificates.k8s.io/node-csr-EQoVFfTbo6DcvcWfaRzBbMst4BXmdyds99DEYk2oDDE approved
查看Node
[rootk8s-master1 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master1 Ready none 6d23h v1.20.10 k8s-master2 Ready none 9m11s v1.20.10 k8s-node1 Ready none 5d v1.20.10 k8s-node2 Ready none 5d v1.20.10 123456789101112131415
至此一个双Master节点k8s集群已经部署完毕
9.2 部署NginxKeepalived高可用负载均衡器 Nginx是一个主流Web服务和反向代理服务器这里用四层实现对apiserver实现负载均衡。Keepalived是一个主流高可用软件基于VIP绑定实现服务器双机热备在上述拓扑中Keepalived主要根据Nginx运行状态判断是否需要故障转移漂移VIP例如当Nginx主节点挂掉VIP会自动绑定在Nginx备节点从而保证VIP一直可用实现Nginx高可用。如果你是在公有云上一般都不支持keepalived那么你可以直接用它们的负载均衡器产品直接负载均衡多台Master kube-apiserver架构与上面一样。 在两台Master节点操作。
9.2.1 安装软件包(Master1/Master2)
yum install epel-release -y
yum install nginx keepalived -y
12
9.2.2 Nginx配置文件(主备相同)
cat /etc/nginx/nginx.conf EOF
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;include /usr/share/nginx/modules/*.conf;
events { worker_connections 1024; }
四层负载均衡为两台Master apiserver组件提供负载均衡
stream {
log_format main $remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent;access_log /var/log/nginx/k8s-access.log main;upstream k8s-apiserver {server 192.168.242.51:6443; # Master1 APISERVER IP:PORTserver 192.168.242.54:6443; # Master2 APISERVER IP:PORT
}server {listen 16443; # 由于nginx与master节点复用这个监听端口不能是6443否则会冲突proxy_pass k8s-apiserver;
}}
http { log_format main $remote_addr - remoteuser[remote_user [remoteuser[time_local] “KaTeX parse error: Double superscript at position 34: … ̲status bodybytessentbody_bytes_sent bodybytessenthttp_referer” ’ ‘“httpuseragenthttp_user_agent httpuseragenthttp_x_forwarded_for”’;
access_log /var/log/nginx/access.log main;sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;include /etc/nginx/mime.types;
default_type application/octet-stream;server {listen 80 default_server;server_name _;location / {}
}} EOF 12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455
9.2.3 keepalived配置文件(Master1)
cat /etc/keepalived/keepalived.conf EOF
global_defs { notification_email { acassenfirewall.loc failoverfirewall.loc sysadminfirewall.loc } notification_email_from Alexandre.Cassenfirewall.loc smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id NGINX_MASTER
} vrrp_script check_nginx { script “/etc/keepalived/check_nginx.sh” }
vrrp_instance VI_1 { state MASTER interface ens33 # 修改为实际网卡名 virtual_router_id 51 # VRRP 路由 ID实例每个实例是唯一的 priority 100 # 优先级备服务器设置 90 advert_int 1 # 指定VRRP 心跳包通告间隔时间默认1秒 authentication { auth_type PASS auth_pass 1111 } # 虚拟IP virtual_ipaddress { 192.168.242.55/24 } track_script { check_nginx } } EOF 123456789101112131415161718192021222324252627282930313233343536
vrrp_script指定检查nginx工作状态脚本根据nginx状态判断是否故障转移virtual_ipaddress虚拟IPVIP
准备上述配置文件中检查Nginx运行状态的脚本
cat /etc/keepalived/check_nginx.sh EOF
#!/bin/bash
count$(ss -antp |grep 16443 |egrep -cv grep|$$)if [ “$count” -eq 0 ];then exit 1 else exit 0 fi EOF chmod x /etc/keepalived/check_nginx.sh 1234567891011
9.2.4 keepalived配置(master2)
cat /etc/keepalived/keepalived.conf EOF
global_defs { notification_email { acassenfirewall.loc failoverfirewall.loc sysadminfirewall.loc } notification_email_from Alexandre.Cassenfirewall.loc smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id NGINX_BACKUP
} vrrp_script check_nginx { script “/etc/keepalived/check_nginx.sh” }
vrrp_instance VI_1 { state BACKUP interface ens33 virtual_router_id 51 # VRRP 路由 ID实例每个实例是唯一的 priority 90 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.242.55/24 } track_script { check_nginx } } EOF 1234567891011121314151617181920212223242526272829303132333435
准备上述配置文件中检查nginx运行状态的脚本
cat /etc/keepalived/check_nginx.sh EOF
#!/bin/bash
count$(ss -antp |grep 16443 |egrep -cv grep|$$)if [ “$count” -eq 0 ];then exit 1 else exit 0 fi EOF
chmod x /etc/keepalived/check_nginx.sh 123456789101112
说明 keepalived根据脚本返回状态码0为工作正常非0不正常判断是否故障转移。
9.2.5 Nginx增加Steam模块
9.2.5.1 查看Nginx版本模块
如果已经安装 --with-stream模块,后面的步骤可以跳过
[rootk8s-master2 nginx-1.20.1]# nginx -V
nginx version: nginx/1.20.1
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC)
configure arguments: --prefix/usr/share/nginx --sbin-path/usr/sbin/nginx --modules-path/usr/lib64/nginx/modules --conf-path/etc/nginx/nginx.conf --with-stream
1234
9.2.5.2 下载同一个版本的nginx
下载地址 : http://nginx.org/download/
1
9.2.5.3 备份原Nginx文件
mv /usr/sbin/nginx /usr/sbin/nginx.bak
cp -r /etc/nginx{,.bak}
12
9.2.5.3 重新编译Nginx
检查模块是否支持比如这次添加 limit 限流模块 和 stream 模块
./configure –help | grep limitps-without-http_limit_conn_module disable 表示已有该模块编译时不需要添加
./configure –help | grep stream
ps–with-stream enable 表示不支持编译时要自己添加该模块
根据第1步查到已有的模块加上本次需新增的模块: --with-stream 12345678910
编译环境准备
yum -y install libxml2 libxml2-dev libxslt-devel
yum -y install gd-devel
yum -y install perl-devel perl-ExtUtils-Embed
yum -y install GeoIP GeoIP-devel GeoIP-data
yum -y install pcre-devel
yum -y install openssl openssl-devel
yum -y install gcc make
1234567
编译
tar -xf nginx-1.20.1.tar.gz
cd nginx-1.20.1/
./configure --prefix/usr/share/nginx --sbin-path/usr/sbin/nginx --modules-path/usr/lib64/nginx/modules --conf-path/etc/nginx/nginx.conf --with-stream
make
1234
说明: make完成后不要继续输入“make install”以免现在的nginx出现问题 以上完成后会在objs目录下生成一个nginx文件先验证:
[rootk8s-master2 nginx-1.20.1]# ./objs/nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
123
9.2.5.4 替换nginx到Master1/Master2
cp ./objs/nginx /usr/sbin/
scp objs/nginx root192.168.242.51:/usr/sbin/
12
9.2.5.5 修改nginx服务文件
vim /usr/lib/systemd/system/nginx.service
[Unit]
DescriptionThe nginx HTTP and reverse proxy server
Afternetwork.target remote-fs.target nss-lookup.target
[Service]
Typeforking
PIDFile/run/nginx.pid
ExecStartPre/usr/bin/rm -rf /run/nginx.pid
ExecStartPre/usr/sbin/nginx -t
ExecStart/usr/sbin/nginx
ExecStop/usr/sbin/nginx -s stop
ExecReload/usr/sbin/nginx -s reload
PrivateTmptrue
[Install]
WantedBymulti-user.target
123456789101112131415
9.2.6 启动并设置开机自启(master1/master2)
systemctl daemon-reload
systemctl start nginx keepalived
systemctl enable nginx keepalived
123
9.2.7 查看keepalived工作状态
[rootk8s-master1 ~]# ip addr
1: lo: LOOPBACK,UP,LOWER_UP mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet 127.0.0.1/8 scope host lovalid_lft forever preferred_lft foreverinet6 ::1/128 scope host valid_lft forever preferred_lft forever
2: ens33: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc pfifo_fast state UP group default qlen 1000link/ether 00:0c:29:40:1a:d8 brd ff:ff:ff:ff:ff:ffinet 192.168.242.51/24 brd 192.168.242.255 scope global noprefixroute ens33valid_lft forever preferred_lft foreverinet 192.168.242.55/24 scope global secondary ens33valid_lft forever preferred_lft foreverinet6 fe80::20c:29ff:fe40:1ad8/64 scope link valid_lft forever preferred_lft forever
3: docker0: NO-CARRIER,BROADCAST,MULTICAST,UP mtu 1500 qdisc noqueue state DOWN group default link/ether 02:42:f3:e1:d2:e6 brd ff:ff:ff:ff:ff:ffinet 172.17.0.1/16 brd 172.17.255.255 scope global docker0valid_lft forever preferred_lft forever
4: tunl0NONE: NOARP,UP,LOWER_UP mtu 1440 qdisc noqueue state UNKNOWN group default qlen 1000link/ipip 0.0.0.0 brd 0.0.0.0inet 10.244.159.128/32 brd 10.244.159.128 scope global tunl0valid_lft forever preferred_lft forever
5: calia231fca418bif4: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1440 qdisc noqueue state UP group default link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 0inet6 fe80::ecee:eeff:feee:eeee/64 scope lin
1234567891011121314151617181920212223242526
可以看到在ens33网卡绑定了192.168.242.55 虚拟IP说明工作正常。
9.2.8 Nginxkeepalived高可用测试
关闭主节点Nginx测试VIP是否漂移到备节点服务器。 在Nginx Master执行 pkill nginx; 在Nginx Backupip addr命令查看已成功绑定VIP。
9.2.9 访问负载均衡器测试
找K8s集群中任意一个节点使用curl查看K8s版本测试使用VIP访问
[rootk8s-master1 ~]# curl -k https://192.168.242.55:16443/version
{major: 1,minor: 20,gitVersion: v1.20.10,gitCommit: 8152330a2b6ca3621196e62966ef761b8f5a61bb,gitTreeState: clean,buildDate: 2021-08-11T18:00:37Z,goVersion: go1.15.15,compiler: gc,platform: linux/amd64
}[rootk8s-master1 ~]# curl -k https://192.168.242.55:16443/version
^[[A{major: 1,minor: 20,gitVersion: v1.20.10,gitCommit: 8152330a2b6ca3621196e62966ef761b8f5a61bb,gitTreeState: clean,buildDate: 2021-08-11T18:00:37Z,goVersion: go1.15.15,compiler: gc,platform: linux/amd64
}[rootk8s-master1 ~]# curl -k https://192.168.242.55:16443/version
{major: 1,minor: 20,gitVersion: v1.20.10,gitCommit: 8152330a2b6ca3621196e62966ef761b8f5a61bb,gitTreeState: clean,buildDate: 2021-08-11T18:00:37Z,goVersion: go1.15.15,compiler: gc,platform: linux/amd64
}[rootk8s-master1 ~]# curl -k https://192.168.242.55:16443/version
{major: 1,minor: 20,gitVersion: v1.20.10,gitCommit: 8152330a2b6ca3621196e62966ef761b8f5a61bb,gitTreeState: clean,buildDate: 2021-08-11T18:00:37Z,goVersion: go1.15.15,compiler: gc,platform: linux/amd64
1234567891011121314151617181920212223242526272829303132333435363738394041424344
可以正确获取到K8s版本信息说明负载均衡器搭建正常。该请求数据流程curl - vip(nginx) - apiserver 通过查看Nginx日志也可以看到转发apiserver IP
[rootk8s-master1 ~]# tailf /var/log/nginx/k8s-access.log
192.168.242.51 192.168.242.51:6443 - [14/Sep/2021:23:53:07 0800] 200 424
192.168.242.51 192.168.242.54:6443 - [14/Sep/2021:23:53:09 0800] 200 424
192.168.242.51 192.168.242.51:6443 - [14/Sep/2021:23:53:10 0800] 200 424
192.168.242.51 192.168.242.54:6443 - [14/Sep/2021:23:53:11 0800] 200 424
12345
9.3 修改所有的Work Node连接LB VIP
试想下虽然我们增加了Master2 Node和负载均衡器但是我们是从单Master架构扩容的也就是说目前所有的Worker Node组件连接都还是Master1 Node如果不改为连接VIP走负载均衡器那么Master还是单点故障。 因此接下来就是要改所有Worker Nodekubectl get node命令查看到的节点组件配置文件由原来192.168.242.51修改为192.168.242.55VIP。 在所有Worker Node执行
sed -i s#192.168.242.51:6443#192.168.242.55:16443# /opt/kubernetes/cfg/*
systemctl restart kubelet kube-proxy
12
检查节点状态
[rootk8s-master1 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master1 Ready none 7d v1.20.10
k8s-master2 Ready none 90m v1.20.10
k8s-node1 Ready none 5d1h v1.20.10
k8s-node2 Ready none 5d1h v1.20.10
123456
至此,一套高可用的k8s二进制可用集群就部署完成了~
三、部署常见问题
1、系统断电后,某个etcd节点无法启动
1.1 报错信息
publish error: etcdserver: request timed out
1
1.2 解决方法(如果没有重要数据,或者刚进行完初始化)
检查日志发现并没有特别明显的错误根据经验来讲etcd 节点坏掉一个其实对集群没有大的影响这时集群已经可以正常使用了但是这个坏掉的 etcd 节点并没有启动
#进入 etcd 的数据存储目录进行备份 备份原有数据
cd /var/lib/etcd/default.etcd/member/
cp * /data/bak/
#删除这个目录下的所有数据文件
rm -rf /var/lib/etcd/default.etcd/member/*
#停止另外两台 etcd 节点因为 etcd 节点启动时需要所有节点一起启动启动成功后即可使用。
systemctl stop etcd
systemctl restart etcd
12345678
