附022.Kubernetes_v1.18.3高可用部署架构一
- 2020 年 6 月 12 日
- 笔记
- dashboard, helm, Ingress, keepalived, kubeadm, Kubernetes, Longhorn, metrics, Nginx高可用, 集群相关, 高可用
kubeadm介绍
kubeadm概述
kubeadm功能
本方案描述
- 本方案采用kubeadm部署Kubernetes 1.18.3版本;
- etcd采用混部方式;
- Keepalived:实现VIP高可用;
- Nginx:以Pod形式运行与Kubernetes之上,即in Kubernetes模式,提供反向代理至3个master 6443端口;
- 其他主要部署组件包括:
- Metrics:度量;
- Dashboard:Kubernetes 图形UI界面;
- Helm:Kubernetes Helm包管理工具;
- Ingress:Kubernetes 服务暴露;
- Longhorn:Kubernetes 动态存储组件。
部署规划
节点规划
节点主机名 | IP | 类型 | 运行服务 |
---|---|---|---|
master01 | 172.24.8.71 | Kubernetes master节点 | docker、etcd、kube-apiserver、kube-scheduler、kube-controller-manager、kubectl、kubelet、metrics、calico |
master02 | 172.24.8.72 | Kubernetes master节点 | docker、etcd、kube-apiserver、kube-scheduler、kube-controller-manager、kubectl、kubelet、metrics、calico |
master03 | 172.24.8.73 | Kubernetes master节点 | docker、etcd、kube-apiserver、kube-scheduler、kube-controller-manager、kubectl、kubelet、metrics、calico |
worker01 | 172.24.8.74 | Kubernetes worker节点 | docker、kubelet、proxy、calico |
worker02 | 172.24.8.75 | Kubernetes worker节点 | docker、kubelet、proxy、calico |
worker03 | 172.24.8.76 | Kubernetes worker节点 | docker、kubelet、proxy、calico |
Kubernetes的高可用主要指的是控制平面的高可用,即指多套Master节点组件和Etcd组件,工作节点通过负载均衡连接到各Master。
Kubernetes高可用架构中etcd与Master节点组件混布方式特点:
- Etcd混布方式
- 所需机器资源少
- 部署简单,利于管理
- 容易进行横向扩展
- 风险大,一台宿主机挂了,master和etcd就都少了一套,集群冗余度受到的影响比较大。
初始准备
[root@master01 ~]# hostnamectl set-hostname master01 #其他节点依次修改
[root@master01 ~]# cat >> /etc/hosts << EOF
172.24.8.71 master01··
172.24.8.72 master02
172.24.8.73 master03
172.24.8.74 worker01
172.24.8.75 worker02
172.24.8.76 worker03
EOF
[root@master01 ~]# vi k8sinit.sh
#!/bin/sh
#****************************************************************#
# ScriptName: k8sinit.sh
# Author: xhy
# Create Date: 2020-05-30 16:30
# Modify Author: xhy
# Modify Date: 2020-05-30 16:30
# Version:
#***************************************************************#
# Initialize the machine. This needs to be executed on every machine.
# Add docker user
useradd -m docker
# Disable the SELinux.
sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config
# Turn off and disable the firewalld.
systemctl stop firewalld
systemctl disable firewalld
# Modify related kernel parameters & Disable the swap.
cat > /etc/sysctl.d/k8s.conf << EOF
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.tcp_tw_recycle = 0
vm.swappiness = 0
vm.overcommit_memory = 1
vm.panic_on_oom = 0
net.ipv6.conf.all.disable_ipv6 = 1
EOF
sysctl -p /etc/sysctl.d/k8s.conf >&/dev/null
swapoff -a
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
modprobe br_netfilter
# Add ipvs modules
cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
modprobe -- nf_conntrack
EOF
chmod 755 /etc/sysconfig/modules/ipvs.modules
bash /etc/sysconfig/modules/ipvs.modules
# Install rpm
yum install -y conntrack ntpdate ntp ipvsadm ipset jq iptables curl sysstat libseccomp wget
# Install Docker Compose
sudo curl -L "//down.linuxsb.com:8888/docker/compose/releases/download/1.25.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
# Update kernel
rpm --import //down.linuxsb.com:8888/RPM-GPG-KEY-elrepo.org
rpm -Uvh //down.linuxsb.com:8888/elrepo-release-7.0-4.el7.elrepo.noarch.rpm
yum --disablerepo="*" --enablerepo="elrepo-kernel" install -y kernel-ml
sed -i 's/^GRUB_DEFAULT=.*/GRUB_DEFAULT=0/' /etc/default/grub
grub2-mkconfig -o /boot/grub2/grub.cfg
yum update -y
# Reboot the machine.
# reboot
提示:对于某些特性,可能需要升级内核,内核升级操作见《018.Linux升级内核》。4.19版及以上内核nf_conntrack_ipv4已经改为nf_conntrack。
互信配置
为了更方便远程分发文件和执行命令,本实验配置master节点到其它节点的 ssh 信任关系。
[root@master01 ~]# ssh-keygen -f ~/.ssh/id_rsa -N ''
[root@master01 ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@master01
[root@master01 ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@master02
[root@master01 ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@master03
[root@master01 ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@worker01
[root@master01 ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@worker02
[root@master01 ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@worker03
提示:此操作仅需要在master节点操作。
其他准备
[root@master01 ~]# vi environment.sh
#!/bin/sh
#****************************************************************#
# ScriptName: environment.sh
# Author: xhy
# Create Date: 2020-05-30 16:30
# Modify Author: xhy
# Modify Date: 2020-05-30 16:30
# Version:
#***************************************************************#
# 集群 MASTER 机器 IP 数组
export MASTER_IPS=(172.24.8.71 172.24.8.72 172.24.8.73)
# 集群 MASTER IP 对应的主机名数组
export MASTER_NAMES=(master01 master02 master03)
# 集群 NODE 机器 IP 数组
export NODE_IPS=(172.24.8.74 172.24.8.75 172.24.8.76)
# 集群 NODE IP 对应的主机名数组
export NODE_NAMES=(worker01 worker02 worker03)
# 集群所有机器 IP 数组
export ALL_IPS=(172.24.8.71 172.24.8.72 172.24.8.73 172.24.8.74 172.24.8.75 172.24.8.76)
# 集群所有IP 对应的主机名数组
export ALL_NAMES=(master01 master02 master03 worker01 worker02 worker03)
[root@master01 ~]# source environment.sh
[root@master01 ~]# chmod +x *.sh
[root@master01 ~]# for all_ip in ${ALL_IPS[@]}
do
echo ">>> ${all_ip}"
scp -rp /etc/hosts root@${all_ip}:/etc/hosts
scp -rp k8sinit.sh root@${all_ip}:/root/
ssh root@${all_ip} "bash /root/k8sinit.sh"
done
集群部署
Docker安装
[root@master01 ~]# for all_ip in ${ALL_IPS[@]}
do
echo ">>> ${all_ip}"
ssh root@${all_ip} "yum -y install yum-utils device-mapper-persistent-data lvm2"
ssh root@${all_ip} "yum-config-manager --add-repo //mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo"
ssh root@${all_ip} "yum -y install docker-ce"
ssh root@${all_ip} "mkdir /etc/docker"
ssh root@${all_ip} "cat > /etc/docker/daemon.json <<EOF
{
\"registry-mirrors\": [\"//dbzucv6w.mirror.aliyuncs.com\"],
\"exec-opts\": [\"native.cgroupdriver=systemd\"],
\"log-driver\": \"json-file\",
\"log-opts\": {
\"max-size\": \"100m\"
},
\"storage-driver\": \"overlay2\",
\"storage-opts\": [
\"overlay2.override_kernel_check=true\"
]
}
EOF"
ssh root@${all_ip} "systemctl restart docker"
ssh root@${all_ip} "systemctl enable docker"
ssh root@${all_ip} "systemctl status docker"
ssh root@${all_ip} "iptables -nvL"
done
提示:如上仅需Master01节点操作,从而实现所有节点自动化安装。
相关组件包
需要在每台机器上都安装以下的软件包:
- kubeadm: 用来初始化集群的指令;
- kubelet: 在集群中的每个节点上用来启动 pod 和 container 等;
- kubectl: 用来与集群通信的命令行工具。
kubeadm不能安装或管理 kubelet 或 kubectl ,所以得保证他们满足通过 kubeadm 安装的 Kubernetes控制层对版本的要求。如果版本没有满足要求,可能导致一些意外错误或问题。
具体相关组件安装见;附001.kubectl介绍及使用书
提示:Kubernetes 1.18版本所有兼容相应组件的版本参考://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.18.md。
正式安装
[root@master01 ~]# for all_ip in ${ALL_IPS[@]}
do
echo ">>> ${all_ip}"
ssh root@${all_ip} "cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=//mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=//mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg //mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF"
ssh root@${all_ip} "yum install -y kubeadm-1.18.3-0.x86_64 kubelet-1.18.3-0.x86_64 kubectl-1.18.3-0.x86_64 --disableexcludes=kubernetes"
ssh root@${all_ip} "systemctl enable kubelet"
done
[root@master01 ~]# yum search -y kubelet --showduplicates #查看相应版本
提示:如上仅需Master01节点操作,从而实现所有节点自动化安装,同时此时不需要启动kubelet,初始化的过程中会自动启动的,如果此时启动了会出现报错,忽略即可。
说明:同时安装了cri-tools, kubernetes-cni, socat三个依赖:
socat:kubelet的依赖;
cri-tools:即CRI(Container Runtime Interface)容器运行时接口的命令行工具。
部署高可用组件I
Keepalived安装
[root@master01 ~]# for master_ip in ${MASTER_IPS[@]}
do
echo ">>> ${master_ip}"
ssh root@${master_ip} "yum -y install gcc gcc-c++ make libnl libnl-devel libnfnetlink-devel openssl-devel"
ssh root@${master_ip} "wget //down.linuxsb.com:8888/software/keepalived-2.0.20.tar.gz"
ssh root@${master_ip} "tar -zxvf keepalived-2.0.20.tar.gz"
ssh root@${master_ip} "cd keepalived-2.0.20/ && ./configure --sysconf=/etc --prefix=/usr/local/keepalived && make && make install"
ssh root@${master_ip} "systemctl enable keepalived && systemctl start keepalived"
done
提示:如上仅需Master01节点操作,从而实现所有节点自动化安装。
创建配置文件
[root@master01 ~]# wget //down.linuxsb.com:8888/ngkek8s.sh #拉取自动部署脚本
[root@master01 ~]# chmod u+x ngkek8s.sh
[root@master01 ~]# vi ngkek8s.sh
#!/bin/sh
#****************************************************************#
# ScriptName: k8s_ha.sh
# Author: xhy
# Create Date: 2020-05-13 16:32
# Modify Author: xhy
# Modify Date: 2020-06-12 12:53
# Version: v2
#***************************************************************#
#######################################
# set variables below to create the config files, all files will create at ./config directory
#######################################
# master keepalived virtual ip address
export K8SHA_VIP=172.24.8.100
# master01 ip address
export K8SHA_IP1=172.24.8.71
# master02 ip address
export K8SHA_IP2=172.24.8.72
# master03 ip address
export K8SHA_IP3=172.24.8.73
# master01 hostname
export K8SHA_HOST1=master01
# master02 hostname
export K8SHA_HOST2=master02
# master03 hostname
export K8SHA_HOST3=master03
# master01 network interface name
export K8SHA_NETINF1=eth0
# master02 network interface name
export K8SHA_NETINF2=eth0
# master03 network interface name
export K8SHA_NETINF3=eth0
# keepalived auth_pass config
export K8SHA_KEEPALIVED_AUTH=412f7dc3bfed32194d1600c483e10ad1d
# kubernetes CIDR pod subnet
export K8SHA_PODCIDR=10.10.0.0
# kubernetes CIDR svc subnet
export K8SHA_SVCCIDR=10.20.0.0
[root@master01 ~]# ./ngkek8s.sh
解释:如上仅需Master01节点操作。执行ngkek8s.sh脚本后,会自动生成以下配置文件:
- kubeadm-config.yaml:kubeadm初始化配置文件,位于当前目录
- keepalived:keepalived配置文件,位于各个master节点的/etc/keepalived目录
- nginx-lb:nginx-lb负载均衡配置文件,位于各个master节点的/etc/kubernetes/nginx-lb/目录
- calico.yaml:calico网络组件部署文件,位于config/calico/目录
[root@master01 ~]# cat kubeadm-config.yaml #检查集群初始化配置
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
networking:
serviceSubnet: "10.20.0.0/16" #设置svc网段
podSubnet: "10.10.0.0/16" #设置Pod网段
dnsDomain: "cluster.local"
kubernetesVersion: "v1.18.3" #设置安装版本
controlPlaneEndpoint: "172.24.8.100:16443" #设置相关API VIP地址
apiServer:
certSANs:
- master01
- master02
- master03
- 127.0.0.1
- 172.24.8.71
- 172.24.8.72
- 172.24.8.73
- 172.24.8.100
timeoutForControlPlane: 4m0s
certificatesDir: "/etc/kubernetes/pki"
imageRepository: "k8s.gcr.io"
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
featureGates:
SupportIPVSProxyMode: true
mode: ipvs
提示:如上仅需Master01节点操作,更多config文件参考://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2。
此kubeadm部署初始化配置更多参考://pkg.go.dev/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2?tab=doc。
启动Keepalived
[root@master01 ~]# cat /etc/keepalived/check_apiserver.sh #确认Keepalived配置
[root@master01 ~]# for master_ip in ${MASTER_IPS[@]}
do
echo ">>> ${master_ip}"
ssh root@${master_ip} "systemctl start keepalived.service && systemctl enable keepalived.service"
ssh root@${master_ip} "systemctl status keepalived.service | grep Active"
done
[root@master01 ~]# for all_ip in ${ALL_IPS[@]}
do
echo ">>> ${all_ip}"
ssh root@${all_ip} "ping -c1 172.24.8.100"
done #等待10s左右执行检查
提示:如上仅需Master01节点操作,从而实现所有节点自动启动服务。
启动Nginx
执行ngkek8s.sh脚本后,nginx-lb的配置文件会自动复制到各个master的节点的/etc/kubernetes/nginx-lb目录。
[root@master01 ~]# for master_ip in ${MASTER_IPS[@]}
do
echo ">>> ${master_ip}"
ssh root@${master_ip} "cd /etc/kubernetes/nginx-lb/ && docker-compose up -d"
ssh root@${master_ip} "docker-compose ps"
done
提示:如上仅需Master01节点操作,从而实现所有节点自动启动服务。
初始化集群-Master
拉取镜像
[root@master01 ~]# kubeadm --kubernetes-version=v1.18.3 config images list #列出所需镜像
[root@master01 ~]# cat config/downimage.sh #确认版本,提前下载镜像
#!/bin/sh
#****************************************************************#
# ScriptName: downimage.sh
# Author: xhy
# Create Date: 2020-05-29 19:55
# Modify Author: xhy
# Modify Date: 2020-05-30 16:07
# Version: v2
#***************************************************************#
KUBE_VERSION=v1.18.3
CALICO_VERSION=v3.14.1
CALICO_URL=calico
KUBE_PAUSE_VERSION=3.2
ETCD_VERSION=3.4.3-0
CORE_DNS_VERSION=1.6.7
GCR_URL=k8s.gcr.io
METRICS_SERVER_VERSION=v0.3.6
INGRESS_VERSION=0.32.0
CSI_PROVISIONER_VERSION=v1.4.0
CSI_NODE_DRIVER_VERSION=v1.2.0
CSI_ATTACHER_VERSION=v2.0.0
CSI_RESIZER_VERSION=v0.3.0
ALIYUN_URL=registry.cn-hangzhou.aliyuncs.com/google_containers
UCLOUD_URL=uhub.service.ucloud.cn/uxhy
QUAY_URL=quay.io
kubeimages=(kube-proxy:${KUBE_VERSION}
kube-scheduler:${KUBE_VERSION}
kube-controller-manager:${KUBE_VERSION}
kube-apiserver:${KUBE_VERSION}
pause:${KUBE_PAUSE_VERSION}
etcd:${ETCD_VERSION}
coredns:${CORE_DNS_VERSION}
metrics-server-amd64:${METRICS_SERVER_VERSION}
)
for kubeimageName in ${kubeimages[@]} ; do
docker pull $UCLOUD_URL/$kubeimageName
docker tag $UCLOUD_URL/$kubeimageName $GCR_URL/$kubeimageName
docker rmi $UCLOUD_URL/$kubeimageName
done
calimages=(cni:${CALICO_VERSION}
pod2daemon-flexvol:${CALICO_VERSION}
node:${CALICO_VERSION}
kube-controllers:${CALICO_VERSION})
for calimageName in ${calimages[@]} ; do
docker pull $UCLOUD_URL/$calimageName
docker tag $UCLOUD_URL/$calimageName $CALICO_URL/$calimageName
docker rmi $UCLOUD_URL/$calimageName
done
ingressimages=(nginx-ingress-controller:${INGRESS_VERSION})
for ingressimageName in ${ingressimages[@]} ; do
docker pull $UCLOUD_URL/$ingressimageName
docker tag $UCLOUD_URL/$ingressimageName $QUAY_URL/kubernetes-ingress-controller/$ingressimageName
docker rmi $UCLOUD_URL/$ingressimageName
done
csiimages=(csi-provisioner:${CSI_PROVISIONER_VERSION}
csi-node-driver-registrar:${CSI_NODE_DRIVER_VERSION}
csi-attacher:${CSI_ATTACHER_VERSION}
csi-resizer:${CSI_RESIZER_VERSION}
)
for csiimageName in ${csiimages[@]} ; do
docker pull $UCLOUD_URL/$csiimageName
docker tag $UCLOUD_URL/$csiimageName $QUAY_URL/k8scsi/$csiimageName
docker rmi $UCLOUD_URL/$csiimageName
done
[root@master01 ~]# for all_ip in ${ALL_IPS[@]}
do
echo ">>> ${all_ip}"
scp -rp config/downimage.sh root@${all_ip}:/root/
ssh root@${all_ip} "bash downimage.sh &"
done
提示:如上仅需Master01节点操作,从而实现所有节点自动拉取镜像。
[root@master01 ~]# docker images #确认验证
Master上初始化
[root@master01 ~]# kubeadm init --config=kubeadm-config.yaml --upload-certs
保留如下命令用于后续节点添加:
You can now join any number of the control-plane node running the following command on each as root:
kubeadm join 172.24.8.100:16443 --token xb9wda.v0yf7tlsgo8mdrhk \
--discovery-token-ca-cert-hash sha256:249884d81a23bd821e38d3345866a99e6d55e443b545825c3c448f30f8e52c3b \
--control-plane --certificate-key e30428776a47ed2c7e18c9e2951d9e40e068c9ecec5a4858457f1475f1a2a39a
Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 172.24.8.100:16443 --token xb9wda.v0yf7tlsgo8mdrhk \
--discovery-token-ca-cert-hash sha256:249884d81a23bd821e38d3345866a99e6d55e443b545825c3c448f30f8e52c3b
注意:如上token具有默认24小时的有效期,token和hash值可通过如下方式获取:
kubeadm token list
如果 Token 过期以后,可以输入以下命令,生成新的 Token:
kubeadm token create
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'***
[root@master01 ~]# mkdir -p $HOME/.kube
[root@master01 ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@master01 ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config
[root@master01 ~]# cat << EOF >> ~/.bashrc
export KUBECONFIG=$HOME/.kube/config
EOF #设置KUBECONFIG环境变量
[root@master01 ~]# echo "source <(kubectl completion bash)" >> ~/.bashrc
[root@master01 ~]# source ~/.bashrc
附加:初始化过程大致步骤如下:
- [kubelet-start] 生成kubelet的配置文件”/var/lib/kubelet/config.yaml”
- [certificates]生成相关的各种证书
- [kubeconfig]生成相关的kubeconfig文件
- [bootstraptoken]生成token记录下来,后边使用kubeadm join往集群中添加节点时会用到
提示:初始化仅需要在master01上执行,若初始化异常可通过kubeadm reset && rm -rf $HOME/.kube重置。
添加其他master节点
[root@master02 ~]# kubeadm join 172.24.8.100:16443 --token xb9wda.v0yf7tlsgo8mdrhk \
--discovery-token-ca-cert-hash sha256:249884d81a23bd821e38d3345866a99e6d55e443b545825c3c448f30f8e52c3b \
--control-plane --certificate-key e30428776a47ed2c7e18c9e2951d9e40e068c9ecec5a4858457f1475f1a2a39a
[root@master02 ~]# mkdir -p $HOME/.kube
[root@master02 ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@master02 ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config
[root@master02 ~]# cat << EOF >> ~/.bashrc`
export KUBECONFIG=$HOME/.kube/config
EOF #设置KUBECONFIG环境变量
[root@master02 ~]# echo "source <(kubectl completion bash)" >> ~/.bashrc
[root@master02 ~]# source ~/.bashrc
提示:master03也如上执行添加至集群的controlplane。
提示:若添加异常可通过kubeadm reset && rm -rf $HOME/.kube重置。