附024.Kubernetes_v1.18.3高可用部署架構二
- 2020 年 6 月 16 日
- 筆記
- dashboard介紹, HAProxy, helm, Ingress, keepalived, kubeadm, Kubernetes, Longhorn, metrics, 集群相關, 高可用
附024.Kubernetes_v1.18.3高可用部署架構二
kubeadm介紹
kubeadm概述
kubeadm功能
本方案描述
- 本方案採用kubeadm部署Kubernetes 1.18.3版本;
- etcd採用混部方式;
- KeepAlived:實現VIP高可用;
- HAProxy:以系統systemd形式運行,提供反向代理至3個master 6443埠;
- 其他主要部署組件包括:
- Metrics:度量;
- Dashboard:Kubernetes 圖形UI介面;
- Helm:Kubernetes Helm包管理工具;
- Ingress:Kubernetes 服務暴露;
- Longhorn:Kubernetes 動態存儲組件。
部署規劃
節點規劃
節點主機名 | IP | 類型 | 運行服務 |
---|---|---|---|
master01 | 172.24.8.71 | Kubernetes master節點 | docker、etcd、kube-apiserver、kube-scheduler、kube-controller-manager、 kubectl、kubelet、metrics、calico、HAProxy、KeepAlived |
master02 | 172.24.8.72 | Kubernetes master節點 | docker、etcd、kube-apiserver、kube-scheduler、kube-controller-manager、 kubectl、kubelet、metrics、calico、HAProxy、KeepAlived |
master03 | 172.24.8.73 | Kubernetes master節點 | docker、etcd、kube-apiserver、kube-scheduler、kube-controller-manager、 kubectl、kubelet、metrics、calico、HAProxy、KeepAlived |
worker01 | 172.24.8.74 | Kubernetes node節點1 | docker、kubelet、proxy、calico |
worker02 | 172.24.8.75 | Kubernetes node節點2 | docker、kubelet、proxy、calico |
worker03 | 172.24.8.76 | Kubernetes node節點3 | docker、kubelet、proxy、calico |
VIP | 172.24.8.100 |
Kubernetes的高可用主要指的是控制平面的高可用,即指多套Master節點組件和Etcd組件,工作節點通過負載均衡連接到各Master。
Kubernetes高可用架構中etcd與Master節點組件混布方式特點:
- 所需機器資源少
- 部署簡單,利於管理
- 容易進行橫向擴展
- 風險大,一台宿主機掛了,master和etcd就都少了一套,集群冗餘度受到的影響比較大。
提示:本實驗使用高可用架構一實現Kubernetes的高可用。
初始準備
[root@master01 ~]# hostnamectl set-hostname master01 #其他節點依次修改
[root@master01 ~]# cat >> /etc/hosts << EOF
172.24.8.71 master01
172.24.8.72 master02
172.24.8.73 master03
172.24.8.74 worker01
172.24.8.75 worker02
172.24.8.76 worker03
EOF
[root@master01 ~]# vi k8sinit.sh
# Initialize the machine. This needs to be executed on every machine.
# Install docker
useradd -m docker
yum -y install yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo //mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum -y install docker-ce
mkdir /etc/docker
cat > /etc/docker/daemon.json <<EOF
{
"registry-mirrors": ["//dbzucv6w.mirror.aliyuncs.com"],
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2",
"storage-opts": [
"overlay2.override_kernel_check=true"
]
}
EOF
systemctl restart docker
systemctl enable docker
systemctl status docker
# Disable the SELinux.
sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config
# Turn off and disable the firewalld.
systemctl stop firewalld
systemctl disable firewalld
# Modify related kernel parameters & Disable the swap.
cat > /etc/sysctl.d/k8s.conf << EOF
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.tcp_tw_recycle = 0
vm.swappiness = 0
vm.overcommit_memory = 1
vm.panic_on_oom = 0
net.ipv6.conf.all.disable_ipv6 = 1
EOF
sysctl -p /etc/sysctl.d/k8s.conf >&/dev/null
swapoff -a
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
modprobe br_netfilter
# Add ipvs modules
cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
modprobe -- nf_conntrack
EOF
chmod 755 /etc/sysconfig/modules/ipvs.modules
bash /etc/sysconfig/modules/ipvs.modules
# Install rpm
yum install -y conntrack ntpdate ntp ipvsadm ipset jq iptables curl sysstat libseccomp wget
# Update kernel
rpm --import //down.linuxsb.com:8888/RPM-GPG-KEY-elrepo.org
rpm -Uvh //down.linuxsb.com:8888/elrepo-release-7.0-4.el7.elrepo.noarch.rpm
yum --disablerepo="*" --enablerepo="elrepo-kernel" install -y kernel-ml
sed -i 's/^GRUB_DEFAULT=.*/GRUB_DEFAULT=0/' /etc/default/grub
grub2-mkconfig -o /boot/grub2/grub.cfg
yum update -y
# Reboot the machine.
# reboot
提示:對於某些特性,可能需要升級內核,內核升級操作見《018.Linux升級內核》。
4.19版及以上內核nf_conntrack_ipv4已經改為nf_conntrack。**
互信配置
為了更方便遠程分發文件和執行命令,本實驗配置master01節點到其它節點的 ssh 信任關係。
[root@master01 ~]# ssh-keygen -f ~/.ssh/id_rsa -N ''
[root@master01 ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@master01
[root@master01 ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@master02
[root@master01 ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@master03
[root@master01 ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@worker01
[root@master01 ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@worker02
[root@master01 ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@worker03
其他準備
[root@master01 ~]# vi environment.sh
#!/bin/sh
#****************************************************************#
# ScriptName: environment.sh
# Author: xhy
# Create Date: 2020-05-30 16:30
# Modify Author: xhy
# Modify Date: 2020-06-15 17:55
# Version:
#***************************************************************#
# 集群 MASTER 機器 IP 數組
export MASTER_IPS=(172.24.8.71 172.24.8.72 172.24.8.73)
# 集群 MASTER IP 對應的主機名數組
export MASTER_NAMES=(master01 master02 master03)
# 集群 NODE 機器 IP 數組
export NODE_IPS=(172.24.8.74 172.24.8.75 172.24.8.76)
# 集群 NODE IP 對應的主機名數組
export NODE_NAMES=(worker01 worker02 worker03)
# 集群所有機器 IP 數組
export ALL_IPS=(172.24.8.71 172.24.8.72 172.24.8.73 172.24.8.74 172.24.8.75 172.24.8.76)
# 集群所有IP 對應的主機名數組
export ALL_NAMES=(master01 master02 master03 worker01 worker02 worker03)
[root@master01 ~]# source environment.sh
[root@master01 ~]# chmod +x *.sh
[root@master01 ~]# for all_ip in ${ALL_IPS[@]}
do
echo ">>> ${all_ip}"
scp -rp /etc/hosts root@${all_ip}:/etc/hosts
scp -rp k8sinit.sh root@${all_ip}:/root/
ssh root@${all_ip} "bash /root/k8sinit.sh"
done
集群部署
相關組件包
需要在每台機器上都安裝以下的軟體包:
- kubeadm: 用來初始化集群的指令;
- kubelet: 在集群中的每個節點上用來啟動 pod 和 container 等;
- kubectl: 用來與集群通訊的命令行工具。
kubeadm不能安裝或管理 kubelet 或 kubectl ,所以得保證他們滿足通過 kubeadm 安裝的 Kubernetes 控制層對版本的要求。如果版本沒有滿足要求,可能導致一些意外錯誤或問題。
具體相關組件安裝見《附001.kubectl介紹及使用》。
提示:Kubernetes 1.18版本所有兼容相應組件的版本參考://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.18.md。
正式安裝
[root@master01 ~]# for all_ip in ${ALL_IPS[@]}
do
echo ">>> ${all_ip}"
ssh root@${all_ip} "cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=//mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=//mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg //mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF"
ssh root@${all_ip} "yum install -y kubeadm-1.18.3-0.x86_64 kubelet-1.18.3-0.x86_64 kubectl-1.18.3-0.x86_64 --disableexcludes=kubernetes"
ssh root@${all_ip} "systemctl enable kubelet"
done
[root@master01 ~]# yum search -y kubelet --showduplicates #查看相應版本
提示:如上僅需Master01節點操作,從而實現所有節點自動化安裝,同時此時不需要啟動kubelet,初始化的過程中會自動啟動的,如果此時啟動了會出現報錯,忽略即可。
說明:同時安裝了cri-tools, kubernetes-cni, socat三個依賴:
socat:kubelet的依賴;
cri-tools:即CRI(Container Runtime Interface)容器運行時介面的命令行工具。
部署高可用組件
HAProxy安裝
[root@master01 ~]# for master_ip in ${MASTER_IPS[@]}
do
echo ">>> ${master_ip}"
ssh root@${master_ip} "yum -y install gcc gcc-c++ make libnl libnl-devel libnfnetlink-devel openssl-devel wget openssh-clients systemd-devel zlib-devel pcre-devel libnl3-devel"
ssh root@${master_ip} "wget //down.linuxsb.com:8888/software/haproxy-2.1.6.tar.gz"
ssh root@${master_ip} "tar -zxvf haproxy-2.1.6.tar.gz"
ssh root@${master_ip} "cd haproxy-2.1.6/ && make ARCH=x86_64 TARGET=linux-glibc USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 USE_SYSTEMD=1 PREFIX=/usr/local/haprpxy && make install PREFIX=/usr/local/haproxy"
ssh root@${master_ip} "cp /usr/local/haproxy/sbin/haproxy /usr/sbin/"
ssh root@${master_ip} "useradd -r haproxy && usermod -G haproxy haproxy"
ssh root@${master_ip} "mkdir -p /etc/haproxy && cp -r /root/haproxy-2.1.6/examples/errorfiles/ /usr/local/haproxy/"
done
Keepalived安裝
[root@master01 ~]# for master_ip in ${MASTER_IPS[@]}
do
echo ">>> ${master_ip}"
ssh root@${master_ip} "yum -y install gcc gcc-c++ make libnl libnl-devel libnfnetlink-devel openssl-devel"
ssh root@${master_ip} "wget //down.linuxsb.com:8888/software/keepalived-2.0.20.tar.gz"
ssh root@${master_ip} "tar -zxvf keepalived-2.0.20.tar.gz"
ssh root@${master_ip} "cd keepalived-2.0.20/ && ./configure --sysconf=/etc --prefix=/usr/local/keepalived && make && make install"
done
提示:如上僅需Master01節點操作,從而實現所有節點自動化安裝。
創建配置文件
[root@master01 ~]# wget //down.linuxsb.com:8888/hakek8s.sh #拉取自動部署腳本
[root@master01 ~]# chmod u+x hakek8s.sh
[root@master01 ~]# vi hakek8s.sh
#!/bin/sh
#****************************************************************#
# ScriptName: hakek8s.sh
# Author: xhy
# Create Date: 2020-06-08 20:00
# Modify Author: xhy
# Modify Date: 2020-06-15 18:15
# Version: v2
#***************************************************************#
#######################################
# set variables below to create the config files, all files will create at ./config directory
#######################################
# master keepalived virtual ip address
export K8SHA_VIP=172.24.8.100
# master01 ip address
export K8SHA_IP1=172.24.8.71
# master02 ip address
export K8SHA_IP2=172.24.8.72
# master03 ip address
export K8SHA_IP3=172.24.8.73
# master01 hostname
export K8SHA_HOST1=master01
# master02 hostname
export K8SHA_HOST2=master02
# master03 hostname
export K8SHA_HOST3=master03
# master01 network interface name
export K8SHA_NETINF1=eth0
# master02 network interface name
export K8SHA_NETINF2=eth0
# master03 network interface name
export K8SHA_NETINF3=eth0
# keepalived auth_pass config
export K8SHA_KEEPALIVED_AUTH=412f7dc3bfed32194d1600c483e10ad1d
# kubernetes CIDR pod subnet
export K8SHA_PODCIDR=10.10.0.0
# kubernetes CIDR svc subnet
export K8SHA_SVCCIDR=10.20.0.0
[root@master01 ~]# ./hakek8s.sh
解釋:如上僅需Master01節點操作。執行hakek8s.sh腳本後會生產如下配置文件清單:
- kubeadm-config.yaml:kubeadm初始化配置文件,位於當前目錄
- keepalived:keepalived配置文件,位於各個master節點的/etc/keepalived目錄
- haproxy:haproxy的配置文件,位於各個master節點的/etc/haproxy/目錄
- calico.yaml:calico網路組件部署文件,位於config/calico/目錄