【k8s實戰一】Jenkins 部署應用到 Kubernetes

01 本文主旨

目標是演示整個Jenkins從源碼構建鏡像到部署鏡像到Kubernetes集群過程。

為了簡化流程與容易重現文中效果，做出如下操作：

使用 VBox 創建兩台虛擬機
使用 Minikube 初始化簡單的k8s集群
使用 GitHub 作為程式碼倉庫，本文內容所有配置文件均會上傳至此倉庫 //github.com/hellxz/cicd-demo.git

02 CI/CD流程

根據我畫的這張圖，Jenkins任務整體的流程分為五個步驟：

獲取源碼
構建製品
構建鏡像並推送
發起部署請求
k8s調度pod，拉取鏡像，部署完成

03 準備虛擬機

準備兩個虛擬機，均安裝好docker，參考Ubuntu 16.04及以上安裝/卸載 Docker-CE

VM用途	系統版本	IP
Jenkins	Ubuntu 18.04.2 LTS	192.168.87.128
k8s master	Ubuntu 18.04.2 LTS	192.168.87.129

設置Jenkins機器免密登錄k8s master主機

ssh-keygen -t rsa
ssh-copy-id [email protected]

驗證免密效果

ssh [email protected]
exit

04 部署registry

部署registry之前，可以先將私有倉庫配置到 /etc/docker/daemon.json，兩台虛擬機均需要配置

sudo mkdir -p /etc/docker
sudo vim /etc/docker/daemon.json

內容如下：

{
    "insecure-registries": ["192.168.87.129:5000"]
}

接著在 192.168.87.129 虛擬機上部署registry，生產環境推薦使用Harbor

docker run -d --name registry -p 5000:5000 registry

05 配置maven

在jenkins虛擬機安裝maven

wget //mirrors.aliyun.com/apache/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz
tar zxvf apache-maven-3.6.3-bin.tar.gz
mv apache-maven-3.6.3 maven-3.6.3

修改 /etc/profile，添加 M2_HOME 環境變數映射到 maven 目錄，重寫 PATH 添加 maven 的 bin 目錄，如圖31、32行

06 Tomcat部署Jenkins

在jenkins虛擬機 192.168.87.128上執行命令下載 tomcat 與 jenkins.war

#下載tomcat
wget //mirrors.tuna.tsinghua.edu.cn/apache/tomcat/tomcat-9/v9.0.41/bin/apache-tomcat-9.0.41.tar.gz
#解壓並重命名
tar -zxf apache-tomcat-9.0.41.tar.gz
mv apache-tomcat-9.0.41 /home/hellxz/jenkins
#進入jenkins安裝目錄
cd ~/jenkins/webapps
#刪除webapps下示例項目
rm -rf *
#下載最新版jenkins.war
wget //mirrors.ustc.edu.cn/jenkins/war/latest/jenkins.war
#安裝openjdk、git
sudo apt-update
sudo apt-get install -y default-jdk default-jdk-headless git

設置Jenkins家目錄，編輯 /etc/profile，添加 JENKINS_HOME 環境變數

刷新配置

source /etc/profile

修改tomcat下 conf/context.xml，加大靜態資源快取，提前解決啟動 Jenkins 時的報錯

<Resources cachingAllowed="true" cacheMaxSize="9999999" />

啟動Jenkins

cd ~/jenkins/bin
./startup.sh

訪問 //192.168.87.129:8080，安裝推薦插件，安裝加速可參考 Jenkins安裝插件提速

07 部署Kubernetes集群

部署一個可以通過 kubectl 命令部署應用的k8s環境即可，部署k8s環境不是本文重點，以下為參考

使用kubeadm部署Kubernetes集群，點擊展開查看

參考我之前的部落格吧，差異應該不大
【k8s學習筆記】使用 kubeadm 部署 v1.18.5 版本 Kubernetes集群

使用Minikube部署單節點測試k8s環境，點擊展開查看


#添加阿里apt kubernetes源
sudo apt-get update && sudo apt-get install -y apt-transport-https
curl //mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | sudo apt-key add - 
sudo tee /etc/apt/sources.list.d/kubernetes.list <<-'EOF'
deb //mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF
sudo apt-get update
#安裝kubectl
sudo apt-get install -y kubectl
#下載minikube二進位文件
curl -Lo minikube //kubernetes.oss-cn-hangzhou.aliyuncs.com/minikube/releases/v1.16.0/minikube-linux-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/
#部署k8s環境，默認使用docker驅動。讀者也可使用vbox
minikube start -–insecure-registry=192.168.87.129:5000

本文使用Minikube做測試，坑比較多，想趟坑可以先參考下 //www.kubernetes.org.cn/6388.html

關於Minikube的吐槽

寫文時，用的Win10創建Linux無桌面虛擬機，在虛擬機基礎上裝的minikube，minikube默認使用Docker在虛擬機中又創建了個Docker容器，裡邊才是k8s環境，所以想在虛擬機外訪問套了兩層的內網應用服務，可能還需要在虛擬機中加Nginx，通過它轉發請求到虛擬機中的內網地址……雖然瘋狂套娃挺坑的，但為了演示下效果還是可以的。
PS：在Linux/MacOS環境直接裝Minikube，是可以通過瀏覽器直接看的。由於未直接在Win環境使用Docker Desktop，所以未確定Win下是否可用。

08 準備Demo

Demo分五部分：測試程式碼、Jenkins構建腳本、docker鏡像構建腳本、k8s部署配置yaml、部署腳本，均處在同一倉庫中

倉庫地址：//github.com/hellxz/cicd-demo.git

目錄結構：

demo

使用start.spring.io生成的只添加spring-boot-starter-web依賴，在啟動類上加了個 /test 的介面

Dockerfile

FROM openjdk:8-jdk
#設置工作目錄
WORKDIR /root
#複製製品jar到/root/app.jar位置
ADD target/*.jar app.jar
#JVM參數，給後期調優使用
ENV JVM_OPTS=""
#啟動服務
CMD java ${JVM_OPTS} \
    -Djava.security.egd=file:/dev/./urandom \
    -jar app.jar

shell

artifact2image.sh 製品轉鏡像Shell腳本，本文為了簡單未安裝Jenkins關於docker的插件

#!/bin/bash
# -- 構建鏡像並推送私有倉庫 --
set -eu #如有報錯或取不到變數情況停止執行

#聲名常量
IMG_REGISTRY="192.168.87.129:5000" #鏡像倉庫
IMG_NAME="cicd-demo"
IMG_TAG=`date "+%Y%M%d_%H%M"`      #鏡像標籤，如 20201223_1351
IMG_FULL_NAME="${IMG_REGISTRY}/${IMG_NAME}:${IMG_TAG}" #鏡像上傳與拉取的名稱

#構建鏡像
docker build -t ${IMG_FULL_NAME} .

#推送鏡像
docker push ${IMG_FULL_NAME}

#刪除本地鏡像
docker rmi ${IMG_FULL_NAME}

#修改deploy.yaml的鏡像標籤
sed -i "s#{{IMAGE_NAME}}#${IMG_FULL_NAME}#g" deploy.yaml

deploy2k8s.sh 部署鏡像到k8s環境腳本

#!/bin/bash
# -- 部署image到k8s --
# !注意：需要提前做ssh免密登錄
set -eu

#定義常量
PROJECT_NAME="cicd-demo"
UPLOAD_DIR="/home/hellxz/apps/${PROJECT_NAME}"
FILE_NAME="${UPLOAD_DIR}/deploy.yaml"
SSH_USER="hellxz"
SSH_IP="192.168.87.129"

#首先刪除待上傳目錄的同名文件
ssh ${SSH_USER}@${SSH_IP} "rm -rf ${FILE_NAME}"

#確保部署文件目錄存在
ssh ${SSH_USER}@${SSH_IP} "mkdir -p ${UPLOAD_DIR}"

#遠程複製部署文件
scp -r deploy.yaml ${SSH_USER}@${SSH_IP}:${FILE_NAME}

#遠程執行部署命令
ssh ${SSH_USER}@${SSH_IP} "kubectl apply -f ${FILE_NAME}"

Jenkinsfile

pipeline{
    agent any
    stages {
        //由於源碼和Jenkinsfile處於同一倉庫，在Jenkins項目執行時，會先將Jenkinsfile所在的倉庫克隆下來，為了簡單，這裡就不重複添加拉取源碼的操作了
        //其它情況，如使用多個倉庫一些構建的，這種就需要額外添加拉取程式碼的stage了。
        stage('Build Artifact') {
            steps{
                sh label:'maven building', script: 'mvn clean package -DskipTests'
            }
        }
        stage('Build Image'){
            steps{
                sh label:'image building', script: '/bin/bash artifact2image.sh'
            }
        }
        stage('Deploy k8s'){
            steps{
                sh label:'deploy image to k8s', script: '/bin/bash deploy2k8s.sh'
            }
        }
    }
    post {
        success{
            //成功清理工作空間，失敗保留現場
            cleanWs()
        }
    }
}

deploy.yaml

apiVersion: apps/v1   #api版本
kind: Deployment      #對象類型
metadata:             #元資訊標識
  name: cicd-demo
  namespace: default
  labels:             #自定義Deployment綁定的標籤
    app: cicd-demo
spec:                 #期待運行狀態
  replicas: 1         #部署實例數
  selector:
    matchLabels:
      app: cicd-demo
  template:           #pod
    metadata:
      labels:         #與spec.selector.matchLabels相同
        app: cicd-demo
    spec:             #pod容器期望的狀態
      containers:
        - name: cicd-demo
          image: {{IMAGE_NAME}}
          imagePullPolicy: IfNotPresent
          env:
            - name: JVM_OPTS
              value: "-Xms128m -Xmx256m"
          ports:
            - containerPort: 8080

---
apiVersion: v1
kind: Service
metadata:
  name: cicd-demo-svc
  namespace: default
  labels:
    service: cicd-demo-svc
spec:
  selector:            #匹配pod標籤
    app: cicd-demo
  type: NodePort     #svc類型，ExternalName, ClusterIP, NodePort, and LoadBalancer
  ports:
    - name: cicd-demo-port
      protocol: TCP
      port: 8080
      nodePort: 30000

09 創建Jenkins流水線項目

訪問 //192.168.87.128:8080/jenkins 並登錄，以下創建新流水線demo構建項目

部署與驗證

執行構建demo程式

日誌輸出：

Started by user admin
Obtained Jenkinsfile from git //github.com/hellxz/cicd-demo.git
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /home/hellxz/jenkins/home/workspace/demo
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Declarative: Checkout SCM)
[Pipeline] checkout
Selected Git installation does not exist. Using Default
The recommended git tool is: NONE
No credentials specified
Cloning the remote Git repository
Cloning repository //github.com/hellxz/cicd-demo.git
 > git init /home/hellxz/jenkins/home/workspace/demo # timeout=10
Fetching upstream changes from //github.com/hellxz/cicd-demo.git
 > git --version # timeout=10
 > git --version # 'git version 2.17.1'
 > git fetch --tags --progress -- //github.com/hellxz/cicd-demo.git +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url //github.com/hellxz/cicd-demo.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
Avoid second fetch
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
Checking out Revision 1fc086590e32c001c8b65a4c575f4aa1b5b20413 (refs/remotes/origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 1fc086590e32c001c8b65a4c575f4aa1b5b20413 # timeout=10
Commit message: "update"
 > git rev-list --no-walk 1fc086590e32c001c8b65a4c575f4aa1b5b20413 # timeout=10
[Pipeline] }
[Pipeline] // stage
[Pipeline] withEnv
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Build Artifact)
[Pipeline] sh (maven building)
+ mvn clean package -DskipTests
NOTE: Picked up JDK_JAVA_OPTIONS:  --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
[INFO] Scanning for projects...

[INFO] 
[INFO] ----------------------< online.hellxz:cicd-demo >-----------------------
[INFO] Building cicd-demo 0.0.1
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ cicd-demo ---
[INFO] 
[INFO] --- maven-resources-plugin:3.2.0:resources (default-resources) @ cicd-demo ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Using 'UTF-8' encoding to copy filtered properties files.
[INFO] Copying 1 resource
[INFO] Copying 0 resource
[INFO] The encoding used to copy filtered properties files have not been set. This means that the same encoding will be used to copy filtered properties files as when copying other filtered resources. This might not be what you want! Run your build with --debug to see which files might be affected. Read more at //maven.apache.org/plugins/maven-resources-plugin/examples/filtering-properties-files.html
[INFO] 
[INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ cicd-demo ---

[INFO] Changes detected - recompiling the module!
[INFO] Compiling 1 source file to /home/hellxz/jenkins/home/workspace/demo/target/classes

[INFO] 
[INFO] --- maven-resources-plugin:3.2.0:testResources (default-testResources) @ cicd-demo ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Using 'UTF-8' encoding to copy filtered properties files.
[INFO] skip non existing resourceDirectory /home/hellxz/jenkins/home/workspace/demo/src/test/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ cicd-demo ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ cicd-demo ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:3.2.0:jar (default-jar) @ cicd-demo ---
[INFO] Building jar: /home/hellxz/jenkins/home/workspace/demo/target/cicd-demo-0.0.1.jar
[INFO] 
[INFO] --- spring-boot-maven-plugin:2.4.1:repackage (repackage) @ cicd-demo ---
[INFO] Replacing main artifact with repackaged archive
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  3.248 s
[INFO] Finished at: 2020-12-23T11:51:20Z
[INFO] ------------------------------------------------------------------------
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Build Image)

[Pipeline] sh (image building)
+ /bin/bash artifact2image.sh
Sending build context to Docker daemon  17.26MB

Step 1/5 : FROM openjdk:8-jdk
 ---> 89f100fa8f9f
Step 2/5 : WORKDIR /root
 ---> Using cache
 ---> 17670bfa01f1
Step 3/5 : ADD target/*.jar app.jar

 ---> b7e4292e4f80
Step 4/5 : ENV JVM_OPTS=""
 ---> Running in 0be8c16c9c8a
Removing intermediate container 0be8c16c9c8a
 ---> 3ef66b100082
Step 5/5 : CMD java -Djava.security.egd=file:/dev/./urandom     ${JVM_OPTS}     -jar app.jar
 ---> Running in c8209a642a2d
Removing intermediate container c8209a642a2d
 ---> bc7859f9b4fe
Successfully built bc7859f9b4fe
Successfully tagged 192.168.87.129:5000/cicd-demo:20205123_1151

The push refers to repository [192.168.87.129:5000/cicd-demo]
44cbb90cc2c3: Preparing
f85e383859a1: Preparing
ffb4778f8a52: Preparing
e528f2c31deb: Preparing
c5f4367d4a59: Preparing
ceecb62b2fcc: Preparing
193bc1d68b80: Preparing
f0e10b20de19: Preparing
ceecb62b2fcc: Waiting
193bc1d68b80: Waiting
f0e10b20de19: Waiting
ffb4778f8a52: Layer already exists
e528f2c31deb: Layer already exists
f85e383859a1: Layer already exists
193bc1d68b80: Layer already exists
c5f4367d4a59: Layer already exists
ceecb62b2fcc: Layer already exists
f0e10b20de19: Layer already exists
44cbb90cc2c3: Pushed

20205123_1151: digest: sha256:24976370741b8e20a7e8c7d18e13f4b72bb492f9c77908927d3b17b8e1ce75d4 size: 2006
Untagged: 192.168.87.129:5000/cicd-demo:20205123_1151
Untagged: 192.168.87.129:5000/cicd-demo@sha256:24976370741b8e20a7e8c7d18e13f4b72bb492f9c77908927d3b17b8e1ce75d4
Deleted: sha256:bc7859f9b4fefc5fdcb39bab4c77dff94980d70fcd21b0b3755f317f517e4291
Deleted: sha256:3ef66b100082a9d6949802c53fe57442d0e156216f0ba7a544b4e44d5a80e7a4
Deleted: sha256:b7e4292e4f807a13f800b5acc8adb0a84926e5ce5c4cc850a2769503558efcb9
Deleted: sha256:2afe950a3c18304e26f7d6b3f3bc48d9de631eb8827203bfa273d361647d0951
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Deploy k8s)
[Pipeline] sh (deploy image to k8s)
+ /bin/bash deploy2k8s.sh

deployment.apps/cicd-demo created
service/cicd-demo-svc created
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Declarative: Post Actions)

[Pipeline] cleanWs
[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
[WS-CLEANUP] done
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

可以看到以下輸出，說明程式已部署到k8s集群中了

deployment.apps/cicd-demo created
service/cicd-demo-svc created

ssh登錄k8s的機器驗證

總結

簡單總結下，Jenkins與Kubernetes的集成，更多地是Jenkins如何控制Kubernetes部署的流程，文中例子是通過 構建並上傳鏡像 和 ssh發送配置和命令 實現的。

本文是參考慕課網上的教程，快速掃了下影片實踐所得，是了解其思路後自行實現的。

實際操作上才發現，腦子說會了，然後雙手卻沒有跟上，還是要動手啊！

操作了一天半左右，最開始還想著用vagrant初始化VM，由於工作上有個虛擬機是VBox5的，升級新版不能用。搞了半天放棄了Vagrant（VBox6.1升級了命令參數不兼容舊版，導致Vagrant為兼容它把幾乎所有版本Vagrant都改了……）。

文中難免因為安裝部分的描述而過於冗長，感謝大家能看到這裡，最後再次感謝慕課網的鹿哥

參考

//www.imooc.com/learn/1112
//cloud.tencent.com/developer/article/1394657
//www.jianshu.com/p/3f13850d66ea
//www.cnblogs.com/hellxz/p/use-kubeadm-init-kubernetes-cluster.html
//www.cnblogs.com/hellxz/p/how-kubernetes-deploy-application.html
//developer.aliyun.com/article/221687?spm=a2c6h.12873639.0.0.755c6235EZuEht
//minikube.sigs.k8s.io/docs/start
//www.cnblogs.com/hellxz/p/install_jenkins.html
//www.cnblogs.com/hellxz/p/jenkins_install_plugins_faster.html

Tags: Kubernetes/OpenShift 持續交付CI/CD