4. 死磕 k8s系列之安裝包管理工具(Helm)

2020 年 2 月 10 日
筆記

註：本文基於k8s集群v1.16.2版本。

註：如無特殊說明，以下所有操作都是在master節點上執行。

簡介

Helm可以看作是k8s集群的包管理工具，通過Helm可以快速安裝很多軟體，比如mysql，nginx等，當然，也可以把自己的應用交給Helm來管理和安裝。

Helm架構由Helm客戶端、Helm服務端Tiller和Helm倉庫Chart組成。

其中，Helm客戶端可以部署在任意地方，只要能夠訪問k8s集群即可; Tiller服務端部署在k8s集群中。

註：為了方便，本文直接把Helm客戶端安裝在master節點上。

前提

必須有一個k8s集群。

安裝Helm客戶端

（1）下載Helm發布包

打開Helm Release，下載一個穩定版本，我這裡下載的是v2.15.2版本的 Linuxamd64。

註：這裡要下載穩定版本，不要下載RC（Release Candidate）版本，因為我們後面要修改鏡像，非穩定版本中國的鏡像倉庫不一定有。

下載下來之後，解壓，然後把helm可執行文件拷貝到/usr/local/bin目錄下，之後就可以執行helm命令了。

[root@instance-ji0xk9uh-1 ~]# ls  calico-3.9.2.yaml  helm-v2.15.2-linux-amd64.tar.gz  init_master.sh  install_kubelet.sh  kubeadm-config.yaml  nginx-ingress.yaml  [root@instance-ji0xk9uh-1 ~]# tar zxvf helm-v2.15.2-linux-amd64.tar.gz  linux-amd64/  linux-amd64/tiller  linux-amd64/helm  linux-amd64/README.md  linux-amd64/LICENSE  [root@instance-ji0xk9uh-1 ~]# ls  calico-3.9.2.yaml  helm-v2.15.2-linux-amd64.tar.gz  init_master.sh  install_kubelet.sh  kubeadm-config.yaml  linux-amd64  nginx-ingress.yaml  [root@instance-ji0xk9uh-1 ~]# cp linux-amd64/helm /usr/local/bin/  [root@instance-ji0xk9uh-1 ~]# helm version  Client: &version.Version{SemVer:"v2.15.2", GitCommit:"8dce272473e5f2a7bf58ce79bb5c3691db54c96b", GitTreeState:"clean"}  Error: could not find tiller

安裝tiller服務端

直接執行 helm init即可。

[root@master ~]# helm init  Creating /root/.helm  Creating /root/.helm/repository  Creating /root/.helm/repository/cache  Creating /root/.helm/repository/local  Creating /root/.helm/plugins  Creating /root/.helm/starters  Creating /root/.helm/cache/archive  Creating /root/.helm/repository/repositories.yaml  Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com  Adding local repo with URL: http://127.0.0.1:8879/charts  $HELM_HOME has been configured at /root/.helm.    Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.    Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.  To prevent this, run `helm init` with the --tiller-tls-verify flag.  For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation

檢查是否安裝成功， heml version。

[root@master ~]# helm version  Client: &version.Version{SemVer:"v2.15.2", GitCommit:"8dce272473e5f2a7bf58ce79bb5c3691db54c96b", GitTreeState:"clean"}  Error: could not find a ready tiller pod

發現並沒有安裝成功，再看看pod的狀態， kubectlgetpod-n kube-system。

[root@master ~]# kubectl get pod -n kube-system  NAME                                      READY   STATUS             RESTARTS   AGE  # 省略...  tiller-deploy-58f57c5787-v6z7d            0/1     ImagePullBackOff   0          5m50s

發現pod的狀態為 ImagePullBackOff，出現這個的原因很大可能是鏡像沒有拉取下來。

我們使用 describe查看更詳細的原因。

[root@master ~]# kubectl describe pod/tiller-deploy-58f57c5787-v6z7d -n kube-system  # 省略...  Events:    Type     Reason     Age                    From               Message    ----     ------     ----                   ----               -------    Normal   Scheduled  <unknown>              default-scheduler  Successfully assigned kube-system/tiller-deploy-58f57c5787-v6z7d to node1    Normal   Pulling    2m19s (x4 over 4m23s)  kubelet, node1     Pulling image "gcr.io/kubernetes-helm/tiller:v2.15.2"    Warning  Failed     2m4s (x4 over 4m7s)    kubelet, node1     Failed to pull image "gcr.io/kubernetes-helm/tiller:v2.15.2": rpc error: code = Unknown desc = Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)    Warning  Failed     2m4s (x4 over 4m7s)    kubelet, node1     Error: ErrImagePull    Normal   BackOff    97s (x6 over 4m7s)     kubelet, node1     Back-off pulling image "gcr.io/kubernetes-helm/tiller:v2.15.2"    Warning  Failed     84s (x7 over 4m7s)     kubelet, node1     Error: ImagePullBackOff

從這裡可以看到它使用的鏡像是 gcr.io/kubernetes-helm/tiller:v2.15.2，我們直接在鏡像倉庫搜索這個鏡像。

[root@master ~]# docker search gcr.io/kubernetes-helm/tiller:v2.15.2  Error response from daemon: invalid registry endpoint https://gcr.io/v1/: Get https://gcr.io/v1/_ping: dial tcp 108.177.97.82:443: i/o timeout. If this private registry supports only HTTP or HTTPS with an unknown CA certificate, please add `--insecure-registry gcr.io` to the daemon's arguments. In the case of HTTPS, if you have access to the registry's CA certificate, no need for the flag; simply place the CA certificate at /etc/docker/certs.d/gcr.io/ca.crt

從上面的日誌可以看到是超時了，這時候我們換一個角度，看看我們的鏡像倉庫有沒有 tiller的鏡像。

[root@master ~]# docker search tiller  NAME                                    DESCRIPTION                                     STARS               OFFICIAL            AUTOMATED  jessestuart/tiller                      Nightly multi-architecture (amd64, arm64, ar…   16                                      [OK]  sapcc/tiller                            Mirror of https://gcr.io/kubernetes-helm/til…   8  ist0ne/tiller                           https://gcr.io/kubernetes-helm/tiller           3                                       [OK]  rancher/tiller                                                                          2  jmgao1983/tiller                        from gcr.io/kubernetes-helm/tiller              2                                       [OK]  ibmcom/tiller                           Docker Image for IBM Cloud private-CE (Commu…   1  itinerisltd/tiller                                                                      1  luxas/tiller                                                                            1  ansibleplaybookbundle/tiller-apb        An APB that deploys tiller for use with helm.   1                                       [OK]  cfplatformeng/tiller                                                                    0  cfplatformeng/tiller-ubuntu                                                             0  ibmcom/tiller-ppc64le                   Docker Image for IBM Cloud Private-CE (Commu…   0  kubeapps/tiller-proxy                   A web-based UI for deploying and managing ap…   0  kubeapps/tiller-proxy-ci                Store temporary images generated by CI system   0  sorenmat/tiller                                                                         0  cgetzen/tiller                          Custom Tiller Tests                             0  appscode/tiller                                                                         0  fkpwolf/tiller                                                                          0  itinerisltd/tiller-circleci                                                             0  anjia0532/tiller                                                                        0  pcanham/tiller                          tiller image for Raspberry Pi for testing He…   0  kontenapharos/tiller                                                                    0  renaultdigital/tillerless-helm-gcloud   Add tillerless plugin to helm-gcloud image      0  4admin2root/tiller                      gcr.io/kubernetes-helm/tiller                   0                                       [OK]  zhaowenlei/tiller                       FROM gcr.io/kubernetes-helm/tiller:TAG          0

可以查到很多tiller的鏡像，我們這裡選第二個，看它的描述，是 gcr.io/kubernetes-helm/tiller的鏡像。

我們嘗試拉取看看，注意這裡一定要拉取跟上面安裝的helm相同版本的鏡像。

[root@master ~]# docker pull sapcc/tiller:v2.15.2  v2.15.2: Pulling from sapcc/tiller  89d9c30c1d48: Pull complete  45d707e102b0: Pull complete  d20e148dce16: Pull complete  8ca326d5a0c5: Pull complete  Digest: sha256:4554a65fb8278d93f1c2c1f335ddbfcd6faa016c24b97e8de46c6b8fc1e9e7f5  Status: Downloaded newer image for sapcc/tiller:v2.15.2

可以看到拉取成功了。所以，我們換一種思路，把tiller安裝的yaml中使用的鏡像改成上面這個可用的鏡像，那麼要怎麼修改呢？

其實，也很簡單，k8s提供了命令可以直接修改現在正在運行的pod的yaml配置， kubectl edit pod xxx。

注，這裡也可以修改deployment的配置，這樣更持久，因為pod是由deployment創建的。

[root@master ~]# kubectl edit pod tiller-deploy-58f57c5787-v6z7d -n kube-system  #省略...  spec:    automountServiceAccountToken: true    containers:    - env:      - name: TILLER_NAMESPACE        value: kube-system      - name: TILLER_HISTORY_MAX        value: "0"      image: sapcc/tiller:v2.15.2      imagePullPolicy: IfNotPresent      livenessProbe:        failureThreshold: 3        httpGet:          path: /liveness          port: 44135          scheme: HTTP        initialDelaySeconds: 1        periodSeconds: 10        successThreshold: 1        timeoutSeconds: 1      name: tiller  #省略...

找到image欄位，把後面的鏡像修改為 sapcc/tiller:v2.15.2。

然後，pod會自動更新，過一會再次查看pod的狀態。

[root@master ~]# kubectl get pod -n kube-system  NAME                                      READY   STATUS    RESTARTS   AGE  #省略...  tiller-deploy-58f57c5787-v6z7d            1/1     Running   0          25m

可以看到tiller運行起來了，再次使用 helm version檢查結果。

[root@master ~]# helm version  Client: &version.Version{SemVer:"v2.15.2", GitCommit:"8dce272473e5f2a7bf58ce79bb5c3691db54c96b", GitTreeState:"clean"}  Server: &version.Version{SemVer:"v2.15.2", GitCommit:"8dce272473e5f2a7bf58ce79bb5c3691db54c96b", GitTreeState:"clean"}

至此，說明tiller安裝成功了。

使用helm安裝mysql

helm search mysql，查找mysql的chart。

helm install mysql，安裝mysql。

[root@master ~]# helm search mysql  NAME                                CHART VERSION   APP VERSION DESCRIPTION  stable/mysql                        1.4.0           5.7.27      Fast, reliable, scalable, and easy to use open-source rel...  stable/mysqldump                    2.6.0           2.4.1       A Helm chart to help backup MySQL databases using mysqldump  stable/prometheus-mysql-exporter    0.5.2           v0.11.0     A Helm chart for prometheus mysql exporter with cloudsqlp...  stable/percona                      1.2.0           5.7.17      free, fully compatible, enhanced, open source drop-in rep...  stable/percona-xtradb-cluster       1.0.3           5.7.19      free, fully compatible, enhanced, open source drop-in rep...  stable/phpmyadmin                   4.1.1           4.9.1       phpMyAdmin is an mysql administration frontend  stable/gcloud-sqlproxy              0.6.1           1.11        DEPRECATED Google Cloud SQL Proxy  stable/mariadb                      6.12.2          10.3.18     Fast, reliable, scalable, and easy to use open-source rel...  [root@master ~]# helm install stable/mysql  Error: no available release name found

又報錯了，經查找相關資料，是說k8s集群1.16.x版本加入了RBAC許可權相關的東西，必須給tiller定義一個service account。

OK，使用以下腳本，先卸載tiller（刪除相應的deployment），再設置許可權，重新安裝。

注，如果安裝tiller過程中出現如下錯誤也是執行下面的腳本。

Error:error installing:the server couldnotfind the requested resource

重裝tiller腳本。

# 先卸載tiller，刪除其deployment即可  kubectl delete deployment.apps/tiller-deploy -n kube-system  # 創建serviceaccount  kubectl create serviceaccount --namespace kube-system tiller  # 綁定  kubectl create clusterrolebinding tiller --clusterrole=cluster-admin --serviceaccount=kube-system:tiller  # 重新初始化tiller  helm init --service-account tiller --override spec.selector.matchLabels.'name'='tiller',spec.selector.matchLabels.'app'='helm' --output yaml | sed 's@apiVersion: extensions/v1beta1@apiVersion: apps/v1@' | kubectl apply -f -

執行結果如下：

[root@master ~]# kubectl get deployment -n kube-system  NAME                      READY   UP-TO-DATE   AVAILABLE   AGE  calico-kube-controllers   1/1     1            1           22h  coredns                   2/2     2            2           22h  kuboard                   1/1     1            1           21h  tiller-deploy             1/1     1            1           7m21s  [root@master ~]# kubectl delete deployment.apps/tiller-deploy -n kube-system  deployment.apps "tiller-deploy" deleted  [root@master ~]# kubectl create serviceaccount --namespace kube-system tiller  ec.selector.matchLabels.'app'='helm' --output yaml | sed 's@apiVersion: extensions/v1beta1@apiVersion: apps/v1@' | kubectl apply -f -serviceaccount/tiller created  [root@master ~]# kubectl create clusterrolebinding tiller --clusterrole=cluster-admin --serviceaccount=kube-system:tiller  clusterrolebinding.rbac.authorization.k8s.io/tiller created  [root@master ~]# helm init --service-account tiller --override spec.selector.matchLabels.'name'='tiller',spec.selector.matchLabels.'app'='helm' --output yaml | sed 's@apiVersion: extensions/v1beta1@apiVersion: apps/v1@' | kubectl apply -f -  deployment.apps/tiller-deploy created  Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply  service/tiller-deploy configured

這時候又會出現 ImagePullBackOff錯誤，同樣地，重新修改tiller的yaml文件中的鏡像，會自動更新。

重新安裝後，再重新安裝mysql。

[root@master ~]# helm install stable/mysql  NAME:   bumptious-fish  LAST DEPLOYED: Thu Oct 31 15:21:43 2019  NAMESPACE: default  STATUS: DEPLOYED    RESOURCES:  ==> v1/ConfigMap  NAME                       DATA  AGE  bumptious-fish-mysql-test  1     0s    ==> v1/Deployment  NAME                  READY  UP-TO-DATE  AVAILABLE  AGE  bumptious-fish-mysql  0/1    1           0          0s    ==> v1/PersistentVolumeClaim  NAME                  STATUS   VOLUME  CAPACITY  ACCESS MODES  STORAGECLASS  AGE  bumptious-fish-mysql  Pending  0s      Filesystem    ==> v1/Pod(related)  NAME                                   READY  STATUS   RESTARTS  AGE  bumptious-fish-mysql-68bf985647-649ms  0/1    Pending  0         0s    ==> v1/Secret  NAME                  TYPE    DATA  AGE  bumptious-fish-mysql  Opaque  2     0s    ==> v1/Service  NAME                  TYPE       CLUSTER-IP    EXTERNAL-IP  PORT(S)   AGE  bumptious-fish-mysql  ClusterIP  10.96.43.224  <none>       3306/TCP  0s      NOTES:  MySQL can be accessed via port 3306 on the following DNS name from within your cluster:  bumptious-fish-mysql.default.svc.cluster.local    To get your root password run:        MYSQL_ROOT_PASSWORD=$(kubectl get secret --namespace default bumptious-fish-mysql -o jsonpath="{.data.mysql-root-password}" | base64 --decode; echo)    To connect to your database:    1. Run an Ubuntu pod that you can use as a client:        kubectl run -i --tty ubuntu --image=ubuntu:16.04 --restart=Never -- bash -il    2. Install the mysql client:        $ apt-get update && apt-get install mysql-client -y    3. Connect using the mysql cli, then provide your password:      $ mysql -h bumptious-fish-mysql -p    To connect to your database directly from outside the K8s cluster:      MYSQL_HOST=127.0.0.1      MYSQL_PORT=3306        # Execute the following command to route the connection:      kubectl port-forward svc/bumptious-fish-mysql 3306        mysql -h ${MYSQL_HOST} -P${MYSQL_PORT} -u root -p${MYSQL_ROOT_PASSWORD}

查看mysql是否啟動成功，我們是安裝到默認命名空間下的，所以不需要帶 -n參數了。

[root@master ~]# kubectl get pod  NAME                                    READY   STATUS    RESTARTS   AGE  bumptious-fish-mysql-68bf985647-649ms   0/1     Pending   0          40s

會發現一直是在 Pending狀態，同樣地，使用 kubectl describe命令查看詳細的錯誤。

[root@master ~]# kubectl describe pod bumptious-fish-mysql-68bf985647-649ms  # 省略...  Events:    Type     Reason            Age        From               Message    ----     ------            ----       ----               -------    Warning  FailedScheduling  <unknown>  default-scheduler  pod has unbound immediate PersistentVolumeClaims (repeated 2 times)    Warning  FailedScheduling  <unknown>  default-scheduler  pod has unbound immediate PersistentVolumeClaims (repeated 2 times)

看最後兩行，是說pod沒有綁定到PersistentVolumeClaims（簡寫pvc），我們再查查pvc。

[root@master ~]# kubectl get pvc  NAME                   STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE  bumptious-fish-mysql   Pending                                                     2m18s

可以看到pvc也是 Pending狀態，同樣地，describe一下。

[root@master ~]# kubectl describe pvc bumptious-fish-mysql  # 省略...  Events:    Type    Reason         Age                   From                         Message    ----    ------         ----                  ----                         -------    Normal  FailedBinding  11s (x12 over 2m44s)  persistentvolume-controller  no persistent volumes available for this claim and no storage class is set

看最後一行，說是沒有可用的persistent volume（簡稱pv）且沒有設置storage class，這又是些什麼東西呢。

這裡就牽涉到k8s的整體架構了，針對有狀態的pod，需要先關聯一個pvc，pvc會與具體的pv關聯，pv可以理解成是磁碟上的一塊空間，或者是遠程存儲等。

關於pv的自動創建也有很多方案，比如nfs等，為了不增加複雜性，我們這裡採用手動創建兩塊pv給mysql使用。

手動創建pv的腳本如下：

kubectl create -f - <<EOF  kind: PersistentVolume  apiVersion: v1  metadata:    name: [名稱]    labels:      type: local  spec:    capacity:      storage: [使用的空間]    accessModes:      - ReadWriteOnce    hostPath:      path: "存儲的路徑"  EOF

執行結果如下：

[root@master ~]# kubectl create -f - <<EOF  > kind: PersistentVolume  > apiVersion: v1  > metadata:  >   name: mysql-pv  >   labels:  >     type: local  > spec:  >   capacity:  >     storage: 10Gi  >   accessModes:  >     - ReadWriteOnce  >   hostPath:  >     path: "/data/storage/mysql"  > EOF  persistentvolume/mysql-pv created  [root@master ~]# kubectl get pv,pvc,pod  NAME                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                          STORAGECLASS   REASON   AGE  persistentvolume/mysql-pv   10Gi       RWO            Retain           Bound    default/bumptious-fish-mysql                           67s    NAME                                         STATUS   VOLUME     CAPACITY   ACCESS MODES   STORAGECLASS   AGE  persistentvolumeclaim/bumptious-fish-mysql   Bound    mysql-pv   10Gi       RWO                           5m49s    NAME                                        READY   STATUS    RESTARTS   AGE  pod/bumptious-fish-mysql-68bf985647-649ms   1/1     Running   0          5m49s

這裡的 kubectlgetpv,pvc,pod要過一會再執行，或者前面加一個watch，即 watch kubectlgetpv,pvc,pod，直到pod的狀態變成Running且READY為1/1的時候表示mysql安裝成功了。

到這裡，基本可以確定mysql是安裝成功了，但是沒有連接進去誰又敢百分百說安裝成功了呢，OK，我們下面嘗試連接到mysql中。

連接mysql

查看mysql的service。

[root@master ~]# kubectl get svc  NAME                   TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE  bumptious-fish-mysql   ClusterIP   10.96.43.224   <none>        3306/TCP   9m36s  kubernetes             ClusterIP   10.96.0.1      <none>        443/TCP    23h

可以看到服務暴露的方式為 ClusterIP，我們把它修改為 NodePort，NodePort可以讓我們從外部訪問服務。

[root@master ~]# kubectl edit svc bumptious-fish-mysql    # Please edit the object below. Lines beginning with a '#' will be ignored,  # and an empty file will abort the edit. If an error occurs while saving this file will be  # reopened with the relevant failures.  #  apiVersion: v1  kind: Service  metadata:    creationTimestamp: "2019-10-31T07:21:43Z"    labels:      app: bumptious-fish-mysql      chart: mysql-1.4.0      heritage: Tiller      release: bumptious-fish    name: bumptious-fish-mysql    namespace: default    resourceVersion: "125451"    selfLink: /api/v1/namespaces/default/services/bumptious-fish-mysql    uid: 70be1da0-a10f-4439-9d3e-fd516f8785bf  spec:    clusterIP: 10.96.43.224    externalTrafficPolicy: Cluster    ports:    - name: mysql      nodePort: 32684      port: 3306      protocol: TCP      targetPort: mysql    selector:      app: bumptious-fish-mysql    sessionAffinity: None    type: NodePort  status:    loadBalancer: {}

找到spec.type，將其值修改為 NodePort，保存退出，查看自動生成的埠。

[root@master ~]# kubectl get svc  NAME                   TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE  bumptious-fish-mysql   NodePort    10.96.43.224   <none>        3306:32684/TCP   18h  kubernetes             ClusterIP   10.96.0.1      <none>        443/TCP          41h

可以看到生成的埠為 32684，再查看mysql的初始密碼，命令在安裝mysql的時候提示過，注意這裡拷貝你安裝時提示的命令。

[root@master ~]# echo $(kubectl get secret --namespace default bumptious-fish-mysql -o jsonpath="{.data.mysql-root-password}" | base64 --decode; echo)  3XvXkqckUE

在外部使用mysql客戶端連接（當然，也可以在集群內部裝一個mysql客戶端，這樣就不需要暴露外網ip及埠了）：

ip為任意worker節點的公網ip；

埠為上面獲取到的32684；

用戶名為root，密碼為上面獲取到的3XvXkqckUE；

C:> mysql -h[xx.xx.xx.xx] -P32684 -uroot -p3XvXkqckUE  mysql: [Warning] Using a password on the command line interface can be insecure.  Welcome to the MySQL monitor.  Commands end with ; or g.  Your MySQL connection id is 13658  Server version: 5.7.14 MySQL Community Server (GPL)    Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.    Oracle is a registered trademark of Oracle Corporation and/or its  affiliates. Other names may be trademarks of their respective  owners.    Type 'help;' or 'h' for help. Type 'c' to clear the current input statement.    mysql> show databases;  +--------------------+  | Database           |  +--------------------+  | information_schema |  | mysql              |  | performance_schema |  | sys                |  +--------------------+  4 rows in set (0.04 sec)    mysql> create database test;  Query OK, 1 row affected (0.04 sec)    mysql> use test;  Database changed  mysql> create table user(id int, name varchar(50));  Query OK, 0 rows affected (0.05 sec)    mysql> insert into user values(1,"hello");  Query OK, 1 row affected (0.05 sec)    mysql> select * from user;  +------+-------+  | id   | name  |  +------+-------+  |    1 | hello |  +------+-------+  1 row in set (0.04 sec)

成功連接進去了，說明mysql安裝成功了。

總結

本章成功安裝了Helm，並使用Helm成功安裝了mysql，這裡有幾個需要注意的點。

（1）k8s集群v1.16.x版本需要設置RBAC許可權，才能成功安裝Helm；

（2）網路問題導致無法成功拉取tiller鏡像，所以需要手動修改tiller pod的yaml文件；

（3）mysql安裝需要創建持久卷PV，我們這裡使用手動創建的方式。

參考

官方文檔：https://helm.sh/docs/using_helm

本文在官方文檔的基礎上補充了一些異常的處理。

Tags: