4. 死磕 k8s系列之安装包管理工具(Helm)
- 2020 年 2 月 10 日
- 筆記
注：本文基于k8s集群v1.16.2版本。
注：如无特殊说明，以下所有操作都是在master节点上执行。
简介
Helm可以看作是k8s集群的包管理工具，通过Helm可以快速安装很多软件，比如mysql，nginx等，当然，也可以把自己的应用交给Helm来管理和安装。
Helm架构由Helm客户端、Helm服务端Tiller和Helm仓库Chart组成。
其中，Helm客户端可以部署在任意地方，只要能够访问k8s集群即可; Tiller服务端部署在k8s集群中。
注：为了方便，本文直接把Helm客户端安装在master节点上。
前提
必须有一个k8s集群。
安装Helm客户端
（1）下载Helm发布包
打开Helm Release，下载一个稳定版本，我这里下载的是v2.15.2版本的 Linuxamd64。
注：这里要下载稳定版本，不要下载RC（Release Candidate）版本，因为我们后面要修改镜像，非稳定版本国内的镜像仓库不一定有。
下载下来之后，解压，然后把helm可执行文件拷贝到/usr/local/bin目录下，之后就可以执行helm命令了。
[root@instance-ji0xk9uh-1 ~]# ls calico-3.9.2.yaml helm-v2.15.2-linux-amd64.tar.gz init_master.sh install_kubelet.sh kubeadm-config.yaml nginx-ingress.yaml [root@instance-ji0xk9uh-1 ~]# tar zxvf helm-v2.15.2-linux-amd64.tar.gz linux-amd64/ linux-amd64/tiller linux-amd64/helm linux-amd64/README.md linux-amd64/LICENSE [root@instance-ji0xk9uh-1 ~]# ls calico-3.9.2.yaml helm-v2.15.2-linux-amd64.tar.gz init_master.sh install_kubelet.sh kubeadm-config.yaml linux-amd64 nginx-ingress.yaml [root@instance-ji0xk9uh-1 ~]# cp linux-amd64/helm /usr/local/bin/ [root@instance-ji0xk9uh-1 ~]# helm version Client: &version.Version{SemVer:"v2.15.2", GitCommit:"8dce272473e5f2a7bf58ce79bb5c3691db54c96b", GitTreeState:"clean"} Error: could not find tiller
安装tiller服务端
直接执行 helm init即可。
[root@master ~]# helm init Creating /root/.helm Creating /root/.helm/repository Creating /root/.helm/repository/cache Creating /root/.helm/repository/local Creating /root/.helm/plugins Creating /root/.helm/starters Creating /root/.helm/cache/archive Creating /root/.helm/repository/repositories.yaml Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com Adding local repo with URL: http://127.0.0.1:8879/charts $HELM_HOME has been configured at /root/.helm. Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster. Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy. To prevent this, run `helm init` with the --tiller-tls-verify flag. For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation
检查是否安装成功， heml version。
[root@master ~]# helm version Client: &version.Version{SemVer:"v2.15.2", GitCommit:"8dce272473e5f2a7bf58ce79bb5c3691db54c96b", GitTreeState:"clean"} Error: could not find a ready tiller pod
发现并没有安装成功，再看看pod的状态， kubectlgetpod-n kube-system。
[root@master ~]# kubectl get pod -n kube-system NAME READY STATUS RESTARTS AGE # 省略... tiller-deploy-58f57c5787-v6z7d 0/1 ImagePullBackOff 0 5m50s
发现pod的状态为 ImagePullBackOff，出现这个的原因很大可能是镜像没有拉取下来。
我们使用 describe查看更详细的原因。
[root@master ~]# kubectl describe pod/tiller-deploy-58f57c5787-v6z7d -n kube-system # 省略... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled <unknown> default-scheduler Successfully assigned kube-system/tiller-deploy-58f57c5787-v6z7d to node1 Normal Pulling 2m19s (x4 over 4m23s) kubelet, node1 Pulling image "gcr.io/kubernetes-helm/tiller:v2.15.2" Warning Failed 2m4s (x4 over 4m7s) kubelet, node1 Failed to pull image "gcr.io/kubernetes-helm/tiller:v2.15.2": rpc error: code = Unknown desc = Error response from daemon: Get https://gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Warning Failed 2m4s (x4 over 4m7s) kubelet, node1 Error: ErrImagePull Normal BackOff 97s (x6 over 4m7s) kubelet, node1 Back-off pulling image "gcr.io/kubernetes-helm/tiller:v2.15.2" Warning Failed 84s (x7 over 4m7s) kubelet, node1 Error: ImagePullBackOff
从这里可以看到它使用的镜像是 gcr.io/kubernetes-helm/tiller:v2.15.2，我们直接在镜像仓库搜索这个镜像。
[root@master ~]# docker search gcr.io/kubernetes-helm/tiller:v2.15.2 Error response from daemon: invalid registry endpoint https://gcr.io/v1/: Get https://gcr.io/v1/_ping: dial tcp 108.177.97.82:443: i/o timeout. If this private registry supports only HTTP or HTTPS with an unknown CA certificate, please add `--insecure-registry gcr.io` to the daemon's arguments. In the case of HTTPS, if you have access to the registry's CA certificate, no need for the flag; simply place the CA certificate at /etc/docker/certs.d/gcr.io/ca.crt
从上面的日志可以看到是超时了，这时候我们换一个角度，看看我们的镜像仓库有没有 tiller的镜像。
[root@master ~]# docker search tiller NAME DESCRIPTION STARS OFFICIAL AUTOMATED jessestuart/tiller Nightly multi-architecture (amd64, arm64, ar… 16 [OK] sapcc/tiller Mirror of https://gcr.io/kubernetes-helm/til… 8 ist0ne/tiller https://gcr.io/kubernetes-helm/tiller 3 [OK] rancher/tiller 2 jmgao1983/tiller from gcr.io/kubernetes-helm/tiller 2 [OK] ibmcom/tiller Docker Image for IBM Cloud private-CE (Commu… 1 itinerisltd/tiller 1 luxas/tiller 1 ansibleplaybookbundle/tiller-apb An APB that deploys tiller for use with helm. 1 [OK] cfplatformeng/tiller 0 cfplatformeng/tiller-ubuntu 0 ibmcom/tiller-ppc64le Docker Image for IBM Cloud Private-CE (Commu… 0 kubeapps/tiller-proxy A web-based UI for deploying and managing ap… 0 kubeapps/tiller-proxy-ci Store temporary images generated by CI system 0 sorenmat/tiller 0 cgetzen/tiller Custom Tiller Tests 0 appscode/tiller 0 fkpwolf/tiller 0 itinerisltd/tiller-circleci 0 anjia0532/tiller 0 pcanham/tiller tiller image for Raspberry Pi for testing He… 0 kontenapharos/tiller 0 renaultdigital/tillerless-helm-gcloud Add tillerless plugin to helm-gcloud image 0 4admin2root/tiller gcr.io/kubernetes-helm/tiller 0 [OK] zhaowenlei/tiller FROM gcr.io/kubernetes-helm/tiller:TAG 0
可以查到很多tiller的镜像，我们这里选第二个，看它的描述，是 gcr.io/kubernetes-helm/tiller的镜像。
我们尝试拉取看看，注意这里一定要拉取跟上面安装的helm相同版本的镜像。
[root@master ~]# docker pull sapcc/tiller:v2.15.2 v2.15.2: Pulling from sapcc/tiller 89d9c30c1d48: Pull complete 45d707e102b0: Pull complete d20e148dce16: Pull complete 8ca326d5a0c5: Pull complete Digest: sha256:4554a65fb8278d93f1c2c1f335ddbfcd6faa016c24b97e8de46c6b8fc1e9e7f5 Status: Downloaded newer image for sapcc/tiller:v2.15.2
可以看到拉取成功了。所以，我们换一种思路，把tiller安装的yaml中使用的镜像改成上面这个可用的镜像，那么要怎么修改呢？
其实，也很简单，k8s提供了命令可以直接修改现在正在运行的pod的yaml配置， kubectl edit pod xxx。
注，这里也可以修改deployment的配置，这样更持久，因为pod是由deployment创建的。
[root@master ~]# kubectl edit pod tiller-deploy-58f57c5787-v6z7d -n kube-system #省略... spec: automountServiceAccountToken: true containers: - env: - name: TILLER_NAMESPACE value: kube-system - name: TILLER_HISTORY_MAX value: "0" image: sapcc/tiller:v2.15.2 imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 3 httpGet: path: /liveness port: 44135 scheme: HTTP initialDelaySeconds: 1 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 name: tiller #省略...
找到image字段，把后面的镜像修改为 sapcc/tiller:v2.15.2。
然后，pod会自动更新，过一会再次查看pod的状态。
[root@master ~]# kubectl get pod -n kube-system NAME READY STATUS RESTARTS AGE #省略... tiller-deploy-58f57c5787-v6z7d 1/1 Running 0 25m
可以看到tiller运行起来了，再次使用 helm version检查结果。
[root@master ~]# helm version Client: &version.Version{SemVer:"v2.15.2", GitCommit:"8dce272473e5f2a7bf58ce79bb5c3691db54c96b", GitTreeState:"clean"} Server: &version.Version{SemVer:"v2.15.2", GitCommit:"8dce272473e5f2a7bf58ce79bb5c3691db54c96b", GitTreeState:"clean"}
至此，说明tiller安装成功了。
使用helm安装mysql
helm search mysql，查找mysql的chart。
helm install mysql，安装mysql。
[root@master ~]# helm search mysql NAME CHART VERSION APP VERSION DESCRIPTION stable/mysql 1.4.0 5.7.27 Fast, reliable, scalable, and easy to use open-source rel... stable/mysqldump 2.6.0 2.4.1 A Helm chart to help backup MySQL databases using mysqldump stable/prometheus-mysql-exporter 0.5.2 v0.11.0 A Helm chart for prometheus mysql exporter with cloudsqlp... stable/percona 1.2.0 5.7.17 free, fully compatible, enhanced, open source drop-in rep... stable/percona-xtradb-cluster 1.0.3 5.7.19 free, fully compatible, enhanced, open source drop-in rep... stable/phpmyadmin 4.1.1 4.9.1 phpMyAdmin is an mysql administration frontend stable/gcloud-sqlproxy 0.6.1 1.11 DEPRECATED Google Cloud SQL Proxy stable/mariadb 6.12.2 10.3.18 Fast, reliable, scalable, and easy to use open-source rel... [root@master ~]# helm install stable/mysql Error: no available release name found
又报错了，经查找相关资料，是说k8s集群1.16.x版本加入了RBAC权限相关的东西，必须给tiller定义一个service account。
OK，使用以下脚本，先卸载tiller（删除相应的deployment），再设置权限，重新安装。
注，如果安装tiller过程中出现如下错误也是执行下面的脚本。
Error:error installing:the server couldnotfind the requested resource
重装tiller脚本。
# 先卸载tiller，删除其deployment即可 kubectl delete deployment.apps/tiller-deploy -n kube-system # 创建serviceaccount kubectl create serviceaccount --namespace kube-system tiller # 绑定 kubectl create clusterrolebinding tiller --clusterrole=cluster-admin --serviceaccount=kube-system:tiller # 重新初始化tiller helm init --service-account tiller --override spec.selector.matchLabels.'name'='tiller',spec.selector.matchLabels.'app'='helm' --output yaml | sed 's@apiVersion: extensions/v1beta1@apiVersion: apps/v1@' | kubectl apply -f -
执行结果如下：
[root@master ~]# kubectl get deployment -n kube-system NAME READY UP-TO-DATE AVAILABLE AGE calico-kube-controllers 1/1 1 1 22h coredns 2/2 2 2 22h kuboard 1/1 1 1 21h tiller-deploy 1/1 1 1 7m21s [root@master ~]# kubectl delete deployment.apps/tiller-deploy -n kube-system deployment.apps "tiller-deploy" deleted [root@master ~]# kubectl create serviceaccount --namespace kube-system tiller ec.selector.matchLabels.'app'='helm' --output yaml | sed 's@apiVersion: extensions/v1beta1@apiVersion: apps/v1@' | kubectl apply -f -serviceaccount/tiller created [root@master ~]# kubectl create clusterrolebinding tiller --clusterrole=cluster-admin --serviceaccount=kube-system:tiller clusterrolebinding.rbac.authorization.k8s.io/tiller created [root@master ~]# helm init --service-account tiller --override spec.selector.matchLabels.'name'='tiller',spec.selector.matchLabels.'app'='helm' --output yaml | sed 's@apiVersion: extensions/v1beta1@apiVersion: apps/v1@' | kubectl apply -f - deployment.apps/tiller-deploy created Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply service/tiller-deploy configured
这时候又会出现 ImagePullBackOff错误，同样地，重新修改tiller的yaml文件中的镜像，会自动更新。
重新安装后，再重新安装mysql。
[root@master ~]# helm install stable/mysql NAME: bumptious-fish LAST DEPLOYED: Thu Oct 31 15:21:43 2019 NAMESPACE: default STATUS: DEPLOYED RESOURCES: ==> v1/ConfigMap NAME DATA AGE bumptious-fish-mysql-test 1 0s ==> v1/Deployment NAME READY UP-TO-DATE AVAILABLE AGE bumptious-fish-mysql 0/1 1 0 0s ==> v1/PersistentVolumeClaim NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE bumptious-fish-mysql Pending 0s Filesystem ==> v1/Pod(related) NAME READY STATUS RESTARTS AGE bumptious-fish-mysql-68bf985647-649ms 0/1 Pending 0 0s ==> v1/Secret NAME TYPE DATA AGE bumptious-fish-mysql Opaque 2 0s ==> v1/Service NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE bumptious-fish-mysql ClusterIP 10.96.43.224 <none> 3306/TCP 0s NOTES: MySQL can be accessed via port 3306 on the following DNS name from within your cluster: bumptious-fish-mysql.default.svc.cluster.local To get your root password run: MYSQL_ROOT_PASSWORD=$(kubectl get secret --namespace default bumptious-fish-mysql -o jsonpath="{.data.mysql-root-password}" | base64 --decode; echo) To connect to your database: 1. Run an Ubuntu pod that you can use as a client: kubectl run -i --tty ubuntu --image=ubuntu:16.04 --restart=Never -- bash -il 2. Install the mysql client: $ apt-get update && apt-get install mysql-client -y 3. Connect using the mysql cli, then provide your password: $ mysql -h bumptious-fish-mysql -p To connect to your database directly from outside the K8s cluster: MYSQL_HOST=127.0.0.1 MYSQL_PORT=3306 # Execute the following command to route the connection: kubectl port-forward svc/bumptious-fish-mysql 3306 mysql -h ${MYSQL_HOST} -P${MYSQL_PORT} -u root -p${MYSQL_ROOT_PASSWORD}
查看mysql是否启动成功，我们是安装到默认命名空间下的，所以不需要带 -n参数了。
[root@master ~]# kubectl get pod NAME READY STATUS RESTARTS AGE bumptious-fish-mysql-68bf985647-649ms 0/1 Pending 0 40s
会发现一直是在 Pending状态，同样地，使用 kubectl describe命令查看详细的错误。
[root@master ~]# kubectl describe pod bumptious-fish-mysql-68bf985647-649ms # 省略... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling <unknown> default-scheduler pod has unbound immediate PersistentVolumeClaims (repeated 2 times) Warning FailedScheduling <unknown> default-scheduler pod has unbound immediate PersistentVolumeClaims (repeated 2 times)
看最后两行，是说pod没有绑定到PersistentVolumeClaims（简写pvc），我们再查查pvc。
[root@master ~]# kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE bumptious-fish-mysql Pending 2m18s
可以看到pvc也是 Pending状态，同样地，describe一下。
[root@master ~]# kubectl describe pvc bumptious-fish-mysql # 省略... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal FailedBinding 11s (x12 over 2m44s) persistentvolume-controller no persistent volumes available for this claim and no storage class is set
看最后一行，说是没有可用的persistent volume（简称pv）且没有设置storage class，这又是些什么东西呢。
这里就牵涉到k8s的整体架构了，针对有状态的pod，需要先关联一个pvc，pvc会与具体的pv关联，pv可以理解成是磁盘上的一块空间，或者是远程存储等。
关于pv的自动创建也有很多方案，比如nfs等，为了不增加复杂性，我们这里采用手动创建两块pv给mysql使用。
手动创建pv的脚本如下：
kubectl create -f - <<EOF kind: PersistentVolume apiVersion: v1 metadata: name: [名称] labels: type: local spec: capacity: storage: [使用的空间] accessModes: - ReadWriteOnce hostPath: path: "存储的路径" EOF
执行结果如下：
[root@master ~]# kubectl create -f - <<EOF > kind: PersistentVolume > apiVersion: v1 > metadata: > name: mysql-pv > labels: > type: local > spec: > capacity: > storage: 10Gi > accessModes: > - ReadWriteOnce > hostPath: > path: "/data/storage/mysql" > EOF persistentvolume/mysql-pv created [root@master ~]# kubectl get pv,pvc,pod NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE persistentvolume/mysql-pv 10Gi RWO Retain Bound default/bumptious-fish-mysql 67s NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE persistentvolumeclaim/bumptious-fish-mysql Bound mysql-pv 10Gi RWO 5m49s NAME READY STATUS RESTARTS AGE pod/bumptious-fish-mysql-68bf985647-649ms 1/1 Running 0 5m49s
这里的 kubectlgetpv,pvc,pod要过一会再执行，或者前面加一个watch，即 watch kubectlgetpv,pvc,pod，直到pod的状态变成Running且READY为1/1的时候表示mysql安装成功了。
到这里，基本可以确定mysql是安装成功了，但是没有连接进去谁又敢百分百说安装成功了呢，OK，我们下面尝试连接到mysql中。
连接mysql
查看mysql的service。
[root@master ~]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE bumptious-fish-mysql ClusterIP 10.96.43.224 <none> 3306/TCP 9m36s kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 23h
可以看到服务暴露的方式为 ClusterIP，我们把它修改为 NodePort，NodePort可以让我们从外部访问服务。
[root@master ~]# kubectl edit svc bumptious-fish-mysql # Please edit the object below. Lines beginning with a '#' will be ignored, # and an empty file will abort the edit. If an error occurs while saving this file will be # reopened with the relevant failures. # apiVersion: v1 kind: Service metadata: creationTimestamp: "2019-10-31T07:21:43Z" labels: app: bumptious-fish-mysql chart: mysql-1.4.0 heritage: Tiller release: bumptious-fish name: bumptious-fish-mysql namespace: default resourceVersion: "125451" selfLink: /api/v1/namespaces/default/services/bumptious-fish-mysql uid: 70be1da0-a10f-4439-9d3e-fd516f8785bf spec: clusterIP: 10.96.43.224 externalTrafficPolicy: Cluster ports: - name: mysql nodePort: 32684 port: 3306 protocol: TCP targetPort: mysql selector: app: bumptious-fish-mysql sessionAffinity: None type: NodePort status: loadBalancer: {}
找到spec.type，将其值修改为 NodePort，保存退出，查看自动生成的端口。
[root@master ~]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE bumptious-fish-mysql NodePort 10.96.43.224 <none> 3306:32684/TCP 18h kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 41h
可以看到生成的端口为 32684，再查看mysql的初始密码，命令在安装mysql的时候提示过，注意这里拷贝你安装时提示的命令。
[root@master ~]# echo $(kubectl get secret --namespace default bumptious-fish-mysql -o jsonpath="{.data.mysql-root-password}" | base64 --decode; echo) 3XvXkqckUE
在外部使用mysql客户端连接（当然，也可以在集群内部装一个mysql客户端，这样就不需要暴露外网ip及端口了）：
ip为任意worker节点的公网ip；
端口为上面获取到的32684；
用户名为root，密码为上面获取到的3XvXkqckUE；
C:> mysql -h[xx.xx.xx.xx] -P32684 -uroot -p3XvXkqckUE mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or g. Your MySQL connection id is 13658 Server version: 5.7.14 MySQL Community Server (GPL) Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or 'h' for help. Type 'c' to clear the current input statement. mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | performance_schema | | sys | +--------------------+ 4 rows in set (0.04 sec) mysql> create database test; Query OK, 1 row affected (0.04 sec) mysql> use test; Database changed mysql> create table user(id int, name varchar(50)); Query OK, 0 rows affected (0.05 sec) mysql> insert into user values(1,"hello"); Query OK, 1 row affected (0.05 sec) mysql> select * from user; +------+-------+ | id | name | +------+-------+ | 1 | hello | +------+-------+ 1 row in set (0.04 sec)
成功连接进去了，说明mysql安装成功了。
总结
本章成功安装了Helm，并使用Helm成功安装了mysql，这里有几个需要注意的点。
（1）k8s集群v1.16.x版本需要设置RBAC权限，才能成功安装Helm；
（2）网络问题导致无法成功拉取tiller镜像，所以需要手动修改tiller pod的yaml文件；
（3）mysql安装需要创建持久卷PV，我们这里使用手动创建的方式。
参考
官方文档：https://helm.sh/docs/using_helm
本文在官方文档的基础上补充了一些异常的处理。
