kubernetes系列教程（十二）詳解DaemonSet控制器
- 2019 年 10 月 31 日
- 筆記
寫在前面
上章節中介紹了Deployment，ReplicaSet，ReplicationController等副本控制器的使用和場景，下來介紹kubernetes系列教程控制器DaemonSet使用。
1. DaemonSet控制器
1.1 DaemonSet簡介
介紹DaemonSet時我們先來思考一個問題：相信大家都接觸過監控系統比如zabbix，監控系統需要在被監控機安裝一個agent，安裝agent通常會涉及到以下幾個場景：
- 所有節點都必須安裝agent以便採集監控數據
- 新加入的節點需要配置agent，手動或者運行腳本
- 節點下線後需要手動在監控系統中刪除
kubernetes中經常涉及到在node上安裝部署應用，它是如何解決上述的問題的呢？答案是DaemonSet。DaemonSet守護進程簡稱DS，適用於在所有節點或部分節點運行一個daemon守護進程，如監控我們安裝部署時網絡插件kube-flannel和kube-proxy，DaemonSet具有如下特點：
- DaemonSet確保所有節點運行一個Pod副本
- 指定節點運行一個Pod副本，通過標籤選擇器或者節點親和性
- 新增節點會自動在節點增加一個Pod
- 移除節點時垃圾回收機制會自動清理Pod
DaemonSet適用於每個node節點均需要部署一個守護進程的場景，常見的場景例如：
- 日誌採集agent，如fluentd或logstash
- 監控採集agent，如Prometheus Node Exporter,Sysdig Agent,Ganglia gmond
- 分佈式集群組件，如Ceph MON，Ceph OSD，glusterd，Hadoop Yarn NodeManager等
- k8s必要運行組件，如網絡flannel，weave，calico，kube-proxy等
安裝k8s時默認在kube-system命名空間已經安裝了有兩個DaemonSet，分別為kube-flannel-ds-amd64和kube-proxy，分別負責flannel overlay網絡的互通和service代理的實現，可以通過如下命令查看：
1. 查看kube-system命令空間的DaemonSet列表，當前集群有三個node節點，所以每個DS會運行三個Pod副本
[root@node-1 ~]# kubectl get ds -n kube-system NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE kube-flannel-ds-amd64 3 3 3 3 3 beta.kubernetes.io/arch=amd64 46d kube-proxy 3 3 3 3 3 beta.kubernetes.io/os=linux 46d
2. 查看Pod的副本情況，可以看到DaemonSet在每個節點都運行一個Pod
1.2 DaemonSet定義
DaemonSet的定義和Deployment定義使用相類似，需要定義apiVersion，Kind，metadata和spec屬性信息，spec中不需要定義replicas個數，spec.template即定義DS生成容器的模版信息，如下是運行一個fluentd-elasticsearch鏡像容器的daemon守護進程，運行在每個node上通過fluentd採集日誌上報到ElasticSearch。
1. 通過yaml文件定義DaemonSet
[root@node-1 happylau]# cat fluentd-es-daemonset.yaml apiVersion: apps/v1 #api版本信息 kind: DaemonSet #類型為DaemonSet metadata: #元數據信息 name: fluentd-elasticsearch namespace: kube-system #運行的命名空間 labels: k8s-app: fluentd-logging spec: #DS模版 selector: matchLabels: name: fluentd-elasticsearch template: metadata: labels: name: fluentd-elasticsearch spec: tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule containers: #容器信息 - name: fluentd-elasticsearch image: quay.io/fluentd_elasticsearch/fluentd:v2.5.2 resources: #resource資源 limits: memory: 200Mi requests: cpu: 100m memory: 200Mi volumeMounts: #掛載存儲，agent需要到這些目錄採集日誌 - name: varlog mountPath: /var/log - name: varlibdockercontainers mountPath: /var/lib/docker/containers readOnly: true terminationGracePeriodSeconds: 30 volumes: #將主機的目錄以hostPath的形式掛載到容器Pod中。 - name: varlog hostPath: path: /var/log - name: varlibdockercontainers hostPath: path: /var/lib/docker/containers
DaemonSet定義注意事項：
- daemonset.spec.template定義Pod的模板信息，包含的metadata信息需要和selector保持一致
- template必須定義RestartPolicy的策略，切策略值為Always，保障服務異常時能自動重啟恢復
- Pod運行在特定節點，支持指定調度策略，如nodeSelector，Node affinity，實現靈活調度
2. 生成DaemonSet
[root@node-1 happylau]# kubectl apply -f fluentd-es-daemonset.yaml daemonset.apps/fluentd-elasticsearch created
3. 查看DaemonSet列表
[root@node-1 happylau]# kubectl get daemonsets -n kube-system fluentd-elasticsearch NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE fluentd-elasticsearch 3 3 3 3 3 <none> 16s
4. 查看node上運行Pod的情況,在NODE列可以看到每個node都運行了一個Pod
[root@node-1 happylau]# kubectl get pods -n kube-system -o wide |grep fluentd fluentd-elasticsearch-blpqb 1/1 Running 0 3m7s 10.244.2.79 node-3 <none> <none> fluentd-elasticsearch-ksdlt 1/1 Running 0 3m7s 10.244.0.11 node-1 <none> <none> fluentd-elasticsearch-shtkh 1/1 Running 0 3m7s 10.244.1.64 node-2 <none> <none>
5. 查看DaemonSet詳情，可以看到DaemonSet支持RollingUpdate滾動更新策略
[root@node-1 happylau]# kubectl get daemonsets -n kube-system fluentd-elasticsearch -o yaml apiVersion: extensions/v1beta1 kind: DaemonSet metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"apps/v1","kind":"DaemonSet","metadata":{"annotations":{},"labels":{"k8s-app":"fluentd-logging"},"name":"fluentd-elasticsearch","namespace":"kube-system"},"spec":{"selector":{"matchLabels":{"name":"fluentd-elasticsearch"}},"template":{"metadata":{"labels":{"name":"fluentd-elasticsearch"}},"spec":{"containers":[{"image":"quay.io/fluentd_elasticsearch/fluentd:v2.5.2","name":"fluentd-elasticsearch","resources":{"limits":{"memory":"200Mi"},"requests":{"cpu":"100m","memory":"200Mi"}},"volumeMounts":[{"mountPath":"/var/log","name":"varlog"},{"mountPath":"/var/lib/docker/containers","name":"varlibdockercontainers","readOnly":true}]}],"terminationGracePeriodSeconds":30,"tolerations":[{"effect":"NoSchedule","key":"node-role.kubernetes.io/master"}],"volumes":[{"hostPath":{"path":"/var/log"},"name":"varlog"},{"hostPath":{"path":"/var/lib/docker/containers"},"name":"varlibdockercontainers"}]}}}} creationTimestamp: "2019-10-30T15:19:20Z" generation: 1 labels: k8s-app: fluentd-logging name: fluentd-elasticsearch namespace: kube-system resourceVersion: "6046222" selfLink: /apis/extensions/v1beta1/namespaces/kube-system/daemonsets/fluentd-elasticsearch uid: c2c02c48-9f93-48f3-9d6c-32bfa671db0e spec: revisionHistoryLimit: 10 selector: matchLabels: name: fluentd-elasticsearch template: metadata: creationTimestamp: null labels: name: fluentd-elasticsearch spec: containers: - image: quay.io/fluentd_elasticsearch/fluentd:v2.5.2 imagePullPolicy: IfNotPresent name: fluentd-elasticsearch resources: limits: memory: 200Mi requests: cpu: 100m memory: 200Mi terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /var/log name: varlog - mountPath: /var/lib/docker/containers name: varlibdockercontainers readOnly: true dnsPolicy: ClusterFirst restartPolicy: Always #重啟策略必須為Always，保障異常時能自動恢復 schedulerName: default-scheduler #默認調度策略 securityContext: {} terminationGracePeriodSeconds: 30 tolerations: - effect: NoSchedule key: node-role.kubernetes.io/master volumes: - hostPath: path: /var/log type: "" name: varlog - hostPath: path: /var/lib/docker/containers type: "" name: varlibdockercontainers templateGeneration: 1 updateStrategy: #滾動更新策略 rollingUpdate: maxUnavailable: 1 type: RollingUpdate status: currentNumberScheduled: 3 desiredNumberScheduled: 3 numberAvailable: 3 numberMisscheduled: 0 numberReady: 3 observedGeneration: 1 updatedNumberScheduled: 3
1.3 滾動更新與回滾
1. 更新鏡像至最新版本
[root@node-1 ~]# kubectl set image daemonsets fluentd-elasticsearch fluentd-elasticsearch=quay.io/fluentd_elasticsearch/fluentd:latest -n kube-system daemonset.extensions/fluentd-elasticsearch image updated
2. 查看滾動更新狀態
[root@node-1 ~]# kubectl rollout status daemonset -n kube-system fluentd-elasticsearch Waiting for daemon set "fluentd-elasticsearch" rollout to finish: 1 out of 3 new pods have been updated... Waiting for daemon set "fluentd-elasticsearch" rollout to finish: 1 out of 3 new pods have been updated... Waiting for daemon set "fluentd-elasticsearch" rollout to finish: 1 out of 3 new pods have been updated... Waiting for daemon set "fluentd-elasticsearch" rollout to finish: 2 out of 3 new pods have been updated... Waiting for daemon set "fluentd-elasticsearch" rollout to finish: 2 out of 3 new pods have been updated... Waiting for daemon set "fluentd-elasticsearch" rollout to finish: 2 out of 3 new pods have been updated... Waiting for daemon set "fluentd-elasticsearch" rollout to finish: 2 of 3 updated pods are available... daemon set "fluentd-elasticsearch" successfully rolled out
3. 查看DaemonSet詳情，可以看到DS滾動更新的過程：DaemonSet先將node上的pod刪除然後再創建
4. 查看DaemonSet滾動更新版本，REVSION 1為初始的版本
[root@node-1 ~]# kubectl rollout history daemonset -n kube-system fluentd-elasticsearch daemonset.extensions/fluentd-elasticsearch REVISION CHANGE-CAUSE 1 <none> 2 <none>
5. 更新回退，如果配置沒有符合到預期可以回滾到原始的版本
[root@node-1 ~]# kubectl rollout undo daemonset -n kube-system fluentd-elasticsearch --to-revision=1 daemonset.extensions/fluentd-elasticsearch rolled back
6. 確認版本回退情況
7. 觀察版本回退的過程，回退的過程和和滾動更新過程類似，先刪除Pod再創建
8. 刪除DaemonSet
[root@node-1 ~]# kubectl delete daemonsets -n kube-system fluentd-elasticsearch daemonset.extensions "fluentd-elasticsearch" deleted [root@node-1 ~]# kubectl get pods -n kube-system |grep fluentd fluentd-elasticsearch-d6f6f 0/1 Terminating 0 110m
1.4 DaemonSet調度
前面kubernetes系列教程（七）深入玩轉pod調度文章介紹了Pod的調度機制，DaemonSet通過kubernetes默認的調度器scheduler會在所有的node節點上運行一個Pod副本，可以通過如下三種方式將Pod運行在部分節點上：
- 指定nodeName節點運行
- 通過標籤運行nodeSelector
- 通過親和力調度node Affinity和node Anti-affinity
DaemonSet調度算法用於實現將Pod運行在特定的node節點上，如下以通過node affinity親和力將Pod調度到部分的節點上node-2上為例。
1. 為node添加一個app=web的labels
[root@node-1 happylau]# kubectl get nodes --show-labels NAME STATUS ROLES AGE VERSION LABELS node-1 Ready master 47d v1.15.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=node-1,kubernetes.io/os=linux,node-role.kubernetes.io/master= node-2 Ready <none> 47d v1.15.3 app=web,beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=node-2,kubernetes.io/os=linux node-3 Ready <none> 47d v1.15.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=node-3,kubernetes.io/os=linux
2. 添加node affinity親和力調度算法，requiredDuringSchedulingIgnoredDuringExecution設置基本需要滿足條件，preferredDuringSchedulingIgnoredDuringExecution設置優選滿足條件
[root@node-1 happylau]# cat fluentd-es-daemonset.yaml apiVersion: apps/v1 kind: DaemonSet metadata: name: fluentd-elasticsearch namespace: kube-system labels: k8s-app: fluentd-logging spec: selector: matchLabels: name: fluentd-elasticsearch template: metadata: labels: name: fluentd-elasticsearch spec: tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule containers: - name: fluentd-elasticsearch image: quay.io/fluentd_elasticsearch/fluentd:v2.5.2 resources: limits: memory: 200Mi requests: cpu: 100m memory: 200Mi volumeMounts: - name: varlog mountPath: /var/log - name: varlibdockercontainers mountPath: /var/lib/docker/containers readOnly: true affinity: nodeAffinity: preferredDuringSchedulingIgnoredDuringExecution: #優先滿足條件 - weight: 1 preference: matchExpressions: - key: app operator: In values: - web requiredDuringSchedulingIgnoredDuringExecution: #要求滿足條件 nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - node-2 - node-3 terminationGracePeriodSeconds: 30 volumes: - name: varlog hostPath: path: /var/log - name: varlibdockercontainers hostPath: path: /var/lib/docker/containers
3. 生成DS，並查看列表
[root@node-1 happylau]# kubectl delete ds -n kube-system fluentd-elasticsearch daemonset.extensions "fluentd-elasticsearch" deleted [root@node-1 happylau]# kubectl get daemonsets -n kube-system fluentd-elasticsearch NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE fluentd-elasticsearch 1 1 1 1 1 <none> 112s
4. 校驗Pod運行的情況，DaemonSet的Pod調度到node-2節點上
[root@node-1 happylau]# kubectl get pods -n kube-system -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES <none> fluentd-elasticsearch-9kngs 1/1 Running 0 2m39s 10.244.1.82 node-2 <none> <none>
寫在最後
本文介紹了kubernetes中DaemonSet控制器，DS控制器能確保所有的節點運行一個特定的daemon守護進程，此外通過nodeSelector或node Affinity能夠實現將Pod調度到特定的node節點。
參考文檔
DaemonSet：https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/
當你的才華撐不起你的野心時，你就應該靜下心來學習
