Kubernetes實戰之部署ELK Stack收集平台日誌
- 2019 年 12 月 11 日
- 筆記
——不積跬步，無以至千里；不積細流，不以成江河
主要內容
- 1 ELK概念
- 2 K8S需要收集哪些日誌
- 3 ELK Stack日誌方案
- 4 容器中的日誌怎麼收集
- 5 K8S平台中應用日誌收集
準備環境
一套正常運行的k8s集群，kubeadm安裝部署或者二進位部署即可
ip地址 | 角色 | 備註 |
|---|---|---|
192.168.73.136 | nfs | |
192.168.73.138 | k8s-master | |
192.168.73.139 | k8s-node01 | |
192.168.73.140 | k8s-node02 |
1 ELK概念
ELK是Elasticsearch、Logstash、Kibana三大開源框架首字母大寫簡稱。市面上也被成為Elastic Stack。其中Elasticsearch是一個基於Lucene、分散式、通過Restful方式進行交互的近實時搜索平台框架。像類似百度、Google這種大數據全文搜索引擎的場景都可以使用Elasticsearch作為底層支援框架，可見Elasticsearch提供的搜索能力確實強大,市面上很多時候我們簡稱Elasticsearch為es。Logstash是ELK的中央數據流引擎，用於從不同目標（文件/數據存儲/MQ）收集的不同格式數據，經過過濾後支援輸出到不同目的地（文件/MQ/redis/elasticsearch/kafka等）。Kibana可以將elasticsearch的數據通過友好的頁面展示出來，提供實時分析的功能。
通過上面對ELK簡單的介紹，我們知道了ELK字面意義包含的每個開源框架的功能。市面上很多開發只要提到ELK能夠一致說出它是一個日誌分析架構技術棧總稱，但實際上ELK不僅僅適用於日誌分析，它還可以支援其它任何數據分析和收集的場景，日誌分析和收集只是更具有代表性。並非唯一性。我們本教程主要也是圍繞通過ELK如何搭建一個生產級的日誌分析平台來講解ELK的使用。 官方網站：https://www.elastic.co/cn/products/
2 日誌管理平台
在過往的單體應用時代，我們所有組件都部署到一台伺服器中，那時日誌管理平台的需求可能並沒有那麼強烈，我們只需要登錄到一台伺服器通過shell命令就可以很方便的查看系統日誌，並快速定位問題。隨著互聯網的發展，互聯網已經全面滲入到生活的各個領域，使用互聯網的用戶量也越來越多，單體應用已不能夠支援龐大的用戶的並發量，尤其像中國這種人口大國。那麼將單體應用進行拆分，通過水平擴展來支援龐大用戶的使用迫在眉睫，微服務概念就是在類似這樣的階段誕生，在微服務盛行的互聯網技術時代，單個應用被拆分為多個應用，每個應用集群部署進行負載均衡，那麼如果某項業務發生系統錯誤，開發或運維人員還是以過往單體應用方式登錄一台一台登錄伺服器查看日誌來定位問題，這種解決線上問題的效率可想而知。日誌管理平台的建設就顯得極其重要。通過Logstash去收集每台伺服器日誌文件，然後按定義的正則模板過濾後傳輸到Kafka或redis，然後由另一個Logstash從KafKa或redis讀取日誌存儲到elasticsearch中創建索引，最後通過Kibana展示給開發者或運維人員進行分析。這樣大大提升了運維線上問題的效率。除此之外，還可以將收集的日誌進行大數據分析，得到更有價值的數據給到高層進行決策。
3 K8S需要收集哪些日誌
這裡只是以主要收集日誌為例：
- K8S系統的組件日誌
- K8S Cluster裡面部署的應用程式日誌 -標準輸出 -日誌文件
4 K8S中的ELK Stack日誌採集方案
- 方案一：Node上部署一個日誌收集程式 使用DaemonSet的方式去給每一個node上部署日誌收集程式logging-agent 然後使用這個agent對本node節點上的/var/log和/var/lib/docker/containers/兩個目錄下的日誌進行採集 或者把Pod中容器日誌目錄掛載到宿主機統一目錄上，這樣進行收集
- 方案二：Pod中附加專用日誌收集的容器 每個運行應用程式的Pod中增加一個日誌收集容器，使用emtyDir共享日誌目錄讓日誌收集程式讀取到。
- 方案三：應用程式直接推送日誌 這個方案需要開發在程式碼中修改直接把應用程式直接推送到遠程的存儲上，不再輸入出控制台或者本地文件了，使用不太多，超出Kubernetes範圍
方式 | 優點 | 缺點 |
|---|---|---|
方案一：Node上部署一個日誌收集程式 | 每個Node僅需部署一個日誌收集程式，資源消耗少，對應用無侵入 | 應用程式日誌需要寫到標準輸出和標準錯誤輸出，不支援多行日誌 |
方案二：Pod中附加專用日誌收集的容器 | 低耦合 | 每個Pod啟動一個日誌收集代理，增加資源消耗，並增加運維維護成本 |
方案三：應用程式直接推送日誌 | 無需額外收集工具 | 浸入應用，增加應用複雜度 |
5 單節點方式部署ELK
單節點部署ELK的方法較簡單，可以參考下面的yaml編排文件，整體就是創建一個es，然後創建kibana的可視化展示，創建一個es的service服務，然後通過ingress的方式對外暴露域名訪問
首先，編寫es的yaml，這裡部署的是單機版，在k8s集群內中，通常當日誌量每天超過20G以上的話，還是建議部署在k8s集群外部，支援分散式集群的架構，這裡使用的是有狀態部署的方式，並且使用動態存儲進行持久化，需要提前創建好存儲類，才能運行該yaml
[root@k8s-master fek]# vim elasticsearch.yaml apiVersion: apps/v1 kind: StatefulSet metadata: name: elasticsearch namespace: kube-system labels: k8s-app: elasticsearch spec: serviceName: elasticsearch selector: matchLabels: k8s-app: elasticsearch template: metadata: labels: k8s-app: elasticsearch spec: containers: - image: elasticsearch:7.3.1 name: elasticsearch resources: limits: cpu: 1 memory: 2Gi requests: cpu: 0.5 memory: 500Mi env: - name: "discovery.type" value: "single-node" - name: ES_JAVA_OPTS value: "-Xms512m -Xmx2g" ports: - containerPort: 9200 name: db protocol: TCP volumeMounts: - name: elasticsearch-data mountPath: /usr/share/elasticsearch/data volumeClaimTemplates: - metadata: name: elasticsearch-data spec: storageClassName: "managed-nfs-storage" accessModes: [ "ReadWriteOnce" ] resources: requests: storage: 20Gi --- apiVersion: v1 kind: Service metadata: name: elasticsearch namespace: kube-system spec: clusterIP: None ports: - port: 9200 protocol: TCP targetPort: db selector: k8s-app: elasticsearch
使用剛才編寫好的yaml文件創建Elasticsearch，然後檢查是否啟動，如下所示能看到一個elasticsearch-0 的pod副本被創建，正常運行；如果不能正常啟動可以使用kubectl describe查看詳細描述，排查問題
[root@k8s-master fek]# kubectl get pod -n kube-system NAME READY STATUS RESTARTS AGE coredns-5bd5f9dbd9-95flw 1/1 Running 0 17h elasticsearch-0 1/1 Running 1 16m php-demo-85849d58df-4bvld 2/2 Running 2 18h php-demo-85849d58df-7tbb2 2/2 Running 0 17h
然後，需要部署一個Kibana來對搜集到的日誌進行可視化展示，使用Deployment的方式編寫一個yaml，使用ingress對外進行暴露訪問，直接引用了es
[root@k8s-master fek]# vim kibana.yaml apiVersion: apps/v1 kind: Deployment metadata: name: kibana namespace: kube-system labels: k8s-app: kibana spec: replicas: 1 selector: matchLabels: k8s-app: kibana template: metadata: labels: k8s-app: kibana spec: containers: - name: kibana image: kibana:7.3.1 resources: limits: cpu: 1 memory: 500Mi requests: cpu: 0.5 memory: 200Mi env: - name: ELASTICSEARCH_HOSTS value: http://elasticsearch:9200 ports: - containerPort: 5601 name: ui protocol: TCP --- apiVersion: v1 kind: Service metadata: name: kibana namespace: kube-system spec: ports: - port: 5601 protocol: TCP targetPort: ui selector: k8s-app: kibana --- apiVersion: extensions/v1beta1 kind: Ingress metadata: name: kibana namespace: kube-system spec: rules: - host: kibana.ctnrs.com http: paths: - path: / backend: serviceName: kibana servicePort: 5601
使用剛才編寫好的yaml創建kibana，可以看到最後生成了一個kibana-b7d98644-lshsz的pod，並且正常運行
[root@k8s-master fek]# kubectl apply -f kibana.yaml deployment.apps/kibana created service/kibana created ingress.extensions/kibana created [root@k8s-master fek]# kubectl get pod -n kube-system NAME READY STATUS RESTARTS AGE coredns-5bd5f9dbd9-95flw 1/1 Running 0 17h elasticsearch-0 1/1 Running 1 16m kibana-b7d98644-48gtm 1/1 Running 1 17h php-demo-85849d58df-4bvld 2/2 Running 2 18h php-demo-85849d58df-7tbb2 2/2 Running 0 17h
最後，需要編寫yaml在每個node上創建一個ingress-nginx控制器來對外提供訪問
[root@k8s-master demo2]# vim mandatory.yaml apiVersion: v1 kind: Namespace metadata: name: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx --- kind: ConfigMap apiVersion: v1 metadata: name: nginx-configuration namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx --- kind: ConfigMap apiVersion: v1 metadata: name: tcp-services namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx --- kind: ConfigMap apiVersion: v1 metadata: name: udp-services namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx --- apiVersion: v1 kind: ServiceAccount metadata: name: nginx-ingress-serviceaccount namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRole metadata: name: nginx-ingress-clusterrole labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx rules: - apiGroups: - "" resources: - configmaps - endpoints - nodes - pods - secrets verbs: - list - watch - apiGroups: - "" resources: - nodes verbs: - get - apiGroups: - "" resources: - services verbs: - get - list - watch - apiGroups: - "" resources: - events verbs: - create - patch - apiGroups: - "extensions" - "networking.k8s.io" resources: - ingresses verbs: - get - list - watch - apiGroups: - "extensions" - "networking.k8s.io" resources: - ingresses/status verbs: - update --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: Role metadata: name: nginx-ingress-role namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx rules: - apiGroups: - "" resources: - configmaps - pods - secrets - namespaces verbs: - get - apiGroups: - "" resources: - configmaps resourceNames: # Defaults to "<election-id>-<ingress-class>" # Here: "<ingress-controller-leader>-<nginx>" # This has to be adapted if you change either parameter # when launching the nginx-ingress-controller. - "ingress-controller-leader-nginx" verbs: - get - update - apiGroups: - "" resources: - configmaps verbs: - create - apiGroups: - "" resources: - endpoints verbs: - get --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: RoleBinding metadata: name: nginx-ingress-role-nisa-binding namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: nginx-ingress-role subjects: - kind: ServiceAccount name: nginx-ingress-serviceaccount namespace: ingress-nginx --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: nginx-ingress-clusterrole-nisa-binding labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: nginx-ingress-clusterrole subjects: - kind: ServiceAccount name: nginx-ingress-serviceaccount namespace: ingress-nginx --- apiVersion: apps/v1 kind: DaemonSet metadata: name: nginx-ingress-controller namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx spec: selector: matchLabels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx template: metadata: labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx annotations: prometheus.io/port: "10254" prometheus.io/scrape: "true" spec: serviceAccountName: nginx-ingress-serviceaccount hostNetwork: true containers: - name: nginx-ingress-controller image: lizhenliang/nginx-ingress-controller:0.20.0 args: - /nginx-ingress-controller - --configmap=$(POD_NAMESPACE)/nginx-configuration - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services - --udp-services-configmap=$(POD_NAMESPACE)/udp-services - --publish-service=$(POD_NAMESPACE)/ingress-nginx - --annotations-prefix=nginx.ingress.kubernetes.io securityContext: allowPrivilegeEscalation: true capabilities: drop: - ALL add: - NET_BIND_SERVICE # www-data -> 33 runAsUser: 33 env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace ports: - name: http containerPort: 80 - name: https containerPort: 443 livenessProbe: failureThreshold: 3 httpGet: path: /healthz port: 10254 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 10 readinessProbe: failureThreshold: 3 httpGet: path: /healthz port: 10254 scheme: HTTP periodSeconds: 10 successThreshold: 1 timeoutSeconds: 10 ---
創建ingress控制器，可以看到使用的DaemonSet 的方式在每一個node都部署了ingress控制器，我們可以在本地host中綁定任意一個node ip，然後使用域名都可以訪問
[root@k8s-master demo2]# kubectl apply -f mandatory.yaml [root@k8s-master demo2]# kubectl get pod -n ingress-nginx NAME READY STATUS RESTARTS AGE nginx-ingress-controller-98769 1/1 Running 6 13h nginx-ingress-controller-n6wpq 1/1 Running 0 13h nginx-ingress-controller-tbfxq 1/1 Running 29 13h nginx-ingress-controller-trxnj 1/1 Running 6 13h
綁定本機hosts，訪問域名驗證 windows系統，hosts文件地址：C:WindowsSystem32driversetc，Mac系統sudo vi /private/etc/hosts 編輯hosts文件，在底部加入域名和ip，用於解析，這個ip地址為任意node節點ip地址 加入如下命令，然後保存
192.168.73.139 kibana.ctnrs.com
最後在瀏覽器中，輸入kibana.ctnrs.com，就會進入kibana的web介面，已設置了不需要進行登陸，當前頁面都是全英文模式，可以修改上網搜一下修改配置文件的位置，建議使用英文版本
5.1 方案一：Node上部署一個filebeat採集器採集k8s組件日誌
es和kibana部署好了之後，我們如何採集pod日誌呢，我們採用方案一的方式，首先在每一個node上中部署一個filebeat的採集器，採用的是7.3.1版本，因為filebeat是對k8s有支援，可以連接api給pod日誌打標籤，所以yaml中需要進行認證，最後在配置文件中對獲取數據採集了之後輸入到es中，已在yaml中配置好
[root@k8s-master fek]# vim filebeat-kubernetes.yaml --- apiVersion: v1 kind: ConfigMap metadata: name: filebeat-config namespace: kube-system labels: k8s-app: filebeat data: filebeat.yml: |- filebeat.config: inputs: # Mounted `filebeat-inputs` configmap: path: ${path.config}/inputs.d/*.yml # Reload inputs configs as they change: reload.enabled: false modules: path: ${path.config}/modules.d/*.yml # Reload module configs as they change: reload.enabled: false # To enable hints based autodiscover, remove `filebeat.config.inputs` configuration and uncomment this: #filebeat.autodiscover: # providers: # - type: kubernetes # hints.enabled: true output.elasticsearch: hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}'] --- apiVersion: v1 kind: ConfigMap metadata: name: filebeat-inputs namespace: kube-system labels: k8s-app: filebeat data: kubernetes.yml: |- - type: docker containers.ids: - "*" processors: - add_kubernetes_metadata: in_cluster: true --- apiVersion: extensions/v1beta1 kind: DaemonSet metadata: name: filebeat namespace: kube-system labels: k8s-app: filebeat spec: template: metadata: labels: k8s-app: filebeat spec: serviceAccountName: filebeat terminationGracePeriodSeconds: 30 containers: - name: filebeat image: elastic/filebeat:7.3.1 args: [ "-c", "/etc/filebeat.yml", "-e", ] env: - name: ELASTICSEARCH_HOST value: elasticsearch - name: ELASTICSEARCH_PORT value: "9200" securityContext: runAsUser: 0 # If using Red Hat OpenShift uncomment this: #privileged: true resources: limits: memory: 200Mi requests: cpu: 100m memory: 100Mi volumeMounts: - name: config mountPath: /etc/filebeat.yml readOnly: true subPath: filebeat.yml - name: inputs mountPath: /usr/share/filebeat/inputs.d readOnly: true - name: data mountPath: /usr/share/filebeat/data - name: varlibdockercontainers mountPath: /var/lib/docker/containers readOnly: true volumes: - name: config configMap: defaultMode: 0600 name: filebeat-config - name: varlibdockercontainers hostPath: path: /var/lib/docker/containers - name: inputs configMap: defaultMode: 0600 name: filebeat-inputs # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart - name: data hostPath: path: /var/lib/filebeat-data type: DirectoryOrCreate --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: filebeat subjects: - kind: ServiceAccount name: filebeat namespace: kube-system roleRef: kind: ClusterRole name: filebeat apiGroup: rbac.authorization.k8s.io --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRole metadata: name: filebeat labels: k8s-app: filebeat rules: - apiGroups: [""] # "" indicates the core API group resources: - namespaces - pods verbs: - get - watch - list --- apiVersion: v1 kind: ServiceAccount metadata: name: filebeat namespace: kube-system labels: k8s-app: filebeat ---除此之外，需要對k8s組件的日誌進行採集，因為我的環境是用的kubeadm進行部署的，因此我的組件日誌都在/var/log/message裡面，因此我們還需要部署一個採集k8s組件日誌的pod副本，自定義了索引k8s-module-%{+yyyy.MM.dd}，編寫yaml如下：
[root@k8s-master elk]# vim k8s-logs.yaml apiVersion: v1 kind: ConfigMap metadata: name: k8s-logs-filebeat-config namespace: kube-system data: filebeat.yml: | filebeat.inputs: - type: log paths: - /var/log/messages fields: app: k8s type: module fields_under_root: true setup.ilm.enabled: false setup.template.name: "k8s-module" setup.template.pattern: "k8s-module-*" output.elasticsearch: hosts: ['elasticsearch.kube-system:9200'] index: "k8s-module-%{+yyyy.MM.dd}" --- apiVersion: apps/v1 kind: DaemonSet metadata: name: k8s-logs namespace: kube-system spec: selector: matchLabels: project: k8s app: filebeat template: metadata: labels: project: k8s app: filebeat spec: containers: - name: filebeat image: elastic/filebeat:7.3.1 args: [ "-c", "/etc/filebeat.yml", "-e", ] resources: requests: cpu: 100m memory: 100Mi limits: cpu: 500m memory: 500Mi securityContext: runAsUser: 0 volumeMounts: - name: filebeat-config mountPath: /etc/filebeat.yml subPath: filebeat.yml - name: k8s-logs mountPath: /var/log/messages volumes: - name: k8s-logs hostPath: path: /var/log/messages - name: filebeat-config configMap: name: k8s-logs-filebeat-config創建編寫好的yaml，並且檢查是否成功創建，能看到兩個命名為k8s-log-xx的pod副本分別創建在兩個nodes上
[root@k8s-master elk]# kubectl apply -f k8s-logs.yaml [root@k8s-master elk]# kubectl get pod -n kube-system NAME READY STATUS RESTARTS AGE coredns-5bd5f9dbd9-8zdn5 1/1 Running 0 10h elasticsearch-0 1/1 Running 1 13h filebeat-2q5tz 1/1 Running 0 13h filebeat-k6m27 1/1 Running 2 13h k8s-logs-52xgk 1/1 Running 0 5h45m k8s-logs-jpkqp 1/1 Running 0 5h45m kibana-b7d98644-tllmm 1/1 Running 0 10h
5.1.1 在kibana的web介面進行配置日誌可視化
首先打開kibana的web介面，點擊左邊菜單欄匯中的設置，然後點擊在Kibana下面的索引按鈕，然後點擊左上角的然後根據如圖所示分別創建一個filebeat-7.3.1-和k8s-module-的filebeat採集器的索引匹配
然後按照時間過濾,完成創建
索引匹配創建以後，點擊左邊最上面的菜單Discove，然後可以在左側看到我們剛才創建的索引，然後就可以在下面添加要展示的標籤，也可以對標籤進行篩選，最終效果如圖所示，可以看到採集到的日誌的所有資訊
在其中一個node上，輸入echo hello logs >>/var/log/messages，然後在web上選擇k8s-module-*的索引匹配，就可以在採集到的日誌中看到剛才輸入的hello logs，則證明採集成功，如圖所示
5.2 方案二：Pod中附加專用日誌收集的容器
我們也可以使用方案的方式，通過在pod中注入一個日誌收集的容器來採集pod的日誌，以一個php-demo的應用為例，使用emptyDir的方式把日誌目錄共享給採集器的容器收集，編寫nginx-deployment.yaml ，直接在pod中加入filebeat的容器，並且自定義索引為nginx-access-%{+yyyy.MM.dd}
[root@k8s-master fek]# vim nginx-deployment.yaml apiVersion: apps/v1beta1 kind: Deployment metadata: name: php-demo namespace: kube-system spec: replicas: 2 selector: matchLabels: project: www app: php-demo template: metadata: labels: project: www app: php-demo spec: imagePullSecrets: - name: registry-pull-secret containers: - name: nginx image: lizhenliang/nginx-php ports: - containerPort: 80 name: web protocol: TCP resources: requests: cpu: 0.5 memory: 256Mi limits: cpu: 1 memory: 1Gi livenessProbe: httpGet: path: /status.html port: 80 initialDelaySeconds: 20 timeoutSeconds: 20 readinessProbe: httpGet: path: /status.html port: 80 initialDelaySeconds: 20 timeoutSeconds: 20 volumeMounts: - name: nginx-logs mountPath: /usr/local/nginx/logs - name: filebeat image: elastic/filebeat:7.3.1 args: [ "-c", "/etc/filebeat.yml", "-e", ] resources: limits: memory: 500Mi requests: cpu: 100m memory: 100Mi securityContext: runAsUser: 0 volumeMounts: - name: filebeat-config mountPath: /etc/filebeat.yml subPath: filebeat.yml - name: nginx-logs mountPath: /usr/local/nginx/logs volumes: - name: nginx-logs emptyDir: {} - name: filebeat-config configMap: name: filebeat-nginx-config --- apiVersion: v1 kind: ConfigMap metadata: name: filebeat-nginx-config namespace: kube-system data: filebeat.yml: |- filebeat.inputs: - type: log paths: - /usr/local/nginx/logs/access.log # tags: ["access"] fields: app: www type: nginx-access fields_under_root: true setup.ilm.enabled: false setup.template.name: "nginx-access" setup.template.pattern: "nginx-access-*" output.elasticsearch: hosts: ['elasticsearch.kube-system:9200'] index: "nginx-access-%{+yyyy.MM.dd}"創建剛才編寫的nginx-deployment.yaml，創建成果之後會在kube-system命名空間下面pod/php-demo-58d89c9bc4-r5692的2個pod副本，還有一個對外暴露的service/php-demo
[root@k8s-master elk]# kubectl apply -f nginx-deployment.yaml [root@k8s-master fek]# kubectl get pod -n kube-system NAME READY STATUS RESTARTS AGE coredns-5bd5f9dbd9-8zdn5 1/1 Running 0 20helasticsearch-0 1/1 Running 1 23hfilebeat-46nvd 1/1 Running 0 23m filebeat-sst8m 1/1 Running 0 23m k8s-logs-52xgk 1/1 Running 0 15hk8s-logs-jpkqp 1/1 Running 0 15hkibana-b7d98644-tllmm 1/1 Running 0 20hphp-demo-85849d58df-d98gv 2/2 Running 0 26m php-demo-85849d58df-sl5ss 2/2 Running 0 26m
然後打開kibana的web，按照剛才的辦法繼續添加一個索引匹配nginx-access-*，如圖所示
最後點擊左邊最上面的菜單Discove，然後可以在左側看到我們剛才創建的索引匹配，下拉選擇nginx-access-*，然後就可以在下面添加要展示的標籤，也可以對標籤進行篩選，最終效果如圖所示，可以看到採集到的日誌的所有資訊
以上文章來源於阿冬的運維技術棧，作者Double冬
