K8S之prometheus-operator監控
- 2022 年 11 月 16 日
- 筆記
- k8s, Prometheus
prometheus-operator
1. Prometheus Operator介紹
2016年年末,CoreOs
引入了Operator 模式,並發布了Prometheus Operator 作為Operator模式
的工作示例。Prometheus Operator
自動創建和管理Prometheus
監控實例。
Prometheus Operator
的任務是使得在Kubernetes
運行Prometheus
僅可能容易,同時保留可配置性以及使Kubernetes
配置原生。
Prometheus Operator
使我們的生活更容易——部署和維護。
2. 它如何工作
為了理解這個問題,我們首先需要了解Prometheus Operator
得工作原理。
Prometheus Operator
架構圖.
我們成功部署 Prometheus Operator
後可以看到一個新的CRDs(Custom Resource Defination):
- Prometheus,定義一個期望的
Prometheus deployment
。 - ServiceMonitor,聲明式指定應該如何監控服務組;
Operator
根據定義自動創建Prometheusscrape
配置。 - Alertmanager,定義期望的
Alertmanager deployment
。
當服務新版本更新時,將會常見一個新Pod
。Prometheus
監控k8s API
,因此當它檢測到這種變化時,它將為這個新服務(pod)創建一組新的配置。
3. ServiceMonitor
Prometheus Operator
使用一個CRD
,叫做 ServiceMonitor 將配置抽象到目標。
下面是個ServiceMonitor
的示例:
apiVersion: monitoring.coreos.com/v1alpha1
kind: ServiceMonitor
metadata:
name: frontend
labels:
tier: frontend
spec:
selector:
matchLabels:
tier: frontend
endpoints:
- port: web # 指定exporter埠,這裡指定的是endpoint的名稱
interval: 10s # 刷新間隔時間
這僅僅是定義一組服務應該如何被監控。現在我們需要定義一個包含了該ServiceMonitor
的Prometheus
實例到其配置:
apiVersion: monitoring.coreos.com/v1alpha1
kind: Prometheus
metadata:
name: prometheus-frontend
labels:
prometheus: frontend
spec:
version: v1.3.0
#定義應包括標籤為「tier=frontend」的所有ServiceMonitor 到伺服器的配置中
serviceMonitors:
- selector:
matchLabels:
tier: frontend
現在Prometheus
將會監控每個帶有tier: frontend
label的服務。
4. helm安裝
先決條件:
- 部署了
Helm
準備好動手操作:
helm repo add coreos //s3-eu-west-1.amazonaws.com/coreos-charts/stable/
helm install coreos/prometheus-operator --name prometheus-operator --namespace monitoring
到目前為止,我們已經在我們的集群中安裝了Prometheus Operator
的TPR
。
現在我們來部署Prometheus
,Alertmanager
和Grafana
。
TIP: 當我使用一個龐大的
Helm Charts
時,我更傾向於創建一個獨立的value.yaml
文件將包含我所有自定義的變更。這麼做使我和同事為後期的變化和修改更容易。
helm install coreos/kube-prometheus --name kube-prometheus \
-f my_changes/prometheus.yaml \
-f my_changes/grafana.yaml \
-f my_changes/alertmanager.yaml
檢查一切是否運行正常
kubectl -n monitoring get po
NAME READY STATUS RESTARTS AGE
alertmanager-kube-prometheus-0 2/2 Running 0 1h
kube-prometheus-exporter-kube-state-68dbb4f7c9-tr6rp 2/2 Running 0 1h
kube-prometheus-exporter-node-bqcj4 1/1 Running 0 1h
kube-prometheus-exporter-node-jmcq2 1/1 Running 0 1h
kube-prometheus-exporter-node-qnzsn 1/1 Running 0 1h
kube-prometheus-exporter-node-v4wn8 1/1 Running 0 1h
kube-prometheus-exporter-node-x5226 1/1 Running 0 1h
kube-prometheus-exporter-node-z996c 1/1 Running 0 1h
kube-prometheus-grafana-54c96ffc77-tjl6g 2/2 Running 0 1h
prometheus-kube-prometheus-0 2/2 Running 0 1h
prometheus-operator-1591343780-5vb5q 1/1 Running 0 1h
訪問下Prometheus UI
看一下Targets
頁面:
kubectl -n monitoring port-forward prometheus-kube-prometheus-0 9090
Forwarding from 127.0.0.1:9090 -> 9090
瀏覽器展示如下:
5. yaml文件安裝
此安裝方法本人親測有效,用到的yaml文件都打包好了。解壓之後直接kubectl apply即可用。會自動監控當前集群的所有node節點和pod。只需更改yaml文件中需要用到的鏡像。我這裡都推到了公司公網harbor倉庫。部分鏡像已經打成tar包。直接docker load -i即可用。
kube-state.tar.gz
webhook-dingtalk.tar.gz
prometheus-adapter.tar.gz
5.1 安裝
#軟體包集成了node Exporter alertmanager grafana prometheus ingress 所有服務的配置,只需解壓到K8S master中。
[root@lecode-k8s-master monitor]# ll
total 1820
-rw-r--r-- 1 root root 875 Mar 11 2022 alertmanager-alertmanager.yaml
-rw-r--r-- 1 root root 515 Mar 11 2022 alertmanager-podDisruptionBudget.yaml
-rw-r--r-- 1 root root 4337 Mar 11 2022 alertmanager-prometheusRule.yaml
-rw-r--r-- 1 root root 1483 Mar 14 2022 alertmanager-secret.yaml
-rw-r--r-- 1 root root 301 Mar 11 2022 alertmanager-serviceAccount.yaml
-rw-r--r-- 1 root root 540 Mar 11 2022 alertmanager-serviceMonitor.yaml
-rw-r--r-- 1 root root 614 Mar 11 2022 alertmanager-service.yaml
drwxr-x--- 2 root root 4096 Oct 25 13:49 backsvc #這裡是grafana的service配置。nodeport模式。用於外部訪問。選擇使用
-rw-r--r-- 1 root root 278 Mar 11 2022 blackbox-exporter-clusterRoleBinding.yaml
-rw-r--r-- 1 root root 287 Mar 11 2022 blackbox-exporter-clusterRole.yaml
-rw-r--r-- 1 root root 1392 Mar 11 2022 blackbox-exporter-configuration.yaml
-rw-r--r-- 1 root root 3081 Mar 11 2022 blackbox-exporter-deployment.yaml
-rw-r--r-- 1 root root 96 Mar 11 2022 blackbox-exporter-serviceAccount.yaml
-rw-r--r-- 1 root root 680 Mar 11 2022 blackbox-exporter-serviceMonitor.yaml
-rw-r--r-- 1 root root 540 Mar 11 2022 blackbox-exporter-service.yaml
-rw-r--r-- 1 root root 2521 Oct 25 13:36 dingtalk-dep.yaml
-rw-r--r-- 1 root root 721 Mar 11 2022 grafana-dashboardDatasources.yaml
-rw-r--r-- 1 root root 1448347 Mar 11 2022 grafana-dashboardDefinitions.yaml
-rw-r--r-- 1 root root 625 Mar 11 2022 grafana-dashboardSources.yaml
-rw-r--r-- 1 root root 8098 Mar 11 2022 grafana-deployment.yaml
-rw-r--r-- 1 root root 86 Mar 11 2022 grafana-serviceAccount.yaml
-rw-r--r-- 1 root root 398 Mar 11 2022 grafana-serviceMonitor.yaml
-rw-r--r-- 1 root root 468 Mar 30 2022 grafana-service.yaml
drwxr-xr-x 2 root root 4096 Oct 25 13:32 ingress #這裡ingress資源也是可以直接用,可以把Prometheus和grafana服務暴露在外部。
-rw-r--r-- 1 root root 2639 Mar 14 2022 kube-prometheus-prometheusRule.yaml
-rw-r--r-- 1 root root 3380 Mar 14 2022 kube-prometheus-prometheusRule.yamlbak
-rw-r--r-- 1 root root 63531 Mar 11 2022 kubernetes-prometheusRule.yaml
-rw-r--r-- 1 root root 6912 Mar 11 2022 kubernetes-serviceMonitorApiserver.yaml
-rw-r--r-- 1 root root 425 Mar 11 2022 kubernetes-serviceMonitorCoreDNS.yaml
-rw-r--r-- 1 root root 6431 Mar 11 2022 kubernetes-serviceMonitorKubeControllerManager.yaml
-rw-r--r-- 1 root root 7629 Mar 11 2022 kubernetes-serviceMonitorKubelet.yaml
-rw-r--r-- 1 root root 530 Mar 11 2022 kubernetes-serviceMonitorKubeScheduler.yaml
-rw-r--r-- 1 root root 464 Mar 11 2022 kube-state-metrics-clusterRoleBinding.yaml
-rw-r--r-- 1 root root 1712 Mar 11 2022 kube-state-metrics-clusterRole.yaml
-rw-r--r-- 1 root root 2934 Oct 25 13:40 kube-state-metrics-deployment.yaml
-rw-r--r-- 1 root root 3082 Mar 11 2022 kube-state-metrics-prometheusRule.yaml
-rw-r--r-- 1 root root 280 Mar 11 2022 kube-state-metrics-serviceAccount.yaml
-rw-r--r-- 1 root root 1011 Mar 11 2022 kube-state-metrics-serviceMonitor.yaml
-rw-r--r-- 1 root root 580 Mar 11 2022 kube-state-metrics-service.yaml
-rw-r--r-- 1 root root 444 Mar 11 2022 node-exporter-clusterRoleBinding.yaml
-rw-r--r-- 1 root root 461 Mar 11 2022 node-exporter-clusterRole.yaml
-rw-r--r-- 1 root root 3047 Mar 11 2022 node-exporter-daemonset.yaml
-rw-r--r-- 1 root root 14356 Apr 11 2022 node-exporter-prometheusRule.yaml
-rw-r--r-- 1 root root 270 Mar 11 2022 node-exporter-serviceAccount.yaml
-rw-r--r-- 1 root root 850 Mar 11 2022 node-exporter-serviceMonitor.yaml
-rw-r--r-- 1 root root 492 Mar 11 2022 node-exporter-service.yaml
-rw-r--r-- 1 root root 482 Mar 11 2022 prometheus-adapter-apiService.yaml
-rw-r--r-- 1 root root 576 Mar 11 2022 prometheus-adapter-clusterRoleAggregatedMetricsReader.yaml
-rw-r--r-- 1 root root 494 Mar 11 2022 prometheus-adapter-clusterRoleBindingDelegator.yaml
-rw-r--r-- 1 root root 471 Mar 11 2022 prometheus-adapter-clusterRoleBinding.yaml
-rw-r--r-- 1 root root 378 Mar 11 2022 prometheus-adapter-clusterRoleServerResources.yaml
-rw-r--r-- 1 root root 409 Mar 11 2022 prometheus-adapter-clusterRole.yaml
-rw-r--r-- 1 root root 2204 Mar 11 2022 prometheus-adapter-configMap.yaml
-rw-r--r-- 1 root root 2530 Oct 25 13:39 prometheus-adapter-deployment.yaml
-rw-r--r-- 1 root root 506 Mar 11 2022 prometheus-adapter-podDisruptionBudget.yaml
-rw-r--r-- 1 root root 515 Mar 11 2022 prometheus-adapter-roleBindingAuthReader.yaml
-rw-r--r-- 1 root root 287 Mar 11 2022 prometheus-adapter-serviceAccount.yaml
-rw-r--r-- 1 root root 677 Mar 11 2022 prometheus-adapter-serviceMonitor.yaml
-rw-r--r-- 1 root root 501 Mar 11 2022 prometheus-adapter-service.yaml
-rw-r--r-- 1 root root 447 Mar 11 2022 prometheus-clusterRoleBinding.yaml
-rw-r--r-- 1 root root 394 Mar 11 2022 prometheus-clusterRole.yaml
-rw-r--r-- 1 root root 5000 Mar 11 2022 prometheus-operator-prometheusRule.yaml
-rw-r--r-- 1 root root 715 Mar 11 2022 prometheus-operator-serviceMonitor.yaml
-rw-r--r-- 1 root root 499 Mar 11 2022 prometheus-podDisruptionBudget.yaml
-rw-r--r-- 1 root root 14021 Mar 11 2022 prometheus-prometheusRule.yaml
-rw-r--r-- 1 root root 1184 Mar 11 2022 prometheus-prometheus.yaml
-rw-r--r-- 1 root root 471 Mar 11 2022 prometheus-roleBindingConfig.yaml
-rw-r--r-- 1 root root 1547 Mar 11 2022 prometheus-roleBindingSpecificNamespaces.yaml
-rw-r--r-- 1 root root 366 Mar 11 2022 prometheus-roleConfig.yaml
-rw-r--r-- 1 root root 2047 Mar 11 2022 prometheus-roleSpecificNamespaces.yaml
-rw-r--r-- 1 root root 271 Mar 11 2022 prometheus-serviceAccount.yaml
-rw-r--r-- 1 root root 531 Mar 11 2022 prometheus-serviceMonitor.yaml
-rw-r--r-- 1 root root 558 Mar 11 2022 prometheus-service.yaml
drw-r--r-- 2 root root 4096 Oct 24 12:31 setup
#先apply setup目錄中的yaml文件。然後apply一級目錄下的yaml文件。backsvc中的grafana的service資源清單。根據情況調整為nodeport或ClusterIP。K8S集群會自動在每台K8S節點部署node-exporter並收集數據。登錄grafana後初始帳號密碼為admin admin。添加dashboard即可監控K8S集群
[root@lecode-k8s-master monitor]# cd setup/
[root@lecode-k8s-master setup]# kubectl apply -f .
[root@lecode-k8s-master setup]# cd ..
[root@lecode-k8s-master monitor]# kubectl apply -f .
[root@lecode-k8s-master monitor]# kubectl get po -n monitoring
NAME READY STATUS RESTARTS AGE
alertmanager-main-0 2/2 Running 0 74m
alertmanager-main-1 2/2 Running 0 74m
alertmanager-main-2 2/2 Running 0 74m
blackbox-exporter-6798fb5bb4-d9m7m 3/3 Running 0 74m
grafana-64668d8465-x7x9z 1/1 Running 0 74m
kube-state-metrics-569d89897b-hlqxj 3/3 Running 0 57m
node-exporter-6vqxg 2/2 Running 0 74m
node-exporter-7dxh6 2/2 Running 0 74m
node-exporter-9j5xk 2/2 Running 0 74m
node-exporter-ftrmn 2/2 Running 0 74m
node-exporter-qszkn 2/2 Running 0 74m
node-exporter-wjkgj 2/2 Running 0 74m
prometheus-adapter-5dd78c75c6-h2jf7 1/1 Running 0 58m
prometheus-adapter-5dd78c75c6-qpwzv 1/1 Running 0 58m
prometheus-k8s-0 2/2 Running 0 74m
prometheus-k8s-1 2/2 Running 0 74m
prometheus-operator-75d9b475d9-mmzgs 2/2 Running 0 80m
webhook-dingtalk-6ffc94b49-z9z6l 1/1 Running 0 61m
[root@lecode-k8s-master backsvc]# kubectl get svc -n monitoring
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
alertmanager-main NodePort 10.98.35.93 <none> 9093:30093/TCP 72m
alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 72m
blackbox-exporter ClusterIP 10.109.10.110 <none> 9115/TCP,19115/TCP 72m
grafana NodePort 10.110.48.214 <none> 3000:30300/TCP 72m
kube-state-metrics ClusterIP None <none> 8443/TCP,9443/TCP 72m
node-exporter ClusterIP None <none> 9100/TCP 72m
prometheus-adapter ClusterIP 10.97.23.176 <none> 443/TCP 72m
prometheus-k8s ClusterIP 10.100.92.254 <none> 9090/TCP 72m
prometheus-operated ClusterIP None <none> 9090/TCP 72m
prometheus-operator ClusterIP None <none> 8443/TCP 78m
webhook-dingtalk ClusterIP 10.100.131.63 <none> 80/TCP 72m
5.2 訪問服務
暴露服務三種方法:用service資源的nodeport模式,或者用k8s的ingress暴露服務或者本地nginx代理。本地的nginx代理模式
這裡我grafana用的是nodeport模式。Prometheus用的是nginx代理。附上nginx配置文件
[root@lecode-k8s-master setup]# cat /usr/local/nginx/conf/4-layer-conf.d/lecode-prometheus-operator.conf #代理prometheus內置Dashboard UI upstream prometheus-dashboard { server 10.100.92.254:9090; #這裡ip為prometheus-k8s svc資源的ip } server { listen 9090; proxy_pass prometheus-dashboard; } #代理grafana upstream grafana { server 10.1.82.89:3000; #這裡ip為grafana svc資源的ip } server { listen 3000; proxy_pass grafana; }
訪問Prometheus targets
5.3 接入grafana
訪問grafana(默認密碼是admin admin)
去grafana官網下載對應dashboard 地址://grafana.com/grafana/dashboards/
6. 監控集群外的服務
6.1 exporter安裝
在對應服務的本地安裝對應的exporter用於收集數據(這裡以mysql為例)
#下載對應服務的exporter
#插件下載地址://www.modb.pro/db/216588
#插件下載地址://prometheus.io/download/
#下載完成後解壓mysqld_exporter-0.13.0.linux-amd64.tar.gz
#配置mysql-exporter
在root路徑下,創建.my.cnf文件。內容如下:
[root@lecode-test-001 ~]# cat /root/.my.cnf
[client]
user=mysql_monitor
password=Mysql@123
#創建mysql 用戶並授權
CREATE USER 『mysql_monitor』@『localhost』 IDENTIFIED BY 『Mysql@123』 WITH MAX_USER_CONNECTIONS 3;
GRANT PROCESS, REPLICATION CLIENT, SELECT ON . TO 『mysql_monitor』@『localhost』;
FLUSH PRIVILEGES;
EXIT
#啟動mysqld_exporter
[root@lecode-test-001 mysql-exporter]# nohup mysqld_exporter &
#找到對應的埠
[root@lecode-test-001 mysql-exporter]# tail -f nohup.out
level=info ts=2022-10-25T09:26:54.464Z caller=mysqld_exporter.go:277 msg="Starting msqyld_exporter" version="(version=0.13.0, branch=HEAD, revision=ad2847c7fa67b9debafccd5a08bacb12fc9031f1)"
level=info ts=2022-10-25T09:26:54.464Z caller=mysqld_exporter.go:278 msg="Build context" (gogo1.16.4,userroot@e2043849cb1f,date20210531-07:30:16)=(MISSING)
level=info ts=2022-10-25T09:26:54.464Z caller=mysqld_exporter.go:293 msg="Scraper enabled" scraper=global_status
level=info ts=2022-10-25T09:26:54.464Z caller=mysqld_exporter.go:293 msg="Scraper enabled" scraper=global_variables
level=info ts=2022-10-25T09:26:54.464Z caller=mysqld_exporter.go:293 msg="Scraper enabled" scraper=slave_status
level=info ts=2022-10-25T09:26:54.464Z caller=mysqld_exporter.go:293 msg="Scraper enabled" scraper=info_schema.innodb_cmp
level=info ts=2022-10-25T09:26:54.464Z caller=mysqld_exporter.go:293 msg="Scraper enabled" scraper=info_schema.innodb_cmpmem
level=info ts=2022-10-25T09:26:54.464Z caller=mysqld_exporter.go:293 msg="Scraper enabled" scraper=info_schema.query_response_time
level=info ts=2022-10-25T09:26:54.464Z caller=mysqld_exporter.go:303 msg="Listening on address" address=:9104 #這是exporter的埠
level=info ts=2022-10-25T09:26:54.464Z caller=tls_config.go:191 msg="TLS is disabled." http2=false
#檢查埠
[root@lecode-test-001 mysql-exporter]# ss -lntup |grep 9104
tcp LISTEN 0 128 :::9104 :::* users:(("mysqld_exporter",pid=26115,fd=3))
6.2 K8S配置
創建endpoint資源關聯對應服務主機的exporter埠。綁定service資源,通過ServiceMonitor資源添加Prometheus targets,
1)官方格式
kubectl -n monitoring get prometheus kube-prometheus -o yaml
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
labels:
app: prometheus
chart: prometheus-0.0.14
heritage: Tiller
prometheus: kube-prometheus
release: kube-prometheus
name: kube-prometheus
namespace: monitoring
spec:
...
baseImage: quay.io/prometheus/prometheus
serviceMonitorSelector:
matchLabels:
prometheus: kube-prometheus
#接下來就是按照格式創建對應的ServiceMonitor資源
通過ep資源把外部服務關聯到K8S內部,綁定對應的svc資源。在由serviceMonitor綁定對應的svc資源把數據提交給Prometheus,serviceMonitor通過標籤選擇器關聯service,而service只需要通過埠關聯ep,這裡的標籤和埠一定要注意一致.
2) 創建資源
[root@lecode-k8s-master monitor]# cat mysql.yaml
apiVersion: v1
kind: Endpoints
metadata:
name: mysql-test
namespace: monitoring
subsets:
- addresses:
- ip: 192.168.1.17 # ip為安裝應用伺服器的ip
ports:
- name: mysql
port: 9104 #export的埠
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/component: mysql
app.kubernetes.io/name: mysql
app.kubernetes.io/part-of: kube-prometheus
app.kubernetes.io/version: 0.49.0
name: mysql-test
namespace: monitoring
spec:
clusterIP: None
clusterIPs:
- None
ports:
- name: mysql
port: 9104
protocol: TCP
sessionAffinity: None
type: ClusterIP
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
app.kubernetes.io/component: mysql
app.kubernetes.io/name: mysql
app.kubernetes.io/part-of: kube-prometheus
app.kubernetes.io/version: 0.49.0
name: mysql-test
namespace: monitoring
spec:
endpoints:
- bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
interval: 30s
port: mysql
tlsConfig:
insecureSkipVerify: true
selector:
matchLabels:
app.kubernetes.io/component: mysql
app.kubernetes.io/name: mysql
app.kubernetes.io/part-of: kube-prometheus
#創建
[root@lecode-k8s-master monitor]# kubectl apply -f mysql.yaml
endpoints/mysql-test created
service/mysql-test created
servicemonitor.monitoring.coreos.com/mysql-test created
#檢查
[root@lecode-k8s-master monitor]# kubectl get -f mysql.yaml
NAME ENDPOINTS AGE
endpoints/mysql-test 192.168.1.17:9104 10m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/mysql-test ClusterIP None <none> 9104/TCP 10m
NAME AGE
servicemonitor.monitoring.coreos.com/mysql-test 10m
6.3 接入grafana
7 監控集群外的服務-redis
7.1 exporter安裝
#部署redis-exporter
#插件下載地址://www.modb.pro/db/216588
[root@lecode-test-001 ~]# tar xf redis_exporter-v1.3.2.linux-amd64.tar.gz
[root@lecode-test-001 ~]# ll
drwxr-xr-x 2 root root 4096 Nov 6 2019 redis_exporter-v1.3.2.linux-amd64
-rw-r--r-- 1 root root 3376155 Oct 27 10:26 redis_exporter-v1.3.2.linux-amd64.tar.gz
[root@lecode-test-001 ~]# mv redis_exporter-v1.3.2.linux-amd64 redis_exporter
[root@lecode-test-001 ~]# cd redis_exporter
[root@lecode-test-001 redis_exporter]# ll
total 8488
-rw-r--r-- 1 root root 1063 Nov 6 2019 LICENSE
-rw-r--r-- 1 root root 10284 Nov 6 2019 README.md
-rwxr-xr-x 1 root root 8675328 Nov 6 2019 redis_exporter
[root@lecode-test-001 redis_exporter]# nohup ./redis_exporter -redis.addr 192.168.1.17:6379 -redis.password 'Redislecodetest@shuli123' &
[1] 4564
[root@lecode-test-001 redis_exporter]# nohup: ignoring input and appending output to ânohup.outâ
[root@lecode-test-001 redis_exporter]# tail -f nohup.out
time="2022-10-27T10:26:48+08:00" level=info msg="Redis Metrics Exporter v1.3.2 build date: 2019-11-06-02:25:20 sha1: 175a69f33e8267e0a0ba47caab488db5e83a592e Go: go1.13.4 GOOS: linux GOARCH: amd64"
time="2022-10-27T10:26:48+08:00" level=info msg="Providing metrics at :9121/metrics"
#埠為9121
7.2 K8S配置
#創建redis-serviceMonitor資源
[root@lecode-k8s-master monitor]# cat redis.yaml
apiVersion: v1
kind: Endpoints
metadata:
name: redis-test
namespace: monitoring
subsets:
- addresses:
- ip: 192.168.1.17 # ip為安裝應用伺服器的ip
ports:
- name: redis
port: 9121 #exporter埠
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/component: redis
app.kubernetes.io/name: redis
app.kubernetes.io/part-of: kube-prometheus
app.kubernetes.io/version: 0.49.0
name: redis-test
namespace: monitoring
spec:
clusterIP: None
clusterIPs:
- None
ports:
- name: redis
port: 9121
protocol: TCP
sessionAffinity: None
type: ClusterIP
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
app.kubernetes.io/component: redis
app.kubernetes.io/name: redis
app.kubernetes.io/part-of: kube-prometheus
app.kubernetes.io/version: 0.49.0
name: redis-test
namespace: monitoring
spec:
endpoints:
- bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
interval: 30s
port: redis
tlsConfig:
insecureSkipVerify: true
selector:
matchLabels:
app.kubernetes.io/component: redis
app.kubernetes.io/name: redis
app.kubernetes.io/part-of: kube-prometheus
#創建資源
[root@lecode-k8s-master monitor]# kubectl apply -f redis.yaml
endpoints/redis-test created
service/redis-test created
servicemonitor.monitoring.coreos.com/redis-test created
[root@lecode-k8s-master monitor]# kubectl get ep,svc,serviceMonitor -n monitoring |grep redis
endpoints/redis-test 192.168.1.17:9121 6m2s
service/redis-test ClusterIP None <none> 9121/TCP 6m2s
servicemonitor.monitoring.coreos.com/redis-test 6m2s
7.3 Prometheus檢查
7.4 接入grafana
dashboard ID: 11835
8. 監控集群外的服務-kafka
8.1 安裝exporter
#插件下載地址://www.modb.pro/db/216588
[root@lecode-test-001 ~]# tar xf kafka_exporter-1.4.2.linux-amd64.tar.gz
[root@lecode-test-001 ~]# ll
drwxrwxr-x 2 2000 2000 4096 Sep 16 2021 kafka_exporter-1.4.2.linux-amd64
-rw-r--r-- 1 root root 8499720 Oct 27 15:30 kafka_exporter-1.4.2.linux-amd64.tar.gz
[root@lecode-test-001 ~]# mv kafka_exporter-1.4.2.linux-amd64 kafka_exporter
[root@lecode-test-001 ~]# cd kafka_exporter
[root@lecode-test-001 kafka_exporter]# ll
total 17676
-rwxr-xr-x 1 2000 2000 18086208 Sep 16 2021 kafka_exporter
-rw-rw-r-- 1 2000 2000 11357 Sep 16 2021 LICENSE
[root@lecode-test-001 kafka_exporter]# nohup ./kafka_exporter --kafka.server=192.168.1.17:9092 &
[1] 20777
[root@lecode-test-001 kafka_exporter]# nohup: ignoring input and appending output to 『nohup.out』
[root@lecode-test-001 kafka_exporter]# tail -f nohup.out
I1027 15:32:38.904075 20777 kafka_exporter.go:769] Starting kafka_exporter (version=1.4.2, branch=HEAD, revision=0d5d4ac4ba63948748cc2c53b35ed95c310cd6f2)
I1027 15:32:38.905515 20777 kafka_exporter.go:929] Listening on HTTP :9308
#exporter埠為9308
8.2 K8S配置
[root@lecode-k8s-master monitor]# cat kafka.yaml
apiVersion: v1
kind: Endpoints
metadata:
name: kafka-test
namespace: monitoring
subsets:
- addresses:
- ip: 192.168.1.17 # ip為安裝應用伺服器的ip
ports:
- name: kafka
port: 9308 #export的埠
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/component: kafka
app.kubernetes.io/name: kafka
app.kubernetes.io/part-of: kube-prometheus
app.kubernetes.io/version: 0.49.0
name: kafka-test
namespace: monitoring
spec:
clusterIP: None
clusterIPs:
- None
ports:
- name: kafka
port: 9308
protocol: TCP
sessionAffinity: None
type: ClusterIP
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
app.kubernetes.io/component: kafka
app.kubernetes.io/name: kafka
app.kubernetes.io/part-of: kube-prometheus
app.kubernetes.io/version: 0.49.0
name: kafka-test
namespace: monitoring
spec:
endpoints:
- bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
interval: 30s
port: kafka
tlsConfig:
insecureSkipVerify: true
selector:
matchLabels:
app.kubernetes.io/component: kafka
app.kubernetes.io/name: kafka
app.kubernetes.io/part-of: kube-prometheus
#創建
[root@lecode-k8s-master monitor]# kubectl apply -f kafka.yaml
endpoints/kafka-test created
service/kafka-test created
servicemonitor.monitoring.coreos.com/kafka-test created
[root@lecode-k8s-master monitor]# kubectl get -f kafka.yaml
NAME ENDPOINTS AGE
endpoints/kafka-test 192.168.1.17:9308 8m49s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kafka-test ClusterIP None <none> 9308/TCP 8m49s
NAME AGE
servicemonitor.monitoring.coreos.com/kafka-test 8m48s
8.3 Prometheus檢查
8.4 接入grafana
dashboard ID:7589
9. 監控集群外的服務-zookeeper
9.1 安裝exporter
exporter下載地址://github.com/carlpett/zookeeper_exporter/releases/download/v1.0.2/zookeeper_exporter
[root@lecode-test-001 zookeeper_exporter]# nohup ./zookeeper_exporter -zookeeper 192.168.1.17:2181 -bind-addr :9143 &
[2] 8310
[root@lecode-test-001 zookeeper_exporter]# nohup: ignoring input and appending output to 『nohup.out』
[root@lecode-test-001 zookeeper_exporter]# tail -f nohup.out
time="2022-10-27T15:58:27+08:00" level=info msg="zookeeper_exporter, version v1.0.2 (branch: HEAD, revision: d6e929223f6b3bf5ff25dd0340e8194cbd4d04fc)\n build user: @bd731f434d23\n build date: 2018-05-01T20:40:14+0000\n go version: go1.10.1"
time="2022-10-27T15:58:27+08:00" level=info msg="Starting zookeeper_exporter"
time="2022-10-27T15:58:27+08:00" level=info msg="Starting metric http endpoint on :9143"
#exporter埠為9143
9.2 K8S配置
[root@lecode-k8s-master monitor]# cat zookeeper.yaml
apiVersion: v1
kind: Endpoints
metadata:
name: zookeeper-test
namespace: monitoring
subsets:
- addresses:
- ip: 192.168.1.17
ports:
- name: zookeeper
port: 9143
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/component: zookeeper
app.kubernetes.io/name: zookeeper
app.kubernetes.io/part-of: kube-prometheus
app.kubernetes.io/version: 0.49.0
name: zookeeper-test
namespace: monitoring
spec:
clusterIP: None
clusterIPs:
- None
ports:
- name: zookeeper
port: 9143
protocol: TCP
sessionAffinity: None
type: ClusterIP
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
app.kubernetes.io/component: zookeeper
app.kubernetes.io/name: zookeeper
app.kubernetes.io/part-of: kube-prometheus
app.kubernetes.io/version: 0.49.0
name: zookeeper-test
namespace: monitoring
spec:
endpoints:
- bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
interval: 30s
port: zookeeper
tlsConfig:
insecureSkipVerify: true
selector:
matchLabels:
app.kubernetes.io/component: zookeeper
app.kubernetes.io/name: zookeeper
app.kubernetes.io/part-of: kube-prometheus
#創建
[root@lecode-k8s-master monitor]# kubectl apply -f zookeeper.yaml
endpoints/zookeeper-test created
service/zookeeper-test created
servicemonitor.monitoring.coreos.com/zookeeper-test created
[root@lecode-k8s-master monitor]# kubectl get -f zookeeper.yaml
NAME ENDPOINTS AGE
endpoints/zookeeper-test 192.168.1.17:9143 9m55s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/zookeeper-test ClusterIP None <none> 9143/TCP 9m55s
NAME AGE
servicemonitor.monitoring.coreos.com/zookeeper-test 9m55s
9.3 Prometheus檢查
9.4 接入grafana
dashboard ID:15026