JMXtrans + InfluxDB + Grafana实现Zookeeper性能指标监控
- 2019 年 10 月 22 日
- 笔记
一、总体效果图
这里是将集群全部放在一起，可以根据自己的审美看怎么放
二、监控指标
其中有些指标与第一篇Zookeeper通过四字命令基础监控（Zabbix）的四字命令的指标是有重复的，二者选一个则可
三、实现
1、influxdb的安装
1）设置yum源
cat <<EOF | sudo tee /etc/yum.repos.d/influxdb.repo [influxdb] name = InfluxDB Repository - RHEL $releasever baseurl = https://repos.influxdata.com/rhel/$releasever/$basearch/stable enabled = 1 gpgcheck = 1 gpgkey = https://repos.influxdata.com/influxdb.key EOF
2）安装influxdb
yum install influxdb systemctl start influxdb
3）修改配置文件（元数据以及数据存放目录）
[root@ip-172-0-0-7 influxDB]# cat /etc/influxdb/influxdb.conf |grep "^s*[^# t].*$" [meta] dir = "/influxDB/meta" [data] dir = "/influxDB/data" wal-dir = "/influxDB/wal" series-id-set-cache-size = 100 [coordinator] [retention] [shard-precreation] [monitor] [http] [logging] [subscriber] [[graphite]] [[collectd]] [[opentsdb]] [[udp]] [continuous_queries] [tls]
4、数据目录授权
chown -R influxdb.influxdb /influxDB
5、重启influxdb
systemctl restart influxdb
6、创建zookeeper数据库，以及创建账号密码
CREATE USER "admin" WITH PASSWORD "admin" WITH ALL PRIVILEGES create database zookeeper
2、zookeeper上打开JMX
1）vim ${Zookeeper-Home}/conf/zookeeper-env.sh
JMXLOCALONLY=false JMXDISABLE=false JMXPORT=8999 JMXAUTH=false JMXSSL=false
2、重启Zookeeper
./zkServer.sh restart
3、JMXtrans的安装
1）安装配置jmxtrans
下载地址https://oss.sonatype.org/content/repositories/snapshots/org/jmxtrans/jmxtrans/ 这里有最新的版本，用老的版本有一些问题 ；JMXtrans可以随意安装在任何一台机器上，不是必须安装在Zookeeper机器上，只要JMXtrans与Zookeeper机器的网络及端口是通的就可以
wget https://oss.sonatype.org/content/repositories/snapshots/org/jmxtrans/jmxtrans/271-SNAPSHOT/jmxtrans-271-20190109.091025-10.rpm yum -y install jmxtrans-271-20190109.091025-10.rpm
2）它的目录文件
/var/lib/jmxtrans /var/log/jmxtrans /etc/rc.d/init.d/jmxtrans /etc/jmxtrans /usr/share/jmxtrans /usr/share/jmxtrans/bin/jmxtrans /usr/share/jmxtrans/lib/org/jmxtrans /usr/share/jmxtrans/lib/org/jmxtrans/jmxtrans /usr/bin/jmxtrans /run/jmxtrans /run/lock/subsys/jmxtrans
3）编辑zookeeper.json文件并放到/var/lib/jmxtrans/目录
一般我们的Zookeeper是一个集群，比如当我们的Zookeeper为3台的时候，则需要弄三份zookeeper.json文件，每一份对应Zookeeper集群的每一台机器，通过里面的host和port对应
{ "servers": [{ "port": 8999, "host": "zookeeper-IP", "alias": "zk", "queries": [{ "outputWriters": [{ "@class": "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory", "url": "http://localhost:8086/", "database": "zookeeper", "username": "admin", "password": "admin" }], "obj": "org.apache.ZooKeeperService:name0=ReplicatedServer_id*", "attr": ["QuorumSize"] }, { "outputWriters": [{ "@class": "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory", "url": "http://localhost:8086/", "database": "zookeeper", "username": "admin", "password": "admin" }], "obj": "org.apache.ZooKeeperService:name0=ReplicatedServer_id*,name1=replica*,name2=*", "attr": ["MaxClientCnxnsPerHost", "MaxRequestLatency", "AvgRequestLatency", "MinRequestLatency", "MaxSessionTimeout", "MinSessionTimeout", "OutstandingRequests", "PacketsReceived", "PacketsSent", "PendingRevalidationCount", "TickTime"] }, { "outputWriters": [{ "@class": "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory", "url": "http://localhost:8086/", "database": "zookeeper", "username": "admin", "password": "admin" }], "obj": "org.apache.ZooKeeperService:name0=ReplicatedServer_id*,name1=replica*,name2=*,name3=InMemoryDataTree", "attr": ["NodeCount", "WatchCount"] }, { "outputWriters": [{ "@class": "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory", "url": "http://localhost:8086/", "database": "zookeeper", "username": "admin", "password": "admin" }], "obj": "org.apache.ZooKeeperService:name0=ReplicatedServer_id*,name1=replica*,name2=*,name3=Connections,name4=*,name5=*", "resultAlias": "Clients", "attr": ["AvgLatency", "LastLatency", "MaxLatency", "MinLatency", "OutstandingRequests", "PacketsReceived", "PacketsSent"] }, { "outputWriters": [{ "@class": "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory", "url": "http://localhost:8086/", "database": "zookeeper", "username": "admin", "password": "admin" }], "obj": "org.apache.ZooKeeperService:name0=StandaloneServer_port*", "resultAlias": "Clients", "attr": ["AvgLatency", "LastLatency", "MaxLatency", "MinLatency", "OutstandingRequests", "PacketsReceived", "PacketsSent", "NumAliveConnections"] }, { "outputWriters": [{ "@class": "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory", "url": "http://localhost:8086/", "database": "zookeeper", "username": "admin", "password": "admin" }], "obj": "java.lang:type=Memory", "resultAlias": "Memory", "attr": ["HeapMemoryUsage", "NonHeapMemoryUsage"] }, { "outputWriters": [{ "@class": "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory", "url": "http://localhost:8086/", "database": "zookeeper", "username": "admin", "password": "admin" }], "obj": "java.lang:type=OperatingSystem", "resultAlias": "Process", "attr": ["OpenFileDescriptorCount", "ProcessCpuLoad"] }, { "outputWriters": [{ "@class": "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory", "url": "http://localhost:8086/", "database": "zookeeper", "username": "admin", "password": "admin" }], "obj": "java.lang:type=Threading", "resultAlias": "Threading", "attr": ["ThreadCount"] }], "numQueryThreads": 2 }] }
4）启动JMXtrans
启动后，查看日志/var/log/jmxtrans/jmxtrans.log，看是否启动成功，若有错误则根据对应的错误解决就可以
/usr/share/jmxtrans/bin/jmxtrans start
4、Grafna上配置数据源
当创建Query，选择Influxdb数据源后，若有对应的指标出来，表明是OK的
