mongodb 复制集 运维 遇到的问题
- 2019 年 10 月 4 日
- 笔记
root couse: 对MongoDB复制集的认识不足
机器环境:
192.168.12.6 master状态
192.168.12.4 secondary状态
192.168.12.5 secondary状态
192.168.2.1 dump节点 ,之前因为磁盘不足,mongodb进程已宕机,这个实例也配置有vote投票权!
过程:
1、DBA在 192.168.12.5 这个 secondary节点上,执行了关闭实例命令
2、集群剩余的2台主机:192.168.12.4(secondary) 、192.168.12.6(master) ,都变成了secondary状态
3、业务反馈大量报错
4、DBA恢复 192.168.12.5 上面的mongodb进程,集群状态恢复
复盘:
下面的日志,是在 192.168.12.6 主节点上面看到的:
2019-04-16T15:47:14.196+0800 I ASIO [NetworkInterfaceASIO-Replication-0] Failed to connect to 192.168.12.5:27017 - HostUnreachable: Connection reset by peer 2019-04-16T15:47:14.196+0800 I ASIO [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to 192.168.12.5:27017 due to failed operation on a connection 2019-04-16T15:47:14.196+0800 I ASIO [NetworkInterfaceASIO-Replication-0] Failed to close stream: Transport endpoint is not connected 2019-04-16T15:47:14.196+0800 I REPL [ReplicationExecutor] Error in heartbeat request to 192.168.12.5:27017; HostUnreachable: Connection reset by peer 2019-04-16T15:47:14.196+0800 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to 192.168.12.5:27017 2019-04-16T15:47:14.196+0800 I ASIO [NetworkInterfaceASIO-Replication-0] Failed to connect to 192.168.12.5:27017 - HostUnreachable: Connection refused 2019-04-16T15:47:14.196+0800 I ASIO [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to 192.168.12.5:27017 due to failed operation on a connection 2019-04-16T15:47:14.196+0800 I REPL [ReplicationExecutor] Error in heartbeat request to 192.168.12.5:27017; HostUnreachable: Connection refused 2019-04-16T15:47:14.197+0800 I REPL [ReplicationExecutor] can't see a majority of the set, relinquishing primary 2019-04-16T15:47:14.197+0800 I REPL [ReplicationExecutor] Stepping down from primary in response to heartbeat 2019-04-16T15:47:14.198+0800 I REPL [replExecDBWorker-0] transition to SECONDARY 2019-04-16T15:47:14.274+0800 I NETWORK [conn476944080] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [192.168.3.11:38712]
集群的配置如下:
set01:SECONDARY> rs.conf() { "_id" : "set01", "version" : 130099, "members" : [ { "_id" : 6, "host" : "192.168.2.1:27017", "arbiterOnly" : false, "buildIndexes" : true, "hidden" : true, "priority" : 0, "tags" : { "dc" : "IDC1", "role" : "dump" }, "slaveDelay" : NumberLong(0), "votes" : 1 }, { "_id" : 7, "host" : "192.168.12.4:27017", "arbiterOnly" : false, "buildIndexes" : true, "hidden" : false, "priority" : 1, "tags" : { "dc" : "IDC1" }, "slaveDelay" : NumberLong(0), "votes" : 1 }, { "_id" : 8, "host" : "192.168.12.5:27017", "arbiterOnly" : false, "buildIndexes" : true, "hidden" : false, "priority" : 1, "tags" : { "dc" : "IDC1" }, "slaveDelay" : NumberLong(0), "votes" : 1 }, { "_id" : 9, "host" : "192.168.12.6:27017", "arbiterOnly" : false, "buildIndexes" : true, "hidden" : false, "priority" : 1, "tags" : { "dc" : "IDC1" }, "slaveDelay" : NumberLong(0), "votes" : 1 } ], "settings" : { "chainingAllowed" : true, "heartbeatIntervalMillis" : 2000, "heartbeatTimeoutSecs" : 10, "electionTimeoutMillis" : 10000, "getLastErrorModes" : { }, "getLastErrorDefaults" : { "w" : 1, "wtimeout" : 0 } } }
根据上面的内容,可以判断出 192.168.2.1宕机后,我们再次关闭一台192.168.12.5后,集群就剩2个投票,少于一半节点,整个集群无法选举出Primary,集群退化为只读状态【表现为rs.status()看到的都是secondary角色】 因此,通常建议将复制集成员数量设置为奇数。
解决措施:
将dump节点的投票属性去掉。
参考:
http://www.ttlsa.com/mongodb/mongodb-replicaset-internal/
https://blog.csdn.net/qq_24598601/article/details/81150614