mongodb 复制集 运维 遇到的问题

  • 2019 年 10 月 4 日
  • 笔记

root couse: 对MongoDB复制集的认识不足

机器环境:

192.168.12.6  master状态

192.168.12.4 secondary状态

192.168.12.5  secondary状态

192.168.2.1    dump节点 ,之前因为磁盘不足,mongodb进程已宕机,这个实例也配置有vote投票权!

过程:

1、DBA在 192.168.12.5 这个 secondary节点上,执行了关闭实例命令

2、集群剩余的2台主机:192.168.12.4(secondary) 、192.168.12.6(master)  ,都变成了secondary状态

3、业务反馈大量报错

4、DBA恢复 192.168.12.5 上面的mongodb进程,集群状态恢复

复盘:

下面的日志,是在 192.168.12.6 主节点上面看到的:

2019-04-16T15:47:14.196+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Failed to connect to 192.168.12.5:27017 - HostUnreachable: Connection reset by peer  2019-04-16T15:47:14.196+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to 192.168.12.5:27017 due to failed operation on a connection  2019-04-16T15:47:14.196+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Failed to close stream: Transport endpoint is not connected  2019-04-16T15:47:14.196+0800 I REPL     [ReplicationExecutor] Error in heartbeat request to 192.168.12.5:27017; HostUnreachable: Connection reset by peer  2019-04-16T15:47:14.196+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 192.168.12.5:27017  2019-04-16T15:47:14.196+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Failed to connect to 192.168.12.5:27017 - HostUnreachable: Connection refused   2019-04-16T15:47:14.196+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to 192.168.12.5:27017 due to failed operation on a connection  2019-04-16T15:47:14.196+0800 I REPL     [ReplicationExecutor] Error in heartbeat request to 192.168.12.5:27017; HostUnreachable: Connection refused  2019-04-16T15:47:14.197+0800 I REPL     [ReplicationExecutor] can't see a majority of the set, relinquishing primary  2019-04-16T15:47:14.197+0800 I REPL     [ReplicationExecutor] Stepping down from primary in response to heartbeat  2019-04-16T15:47:14.198+0800 I REPL     [replExecDBWorker-0] transition to SECONDARY  2019-04-16T15:47:14.274+0800 I NETWORK  [conn476944080] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [192.168.3.11:38712]

集群的配置如下:

set01:SECONDARY> rs.conf()  {   "_id" : "set01",   "version" : 130099,   "members" : [   {   "_id" : 6,   "host" : "192.168.2.1:27017",   "arbiterOnly" : false,   "buildIndexes" : true,   "hidden" : true,   "priority" : 0,   "tags" : {   "dc" : "IDC1",   "role" : "dump"   },   "slaveDelay" : NumberLong(0),   "votes" : 1   },   {   "_id" : 7,   "host" : "192.168.12.4:27017",   "arbiterOnly" : false,   "buildIndexes" : true,   "hidden" : false,   "priority" : 1,   "tags" : {   "dc" : "IDC1"   },   "slaveDelay" : NumberLong(0),   "votes" : 1   },   {   "_id" : 8,   "host" : "192.168.12.5:27017",   "arbiterOnly" : false,   "buildIndexes" : true,   "hidden" : false,   "priority" : 1,   "tags" : {   "dc" : "IDC1"   },   "slaveDelay" : NumberLong(0),   "votes" : 1   },   {   "_id" : 9,   "host" : "192.168.12.6:27017",   "arbiterOnly" : false,   "buildIndexes" : true,   "hidden" : false,   "priority" : 1,   "tags" : {   "dc" : "IDC1"   },   "slaveDelay" : NumberLong(0),   "votes" : 1   }   ],   "settings" : {   "chainingAllowed" : true,   "heartbeatIntervalMillis" : 2000,   "heartbeatTimeoutSecs" : 10,   "electionTimeoutMillis" : 10000,   "getLastErrorModes" : {   },   "getLastErrorDefaults" : {   "w" : 1,   "wtimeout" : 0   }   }  }

根据上面的内容,可以判断出 192.168.2.1宕机后,我们再次关闭一台192.168.12.5后,集群就剩2个投票,少于一半节点,整个集群无法选举出Primary,集群退化为只读状态【表现为rs.status()看到的都是secondary角色】 因此,通常建议将复制集成员数量设置为奇数。

解决措施:

    将dump节点的投票属性去掉。

参考: 

http://www.ttlsa.com/mongodb/mongodb-replicaset-internal/

https://blog.csdn.net/qq_24598601/article/details/81150614