OpenStack HA集群3-Pace
节点间主机名必须能解析 [root@controller1 ~]# cat /etc/hosts 192.168.17.149 controller1 192.168.17.141 controller2 192.168.17.166 controller3 192.168.17.111 demo.open-stack.cn 各节点间要互信，无密码能登录 [root@controller1 ~]# ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: 20:79:d4:a4:9f:8b:75:cf:12:58:f4:47:a4:c1:29:f3 root@controller1 The key's randomart p_w_picpath is: +--[ RSA 2048]----+ | .o. ...oo | | o ...o.o+ | | o + .+o . | | o o + E. | | S o | | o o + | | . . . o | | . | | | +-----------------+ [root@controller1 ~]# ssh-copy-id controller2 [root@controller1 ~]# ssh-copy-id controller3 配置YUM源 # vim /etc/yum.repos.d/ha-clustering.repo [network_ha-clustering_Stable] name=Stable High Availability/Clustering packages (CentOS-7) type=rpm-md baseurl=http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-7/ gpgcheck=0 gpgkey=http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-7/repodata/repomd.xml.key enabled=1 这个YUM源可能会冲突，先enabled=0，如果剩下一个crmsh包，再enabled=1打开后安装 Corosync下载地址，目前最新版本2.4.2 http://build.clusterlabs.org/corosync/releases/ http://build.clusterlabs.org/corosync/releases/corosync-2.4.2.tar.gz [root@controller1 ~]# ansible controller -m copy -a "src=/etc/yum.repos.d/ha-cluster.repo dest=/etc/yum.repos.d/" 安装软件包 # yum install pacemaker pcs resource-agents -y cifs-utils quota psmisc corosync fence-agents-all lvm2 resource-agents # yum install crmsh -y 启动pcsd，并确认启动正常 # systemctl enable pcsd # systemctl enable corosync # systemctl start pcsd # systemctl status pcsd [root@controller2 ~]# pacemakerd -$ Pacemaker 1.1.15-11.el7_3.2 Written by Andrew Beekhof [root@controller1 ~]# ansible controller -m command -a "pacemakerd -$" 修改hacluster密码 【all】# echo zoomtech | passwd --stdin hacluster [root@controller1 ~]# ansible controller -m command -a "echo zoomtech | passwd --stdin hacluster" # passwd hacluster 编辑corosync.conf [root@controller3 ~]# vim /etc/corosync/corosync.conf totem { version: 2 secauth: off cluster_name: openstack-cluster transport: udpu } nodelist { node { ring0_addr: controller1 nodeid: 1 } node { ring0_addr: controller2 nodeid: 2 } node { ring0_addr: controller3 nodeid: 3 } } logging { to_logfile: yes logfile: /var/log/cluster/corosync.log to_syslog: yes } quorum { provider: corosync_votequorum } [root@controller1 ~]# scp /etc/corosync/corosync.conf controller2:/etc/corosync/ [root@controller1 ~]# scp /etc/corosync/corosync.conf controller3:/etc/corosync/ [root@controller1 corosync]# ansible controller -m copy -a "src=corosync.conf dest=/etc/corosync" 创建集群 使用pcs设置集群身份认证 [root@controller1 ~]# pcs cluster auth controller1 controller2 controller3 -u hacluster -p zoomtech --force controller3: Authorized controller2: Authorized controller1: Authorized 现在我们创建一个集群并添加一些节点。注意,这个名字不能超过15个字符 [root@controller1 ~]# pcs cluster setup --force --name openstack-cluster controller1 controller2 controller3 Destroying cluster on nodes: controller1, controller2, controller3... controller3: Stopping Cluster (pacemaker)... controller2: Stopping Cluster (pacemaker)... controller1: Stopping Cluster (pacemaker)... controller2: Successfully destroyed cluster controller1: Successfully destroyed cluster controller3: Successfully destroyed cluster Sending cluster config files to the nodes... controller1: Succeeded controller2: Succeeded controller3: Succeeded Synchronizing pcsd certificates on nodes controller1, controller2, controller3... controller3: Success controller2: Success controller1: Success Restarting pcsd on the nodes in order to reload the certificates... controller3: Success controller2: Success controller1: Success 启动集群 [root@controller1 ~]# pcs cluster enable --all controller1: Cluster Enabled controller2: Cluster Enabled controller3: Cluster Enabled [root@controller1 ~]# pcs cluster start --all controller2: Starting Cluster... controller1: Starting Cluster... controller3: Starting Cluster... 查看集群状态 [root@controller1 corosync]# ansible controller -m command -a "pcs cluster status" [root@controller1 ~]# pcs cluster status Cluster Status: Stack: corosync Current DC: controller3 (version 1.1.15-11.el7_3.2-e174ec8) - partition with quorum Last updated: Fri Feb 17 10:39:38 2017 Last change: Fri Feb 17 10:39:29 2017 by hacluster via crmd on controller3 3 nodes and 0 resources configured PCSD Status: controller2: Online controller3: Online controller1: Online [root@controller1 corosync]# ansible controller -m command -a "pcs status" [root@controller1 ~]# pcs status Cluster name: openstack-cluster Stack: corosync Current DC: controller2 (version 1.1.15-11.el7_3.2-e174ec8) - partition with quorum Last updated: Thu Mar 2 17:07:34 2017 Last change: Thu Mar 2 01:44:44 2017 by root via cibadmin on controller1 3 nodes and 1 resource configured Online: [ controller1 controller2 controller3 ] Full list of resources: vip (ocf::heartbeat:IPaddr2): Started controller2 Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled 查看集群状态 [root@controller1 corosync]# ansible controller -m command -a "crm_mon -1" [root@controller1 ~]# crm_mon -1 Stack: corosync Current DC: controller2 (version 1.1.15-11.el7_3.2-e174ec8) - partition with quorum Last updated: Wed Mar 1 17:54:04 2017 Last change: Wed Mar 1 17:44:38 2017 by root via cibadmin on controller1 3 nodes and 1 resource configured Online: [ controller1 controller2 controller3 ] Active resources: vip (ocf::heartbeat:IPaddr2): Started controller1 查看pacemaker进程状态 [root@controller1 ~]# ps aux | grep pacemaker root 75900 0.2 0.5 132632 9216 ? Ss 10:39 0:00 /usr/sbin/pacemaked -f haclust+ 75901 0.3 0.8 135268 15376 ? Ss 10:39 0:00 /usr/libexec/pacemaker/cib root 75902 0.1 0.4 135608 7920 ? Ss 10:39 0:00 /usr/libexec/pacemaker/stonithd root 75903 0.0 0.2 105092 5020 ? Ss 10:39 0:00 /usr/libexec/pacemaker/lrmd haclust+ 75904 0.0 0.4 126924 7636 ? Ss 10:39 0:00 /usr/libexec/pacemaker/attrd haclust+ 75905 0.0 0.2 117040 4560 ? Ss 10:39 0:00 /usr/libexec/pacemaker/pengine haclust+ 75906 0.1 0.5 145328 8988 ? Ss 10:39 0:00 /usr/libexec/pacemaker/crmd root 75997 0.0 0.0 112648 948 pts/0 R+ 10:40 0:00 grep --color=auto pacemaker 查看集群状态 [root@controller1 ~]# corosync-cfgtool -s Printing ring status. Local node ID 1 RING ID 0 id = 192.168.17.132 status = ring 0 active with no faults [root@controller2 corosync]# corosync-cfgtool -s Printing ring status. Local node ID 2 RING ID 0 id = 192.168.17.146 status = ring 0 active with no faults [root@controller3 ~]# corosync-cfgtool -s Printing ring status. Local node ID 3 RING ID 0 id = 192.168.17.138 status = ring 0 active with no faults [root@controller1 ~]# corosync-cmapctl | grep members runtime.totem.pg.mrp.srp.members.1.config_version (u64) = 0 runtime.totem.pg.mrp.srp.members.1.ip (str) = r(0) ip(192.168.17.132) runtime.totem.pg.mrp.srp.members.1.join_count (u32) = 1 runtime.totem.pg.mrp.srp.members.1.status (str) = joined runtime.totem.pg.mrp.srp.members.2.config_version (u64) = 0 runtime.totem.pg.mrp.srp.members.2.ip (str) = r(0) ip(192.168.17.146) runtime.totem.pg.mrp.srp.members.2.join_count (u32) = 1 runtime.totem.pg.mrp.srp.members.2.status (str) = joined runtime.totem.pg.mrp.srp.members.3.config_version (u64) = 0 runtime.totem.pg.mrp.srp.members.3.ip (str) = r(0) ip(192.168.17.138) runtime.totem.pg.mrp.srp.members.3.join_count (u32) = 1 runtime.totem.pg.mrp.srp.members.3.status (str) = joined 查看集群状态 [root@controller1 ~]# pcs status corosync Membership information ---------------------- Nodeid Votes Name 1 1 controller1 (local) 3 1 controller3 2 1 controller2 [root@controller2 corosync]# pcs status corosync Membership information ---------------------- Nodeid Votes Name 1 1 controller1 3 1 controller3 2 1 controller2 (local) [root@controller3 ~]# pcs status corosync Membership information ---------------------- Nodeid Votes Name 1 1 controller1 3 1 controller3 (local) 2 1 controller2 [root@controller1 ~]# crm_verify -L -V error: unpack_resources: Resource start-up disabled since no STONITH resources have been defined error: unpack_resources: Either configure some or disable STONITH with the stonith-enabled option error: unpack_resources: NOTE: Clusters with shared data need STONITH to ensure data integrity Errors found during check: config not valid [root@controller1 ~]# [root@controller1 ~]# pcs property set stonith-enabled=false [root@controller1 ~]# pcs property set no-quorum-policy=ignore [root@controller1 ~]# crm_verify -L -V [root@controller1 corosync]# ansible controller -m command -a "pcs property set stonith-enabled=false [root@controller1 corosync]# ansible controller -m command -a "pcs property set no-quorum-policy=ignore" [root@controller1 corosync]# ansible controller -m command -a "crm_verify -L -V" 配置 VIP [root@controller1 ~]# crm crm(live)# configure crm(live)configure# show node 1: controller1 node 2: controller2 node 3: controller3 property cib-bootstrap-options: have-watchdog=false dc-version=1.1.15-11.el7_3.2-e174ec8 cluster-infrastructure=corosync cluster-name=openstack-cluster stonith-enabled=false no-quorum-policy=ignore crm(live)configure# primitive vip ocf:heartbeat:IPaddr2 params ip=192.168.17.111 cidr_netmask=24 nic=ens37 op start interval=0s timeout=20s op stop interval=0s timeout=20s monitor interval=30s meta priority=100 crm(live)configure# show node 1: controller1 node 2: controller2 node 3: controller3 primitive vip IPaddr2 params ip=192.168.17.111 cidr_netmask=24 nic=ens37 op start interval=0s timeout=20s op stop interval=30s timeout=20s monitor meta priority=100 property cib-bootstrap-options: have-watchdog=false dc-version=1.1.15-11.el7_3.2-e174ec8 cluster-infrastructure=corosync cluster-name=openstack-cluster stonith-enabled=false no-quorum-policy=ignore crm(live)configure# commit crm(live)configure# exit 查看VIP已绑定在ens37网卡上 [root@controller1 ~]# ip a 4: ens37: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:ff:8b:4b brd ff:ff:ff:ff:ff:ff inet 192.168.17.141/24 brd 192.168.17.255 scope global dynamic ens37 valid_lft 2388741sec preferred_lft 2388741sec inet 192.168.17.111/24 brd 192.168.17.255 scope global secondary ens37 valid_lft forever preferred_lft forever 上面指定的网卡名称3个节点必须是同一个名称，否则飘移会出现问题，切换不过去 [root@controller1 ~]# crm status Stack: corosync Current DC: controller1 (version 1.1.15-11.el7_3.2-e174ec8) - partition with quorum Last updated: Wed Feb 22 11:42:07 2017 Last change: Wed Feb 22 11:22:56 2017 by root via cibadmin on controller1 3 nodes and 1 resource configured Online: [ controller1 controller2 controller3 ] Full list of resources: vip (ocf::heartbeat:IPaddr2): Started controller1 查看corosync引擎是否正常启动 [root@controller1 ~]# grep -e "Corosync Cluster Engine" -e "configuration file" /var/log/cluster/corosync.log [51405] controller1 corosyncnotice [MAIN ] Corosync Cluster Engine ('2.4.0'): started and ready to provide service. Mar 01 17:35:20 [51425] controller1 cib: info: retrieveCib: Reading cluster configuration file /var/lib/pacemaker/cib/cib.xml (digest: /var/lib/pacemaker/cib/cib.xml.sig) Mar 01 17:35:20 [51425] controller1 cib: warning: cib_file_read_and_verify: Could not verify cluster configuration file /var/lib/pacemaker/cib/cib.xml: No such file or directory (2) Mar 01 17:35:20 [51425] controller1 cib: warning: cib_file_read_and_verify: Could not verify cluster configuration file /var/lib/pacemaker/cib/cib.xml: No such file or directory (2) Mar 01 17:35:20 [51425] controller1 cib: info: cib_file_write_with_digest: Reading cluster configuration file /var/lib/pacemaker/cib/cib.Apziws (digest: /var/lib/pacemaker/cib/cib.0ZxsVW) Mar 01 17:35:21 [51425] controller1 cib: info: cib_file_write_with_digest: Reading cluster configuration file /var/lib/pacemaker/cib/cib.ObYehI (digest: /var/lib/pacemaker/cib/cib.O8Rntg) Mar 01 17:35:42 [51425] controller1 cib: info: cib_file_write_with_digest: Reading cluster configuration file /var/lib/pacemaker/cib/cib.eqrhsF (digest: /var/lib/pacemaker/cib/cib.6BCfNj) Mar 01 17:35:42 [51425] controller1 cib: info: cib_file_write_with_digest: Reading cluster configuration file /var/lib/pacemaker/cib/cib.riot2E (digest: /var/lib/pacemaker/cib/cib.SAqtzj) Mar 01 17:35:42 [51425] controller1 cib: info: cib_file_write_with_digest: Reading cluster configuration file /var/lib/pacemaker/cib/cib.Q8H9BL (digest: /var/lib/pacemaker/cib/cib.MBljlq) Mar 01 17:38:29 [51425] controller1 cib: info: cib_file_write_with_digest: Reading cluster configuration file /var/lib/pacemaker/cib/cib.OTIiU4 (digest: /var/lib/pacemaker/cib/cib.JnHr1v) Mar 01 17:38:36 [51425] controller1 cib: info: cib_file_write_with_digest: Reading cluster configuration file /var/lib/pacemaker/cib/cib.2cK9Yk (digest: /var/lib/pacemaker/cib/cib.JSqEH8) Mar 01 17:44:38 [51425] controller1 cib: info: cib_file_write_with_digest: Reading cluster configuration file /var/lib/pacemaker/cib/cib.aPFtr3 (digest: /var/lib/pacemaker/cib/cib.E3Ve7X) [root@controller1 ~]# 查看初始化成员节点通知是否正常发出 [root@controller1 ~]# grep TOTEM /var/log/cluster/corosync.log [51405] controller1 corosyncnotice [TOTEM ] Initializing transport (UDP/IP Unicast). [51405] controller1 corosyncnotice [TOTEM ] Initializing transmit/receive security (NSS) crypto: none hash: none [51405] controller1 corosyncnotice [TOTEM ] The network interface [192.168.17.149] is now up. [51405] controller1 corosyncnotice [TOTEM ] adding new UDPU member {192.168.17.149} [51405] controller1 corosyncnotice [TOTEM ] adding new UDPU member {192.168.17.141} [51405] controller1 corosyncnotice [TOTEM ] adding new UDPU member {192.168.17.166} [51405] controller1 corosyncnotice [TOTEM ] A new membership (192.168.17.149:4) was formed. Members joined: 1 [51405] controller1 corosyncnotice [TOTEM ] A new membership (192.168.17.141:12) was formed. Members joined: 2 3 检查启动过程中是否有错误产生 [root@controller1 ~]# grep ERROR: /var/log/cluster/corosync.log
