分散式存儲系統之Ceph集群部署
- 2022 年 10 月 3 日
- 筆記
- ceph, Ceph-deploy, Ceph集群部署, 部署RADOS存儲集群
前文我們了解了Ceph的基礎架構和相關組件的介紹，回顧請參考//www.cnblogs.com/qiuhom-1874/p/16720234.html；今天我們來部署一個ceph集群；
部署工具介紹
1、ceph-deploy：該部署工具是ceph官方的部署工具，它只依賴SSH訪問伺服器，不需要額外的agent；它可以完全運行在自己的工作站上（比如admin host），不需要伺服器，資料庫類似的東西；該工具不是一個通用的部署工具，只針對ceph；相比ansible，puppet，功能相對單一；該工具可以推送配置文件，但它不處理客戶端配置，以及客戶端部署相關依賴等；
2、ceph-ansible：該工具是用ansible寫的劇本角色，我們只需要把對應的項目克隆下來，修改必要的參數，就可以正常的拉起一個ceph集群；但這前提是我們需要熟練使用ansible；項目地址 //github.com/ceph/ceph-ansible；
3、ceph-chef：chef也是類似ansible、puppet這類自動化部署工具，我們需要手動先安裝好chef，然後手動寫程式碼實現部署；ceph-chef就是寫好部署ceph的一個項目，我們可以下載對應項目在本地修改必要參數，也能正常拉起一個ceph集群；前提是我們要熟練使用chef才行；項目下載地址//github.com/ceph/ceph-chef；
4、puppet-ceph：很顯然該工具就是用puppet寫好的部署ceph的模組，也是下載下來修改必要參數，就可以正常拉起一個ceph集群；
不管用什麼工具，我們首先都要熟練知道ceph集群架構，它的必要組件，每個組件是怎麼工作的，有什麼作用，需要怎麼配置等等；除此之外我們還需要熟練使用各種自動化部署工具，才能很好的使用上述工具部署ceph集群；
集群拓撲網路
提示：Public Network是指公共網路，專供客戶端連接ceph集群使用；一般如果節點不多，集群規模和用戶的訪問量不大的情況下，只有一個public network也是完全ok；存在cluster network主要原因是，集群內部的事務處理，可能影響到客戶端在ceph存儲數據；所以cluster network是集群私有網路，專門用於集群內部各組件通訊協調使用；我們用於部署ceph的admin host 只需要有一個公共網路連入集群下發配置即可；
Ceph集群系統基礎環境設定
| 主機地址 | 角色 |
|
public network:192.168.0.70/24 |
admin host |
|
public network:192.168.0.71/24 cluster network:172.16.30.71/24 |
mon01/stor01 |
|
public network:192.168.0.72/24 cluster network:172.16.30.72/24 |
mon02/stor02 |
|
public network:192.168.0.73/24 cluster network:172.16.30.73/24 |
mon03/stor03 |
|
public network:192.168.0.74/24 cluster network:172.16.30.74/24 |
mgr01/stor04 |
|
public network:192.168.0.75/24 cluster network:172.16.30.75/24 |
mgr02/stor05 |
各主機主機名解析
192.168.0.70 ceph-admin ceph-admin.ilinux.io 192.168.0.71 ceph-mon01 ceph-mon01.ilinux.io ceph-stor01 ceph-stor01.ilinux.io 192.168.0.72 ceph-mon02 ceph-mon02.ilinux.io ceph-stor02 ceph-stor02.ilinux.io 192.168.0.73 ceph-mon03 ceph-mon03.ilinux.io ceph-stor03 ceph-stor03.ilinux.io 192.168.0.74 ceph-mgr01 ceph-mgr01.ilinux.io ceph-stor04 ceph-stor04.ilinux.io 192.168.0.75 ceph-mgr02 ceph-mgr02.ilinux.io ceph-stor05 ceph-stor05.ilinux.io 172.16.30.71 ceph-mon01 ceph-mon01.ilinux.io ceph-stor01 ceph-stor01.ilinux.io 172.16.30.72 ceph-mon02 ceph-mon02.ilinux.io ceph-stor02 ceph-stor02.ilinux.io 172.16.30.73 ceph-mon03 ceph-mon03.ilinux.io ceph-stor03 ceph-stor03.ilinux.io 172.16.30.74 ceph-mgr01 ceph-mgr01.ilinux.io ceph-stor04 ceph-stor04.ilinux.io 172.16.30.75 ceph-mgr02 ceph-mgr02.ilinux.io ceph-stor05 ceph-stor05.ilinux.io
各主機配置ntp服務設定各節點時間精準同步
[root@ceph-admin ~]# sed -i 's@^\(server \).*@\1ntp1.aliyun.com iburst@' /etc/chrony.conf [root@ceph-admin ~]# systemctl restart chronyd
提示：上述伺服器都需要在chrony.conf中配置同步時間的伺服器，這裡推薦使用阿里雲，然後重啟chronyd服務即可；
各節點關閉iptables 或firewalld服務
[root@ceph-admin ~]# systemctl stop firewalld [root@ceph-admin ~]# systemctl disable firewalld
提示：centos7 默認沒有安裝iptalbes服務，我們只需要關閉firewalld即可；
各節點關閉並禁用selinux
[root@ceph-admin ~]# sed -i 's@^\(SELINUX=\).*@\1Disabled@' /etc/sysconfig/selinu [root@ceph-admin ~]# setenforce 0
提示：上述sed命令表示查找/etc/sysconfig/selinux配置文件中，以SELINUX開頭的行當所有內容，並將其替換為SELINUX=Disabled；ok，準備好集群基礎環境以後，接下來我們開始部署ceph；
準備yum倉庫配置文件
提示：在阿里雲的鏡像站找ceph，然後找到我們需要安裝的版本；然後找到ceph-release，可以下載，也可以複製對應下載鏈接，然後在伺服器的每個節點都安裝好對應的ceph-release包；
在集群各節點安裝ceph-release包生成ceph倉庫配置文件
rpm -ivh //mirrors.aliyun.com/ceph/rpm-mimic/el7/noarch/ceph-release-1-1.el7.noarch.rpm?spm=a2c6h.25603864.0.0.6e8b635btfWSIx
在集群各節點安裝epel-release生成epel倉庫配置
[root@ceph-admin ~]# yum install -y epel-release
到此ceph的yum倉庫配置文件就準備好了
在集群各節點創建部署ceph的特定用戶帳號
[root@ceph-admin ~]# useradd cephadm && echo "admin" |passwd --stdin cephadm
提示：部署工具ceph-deploy 必須以普通用戶登錄到Ceph集群的各目標節點，且此用戶需要擁有無密碼使用sudo命令的許可權，以便在安裝軟體及生成配置文件的過程中無需中斷配置過程。不過，較新版的ceph-deploy也支援用 」–username「 選項提供可無密碼使用sudo命令的用戶名（包括 root ，但不建議這樣做）。另外，使用」ceph-deploy –username {username} 「命令時，指定的用戶需要能夠通過SSH協議自動認證並連接到各Ceph節點，以免ceph-deploy命令在配置中途需要用戶輸入密碼。
確保集群節點上新創建的cephadm用戶能夠無密碼運行sudo許可權
[root@ceph-admin ~]# echo "cephadm ALL = (root) NOPASSWD:ALL" |sudo tee /etc/sudoers.d/cephadm [root@ceph-admin ~]# chmod 0440 /etc/sudoers.d/cephadm
切換至cephadm用戶，查看sudo許可權
配置cephadm基於密鑰的ssh認證
提示：先切換用戶到cephadm下然後生成密鑰；
拷貝公鑰給自己
複製被地.ssh目錄給其他主機，放置在cephadm用戶家目錄下
[cephadm@ceph-admin ~]$ scp -rp .ssh cephadm@ceph-mon01:/home/cephadm/ [cephadm@ceph-admin ~]$ scp -rp .ssh cephadm@ceph-mon02:/home/cephadm/ [cephadm@ceph-admin ~]$ scp -rp .ssh cephadm@ceph-mon03:/home/cephadm/ [cephadm@ceph-admin ~]$ scp -rp .ssh cephadm@ceph-mgr01:/home/cephadm/ [cephadm@ceph-admin ~]$ scp -rp .ssh cephadm@ceph-mgr02:/home/cephadm/
驗證：以cephadm用戶遠程集群節點，看看是否是免密登錄
提示：能夠正常遠程執行命令說明我們的免密登錄就沒有問題；
在admin host上安裝ceph-deploy
[cephadm@ceph-admin ~]$ sudo yum update [cephadm@ceph-admin ~]$ sudo yum install ceph-deploy python-setuptools python2-subprocess32
驗證ceph-deploy是否成功安裝
[cephadm@ceph-admin ~]$ ceph-deploy --version 2.0.1 [cephadm@ceph-admin ~]$
提示：能夠正常看到ceph-deploy的版本，說明ceph-deploy安裝成功；
部署RADOS存儲集群
1、在admin host以cephadm用戶創建集群相關配置文件目錄
[cephadm@ceph-admin ~]$ mkdir ceph-cluster [cephadm@ceph-admin ~]$ cd ceph-cluster [cephadm@ceph-admin ceph-cluster]$ pwd /home/cephadm/ceph-cluster [cephadm@ceph-admin ceph-cluster]$
2、初始化第一個mon節點
[cephadm@ceph-admin ceph-cluster]$ ceph-deploy --help
usage: ceph-deploy [-h] [-v | -q] [--version] [--username USERNAME]
[--overwrite-conf] [--ceph-conf CEPH_CONF]
COMMAND ...
Easy Ceph deployment
-^-
/ \
|O o| ceph-deploy v2.0.1
).-.(
'/|||\`
| '|` |
'|`
Full documentation can be found at: //ceph.com/ceph-deploy/docs
optional arguments:
-h, --help show this help message and exit
-v, --verbose be more verbose
-q, --quiet be less verbose
--version the current installed version of ceph-deploy
--username USERNAME the username to connect to the remote host
--overwrite-conf overwrite an existing conf file on remote host (if
present)
--ceph-conf CEPH_CONF
use (or reuse) a given ceph.conf file
commands:
COMMAND description
new Start deploying a new cluster, and write a
CLUSTER.conf and keyring for it.
install Install Ceph packages on remote hosts.
rgw Ceph RGW daemon management
mgr Ceph MGR daemon management
mds Ceph MDS daemon management
mon Ceph MON Daemon management
gatherkeys Gather authentication keys for provisioning new nodes.
disk Manage disks on a remote host.
osd Prepare a data disk on remote host.
repo Repo definition management
admin Push configuration and client.admin key to a remote
host.
config Copy ceph.conf to/from remote host(s)
uninstall Remove Ceph packages from remote hosts.
purgedata Purge (delete, destroy, discard, shred) any Ceph data
from /var/lib/ceph
purge Remove Ceph packages from remote hosts and purge all
data.
forgetkeys Remove authentication keys from the local directory.
pkg Manage packages on remote hosts.
calamari Install and configure Calamari nodes. Assumes that a
repository with Calamari packages is already
configured. Refer to the docs for examples
(//ceph.com/ceph-deploy/docs/conf.html)
See 'ceph-deploy <command> --help' for help on a specific command
[cephadm@ceph-admin ceph-cluster]$
提示：我們通過查看ceph-deploy 的幫助可以知道 它的子命令new就是創建一個集群配置和生成一個keyring文件；
查看ceph-deploy new的用法
[cephadm@ceph-admin ceph-cluster]$ ceph-deploy new --help
usage: ceph-deploy new [-h] [--no-ssh-copykey] [--fsid FSID]
[--cluster-network CLUSTER_NETWORK]
[--public-network PUBLIC_NETWORK]
MON [MON ...]
Start deploying a new cluster, and write a CLUSTER.conf and keyring for it.
positional arguments:
MON initial monitor hostname, fqdn, or hostname:fqdn pair
optional arguments:
-h, --help show this help message and exit
--no-ssh-copykey do not attempt to copy SSH keys
--fsid FSID provide an alternate FSID for ceph.conf generation
--cluster-network CLUSTER_NETWORK
specify the (internal) cluster network
--public-network PUBLIC_NETWORK
specify the public network for a cluster
[cephadm@ceph-admin ceph-cluster]$
提示：ceph-deploy new的命令格式 我們只需要對應節點的主機名即可；但是前提是對應主機名做了正確的解析；
[cephadm@ceph-admin ceph-cluster]$ ceph-deploy new ceph-mon01 [ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadm/.cephdeploy.conf [ceph_deploy.cli][INFO ] Invoked (2.0.1): /bin/ceph-deploy new ceph-mon01 [ceph_deploy.cli][INFO ] ceph-deploy options: [ceph_deploy.cli][INFO ] username : None [ceph_deploy.cli][INFO ] func : <function new at 0x7f0660799ed8> [ceph_deploy.cli][INFO ] verbose : False [ceph_deploy.cli][INFO ] overwrite_conf : False [ceph_deploy.cli][INFO ] quiet : False [ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f065ff11b48> [ceph_deploy.cli][INFO ] cluster : ceph [ceph_deploy.cli][INFO ] ssh_copykey : True [ceph_deploy.cli][INFO ] mon : ['ceph-mon01'] [ceph_deploy.cli][INFO ] public_network : None [ceph_deploy.cli][INFO ] ceph_conf : None [ceph_deploy.cli][INFO ] cluster_network : None [ceph_deploy.cli][INFO ] default_release : False [ceph_deploy.cli][INFO ] fsid : None [ceph_deploy.new][DEBUG ] Creating new cluster named ceph [ceph_deploy.new][INFO ] making sure passwordless SSH succeeds [ceph-mon01][DEBUG ] connected to host: ceph-admin.ilinux.io [ceph-mon01][INFO ] Running command: ssh -CT -o BatchMode=yes ceph-mon01 [ceph-mon01][DEBUG ] connection detected need for sudo [ceph-mon01][DEBUG ] connected to host: ceph-mon01 [ceph-mon01][DEBUG ] detect platform information from remote host [ceph-mon01][DEBUG ] detect machine type [ceph-mon01][DEBUG ] find the location of an executable [ceph-mon01][INFO ] Running command: sudo /usr/sbin/ip link show [ceph-mon01][INFO ] Running command: sudo /usr/sbin/ip addr show [ceph-mon01][DEBUG ] IP addresses found: [u'172.16.30.71', u'192.168.0.71'] [ceph_deploy.new][DEBUG ] Resolving host ceph-mon01 [ceph_deploy.new][DEBUG ] Monitor ceph-mon01 at 192.168.0.71 [ceph_deploy.new][DEBUG ] Monitor initial members are ['ceph-mon01'] [ceph_deploy.new][DEBUG ] Monitor addrs are ['192.168.0.71'] [ceph_deploy.new][DEBUG ] Creating a random mon key... [ceph_deploy.new][DEBUG ] Writing monitor keyring to ceph.mon.keyring... [ceph_deploy.new][DEBUG ] Writing initial config to ceph.conf... [cephadm@ceph-admin ceph-cluster]$
提示：我們可以在命令行使用–public-network 選項來指定集群公共網路和使用–cluster-network選項來指定對應集群網路；當然也可以生成好配置文件，然後在配置文件里修改也行；
3、修改配置文件指定集群的公共網路和集群網路
提示：編輯生成的ceph.conf配置文件，在【global】配置段中設置ceph集群面向客戶端通訊的網路ip地址所在公網網路地址和面向集群各節點通訊的網路ip地址所在集群網路地址，如上所示；
4、安裝ceph集群
[cephadm@ceph-admin ceph-cluster]$ ceph-deploy install ceph-mon01 ceph-mon02 ceph-mon03 ceph-mgr01 ceph-mgr02
提示：ceph-deploy命令能夠以遠程的方式連入Ceph集群各節點完成程式包安裝等操作，所以我們只需要告訴ceph-deploy 那些主機需要安裝即可；
集群各主機如果需要獨立安裝ceph程式包，方法如下
yum install ceph ceph-radosgw
提示：前提是該主機基礎環境都配置好了，比如關閉iptables或firewalld，同步時間，ssh免密登錄，關閉selinux等；重要的是對應ceph的yum倉庫配置和epel倉庫配置都已經正常配置；
提示：這裡提示我們沒有安裝applydeltarpm；
查找applydeltarpm安裝包
在集群各節點安裝deltarpm包來解決上述報錯
yum install -y deltarpm-3.6-3.el7.x86_64
再次在admin host上用ceph-deploy安裝ceph集群
提示：這裡告訴我們沒有presto元數據；解決辦法清除所有yum快取，重新生成快取；
yum clean all && yum makecache
提示：以上操作都需要在集群各節點上操作；
再次在admin host上用ceph-deploy安裝ceph集群
提示：如果在最後能夠看到ceph的版本，說明我們指定的節點都已經安裝好對應ceph集群所需的程式包了；
5、配置初始MON節點，並收集所有密鑰
查看ceph-deploy mon幫助
[cephadm@ceph-admin ceph-cluster]$ ceph-deploy mon --help
usage: ceph-deploy mon [-h] {add,create,create-initial,destroy} ...
Ceph MON Daemon management
positional arguments:
{add,create,create-initial,destroy}
add Add a monitor to an existing cluster:
ceph-deploy mon add node1
Or:
ceph-deploy mon add --address 192.168.1.10 node1
If the section for the monitor exists and defines a `mon addr` that
will be used, otherwise it will fallback by resolving the hostname to an
IP. If `--address` is used it will override all other options.
create Deploy monitors by specifying them like:
ceph-deploy mon create node1 node2 node3
If no hosts are passed it will default to use the
`mon initial members` defined in the configuration.
create-initial Will deploy for monitors defined in `mon initial
members`, wait until they form quorum and then
gatherkeys, reporting the monitor status along the
process. If monitors don't form quorum the command
will eventually time out.
destroy Completely remove Ceph MON from remote host(s)
optional arguments:
-h, --help show this help message and exit
[cephadm@ceph-admin ceph-cluster]$
提示：add 是添加mon節點，create是創建一個mon節點，但不初始化，如果要初始化需要在對應節點的配置文件中定義配置mon成員；create-initial是創建並初始化mon成員；destroy是銷毀一個mon移除mon節點；
初始化mon節點
[cephadm@ceph-admin ceph-cluster]$ ceph-deploy mon create-initial
提示：從上面的輸出資訊可以看到該命令是從當前配置文件讀取mon節點資訊，然後初始化；我們在上面的new 命令里只有給了mon01，所以這裡只初始化了mon01；並在當前目錄生成了引導mds、mgr、osd、rgw和客戶端連接ceph集群的管理員密鑰；
6、拷貝配置文件和admin密鑰到集群各節點
查看ceph-deploy admin的幫助
[cephadm@ceph-admin ceph-cluster]$ ceph-deploy admin --help usage: ceph-deploy admin [-h] HOST [HOST ...] Push configuration and client.admin key to a remote host. positional arguments: HOST host to configure for Ceph administration optional arguments: -h, --help show this help message and exit [cephadm@ceph-admin ceph-cluster]$
提示：ceph-deploy admin命令主要作用是向指定的集群主機推送配置文件和客戶端管理員密鑰；以免得每次執行」ceph「命令行時不得不明確指定MON節點地址和ceph.client.admin.keyring；
[cephadm@ceph-admin ceph-cluster]$ ceph-deploy admin ceph-mon01 ceph-mon02 ceph-mon03 ceph-stor04 ceph-stor05
提示：推送配置和管理員密鑰我們只需要後面跟上對應集群主機即可，注意主機名要做對應的解析；這裡還需要多說一句，配置文件是集群每個節點都需要推送的，但是管理密鑰通常只需要推送給需要在對應主機上執行管理命令的主機上使用；所以ceph-deploy config命令就是用於專門推送配置文件，不推送管理密鑰；
查看ceph-deploy config幫助
[cephadm@ceph-admin ceph-cluster]$ ceph-deploy config --help
usage: ceph-deploy config [-h] {push,pull} ...
Copy ceph.conf to/from remote host(s)
positional arguments:
{push,pull}
push push Ceph config file to one or more remote hosts
pull pull Ceph config file from one or more remote hosts
optional arguments:
-h, --help show this help message and exit
[cephadm@ceph-admin ceph-cluster]$ ceph-deploy config push --help
usage: ceph-deploy config push [-h] HOST [HOST ...]
positional arguments:
HOST host(s) to push the config file to
optional arguments:
-h, --help show this help message and exit
[cephadm@ceph-admin ceph-cluster]$ ceph-deploy config pull --help
usage: ceph-deploy config pull [-h] HOST [HOST ...]
positional arguments:
HOST host(s) to pull the config file from
optional arguments:
-h, --help show this help message and exit
[cephadm@ceph-admin ceph-cluster]$
提示：ceph-deploy config 有兩個子命令，一個是push，表示把本機配置推送到對應指定的主機；pull表示把遠端主機的配置拉去到本地；
驗證：查看mon01主機上，對應配置文件和管理員密鑰文件是否都推送過去了？
提示：可以看到對應配置文件和管理員密鑰都推送到對應主機上，但是管理員密鑰的許可權對於cephadm是沒有讀許可權；
設置管理員密鑰能夠被cephadm用戶有讀許可權
提示：上述設置許可權的命令需要在每個節點都要設置；
7、配置Manager節點，啟動ceph-mgr進程（僅Luminious+版本）
查看ceph-deploy mgr幫助
[cephadm@ceph-admin ceph-cluster]$ ceph-deploy mgr --help
usage: ceph-deploy mgr [-h] {create} ...
Ceph MGR daemon management
positional arguments:
{create}
create Deploy Ceph MGR on remote host(s)
optional arguments:
-h, --help show this help message and exit
[cephadm@ceph-admin ceph-cluster]$
提示：mgr子命令就只有一個create命令用於部署mgr；
[cephadm@ceph-admin ceph-cluster]$ ceph-deploy mgr create ceph-mgr01 ceph-mgr02
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadm/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (2.0.1): /bin/ceph-deploy mgr create ceph-mgr01 ceph-mgr02
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] mgr : [('ceph-mgr01', 'ceph-mgr01'), ('ceph-mgr02', 'ceph-mgr02')]
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] subcommand : create
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f58a0514950>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] func : <function mgr at 0x7f58a0d8d230>
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.mgr][DEBUG ] Deploying mgr, cluster ceph hosts ceph-mgr01:ceph-mgr01 ceph-mgr02:ceph-mgr02
[ceph-mgr01][DEBUG ] connection detected need for sudo
[ceph-mgr01][DEBUG ] connected to host: ceph-mgr01
[ceph-mgr01][DEBUG ] detect platform information from remote host
[ceph-mgr01][DEBUG ] detect machine type
[ceph_deploy.mgr][INFO ] Distro info: CentOS Linux 7.9.2009 Core
[ceph_deploy.mgr][DEBUG ] remote host will use systemd
[ceph_deploy.mgr][DEBUG ] deploying mgr bootstrap to ceph-mgr01
[ceph-mgr01][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-mgr01][WARNIN] mgr keyring does not exist yet, creating one
[ceph-mgr01][DEBUG ] create a keyring file
[ceph-mgr01][DEBUG ] create path recursively if it doesn't exist
[ceph-mgr01][INFO ] Running command: sudo ceph --cluster ceph --name client.bootstrap-mgr --keyring /var/lib/ceph/bootstrap-mgr/ceph.keyring auth get-or-create mgr.ceph-mgr01 mon allow profile mgr osd allow * mds allow * -o /var/lib/ceph/mgr/ceph-ceph-mgr01/keyring
[ceph-mgr01][INFO ] Running command: sudo systemctl enable ceph-mgr@ceph-mgr01
[ceph-mgr01][WARNIN] Created symlink from /etc/systemd/system/ceph-mgr.target.wants/[email protected] to /usr/lib/systemd/system/[email protected].
[ceph-mgr01][INFO ] Running command: sudo systemctl start ceph-mgr@ceph-mgr01
[ceph-mgr01][INFO ] Running command: sudo systemctl enable ceph.target
[ceph-mgr02][DEBUG ] connection detected need for sudo
[ceph-mgr02][DEBUG ] connected to host: ceph-mgr02
[ceph-mgr02][DEBUG ] detect platform information from remote host
[ceph-mgr02][DEBUG ] detect machine type
[ceph_deploy.mgr][INFO ] Distro info: CentOS Linux 7.9.2009 Core
[ceph_deploy.mgr][DEBUG ] remote host will use systemd
[ceph_deploy.mgr][DEBUG ] deploying mgr bootstrap to ceph-mgr02
[ceph-mgr02][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-mgr02][WARNIN] mgr keyring does not exist yet, creating one
[ceph-mgr02][DEBUG ] create a keyring file
[ceph-mgr02][DEBUG ] create path recursively if it doesn't exist
[ceph-mgr02][INFO ] Running command: sudo ceph --cluster ceph --name client.bootstrap-mgr --keyring /var/lib/ceph/bootstrap-mgr/ceph.keyring auth get-or-create mgr.ceph-mgr02 mon allow profile mgr osd allow * mds allow * -o /var/lib/ceph/mgr/ceph-ceph-mgr02/keyring
[ceph-mgr02][INFO ] Running command: sudo systemctl enable ceph-mgr@ceph-mgr02
[ceph-mgr02][WARNIN] Created symlink from /etc/systemd/system/ceph-mgr.target.wants/[email protected] to /usr/lib/systemd/system/[email protected].
[ceph-mgr02][INFO ] Running command: sudo systemctl start ceph-mgr@ceph-mgr02
[ceph-mgr02][INFO ] Running command: sudo systemctl enable ceph.target
[cephadm@ceph-admin ceph-cluster]$
在集群節點上執行ceph -s來查看現在ceph集群的狀態
提示：可以看到現在集群有一個mon節點和兩個mgr節點；mgr01處於當前活躍狀態，mgr02處於備用狀態；對應沒有osd,所以集群狀態顯示health warning；
向RADOS集群添加OSD
列出並擦凈磁碟
查看ceph-deploy disk命令的幫助
提示：ceph-deploy disk命令有兩個子命令，list表示列出對應主機上的磁碟；zap表示擦凈對應主機上的磁碟；
擦凈mon01的sdb和sdc
[cephadm@ceph-admin ceph-cluster]$ ceph-deploy disk zap --help usage: ceph-deploy disk zap [-h] [--debug] [HOST] DISK [DISK ...] positional arguments: HOST Remote HOST(s) to connect DISK Disk(s) to zap optional arguments: -h, --help show this help message and exit --debug Enable debug mode on remote ceph-volume calls [cephadm@ceph-admin ceph-cluster]$ ceph-deploy disk zap ceph-mon01 /dev/sdb /dev/sdc [ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadm/.cephdeploy.conf [ceph_deploy.cli][INFO ] Invoked (2.0.1): /bin/ceph-deploy disk zap ceph-mon01 /dev/sdb /dev/sdc [ceph_deploy.cli][INFO ] ceph-deploy options: [ceph_deploy.cli][INFO ] username : None [ceph_deploy.cli][INFO ] verbose : False [ceph_deploy.cli][INFO ] debug : False [ceph_deploy.cli][INFO ] overwrite_conf : False [ceph_deploy.cli][INFO ] subcommand : zap [ceph_deploy.cli][INFO ] quiet : False [ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f35f8500f80> [ceph_deploy.cli][INFO ] cluster : ceph [ceph_deploy.cli][INFO ] host : ceph-mon01 [ceph_deploy.cli][INFO ] func : <function disk at 0x7f35f84d1a28> [ceph_deploy.cli][INFO ] ceph_conf : None [ceph_deploy.cli][INFO ] default_release : False [ceph_deploy.cli][INFO ] disk : ['/dev/sdb', '/dev/sdc'] [ceph_deploy.osd][DEBUG ] zapping /dev/sdb on ceph-mon01 [ceph-mon01][DEBUG ] connection detected need for sudo [ceph-mon01][DEBUG ] connected to host: ceph-mon01 [ceph-mon01][DEBUG ] detect platform information from remote host [ceph-mon01][DEBUG ] detect machine type [ceph-mon01][DEBUG ] find the location of an executable [ceph_deploy.osd][INFO ] Distro info: CentOS Linux 7.9.2009 Core [ceph-mon01][DEBUG ] zeroing last few blocks of device [ceph-mon01][DEBUG ] find the location of an executable [ceph-mon01][INFO ] Running command: sudo /usr/sbin/ceph-volume lvm zap /dev/sdb [ceph-mon01][WARNIN] --> Zapping: /dev/sdb [ceph-mon01][WARNIN] --> --destroy was not specified, but zapping a whole device will remove the partition table [ceph-mon01][WARNIN] Running command: /bin/dd if=/dev/zero of=/dev/sdb bs=1M count=10 conv=fsync [ceph-mon01][WARNIN] stderr: 10+0 records in [ceph-mon01][WARNIN] 10+0 records out [ceph-mon01][WARNIN] stderr: 10485760 bytes (10 MB) copied, 0.0721997 s, 145 MB/s [ceph-mon01][WARNIN] --> Zapping successful for: <Raw Device: /dev/sdb> [ceph_deploy.osd][DEBUG ] zapping /dev/sdc on ceph-mon01 [ceph-mon01][DEBUG ] connection detected need for sudo [ceph-mon01][DEBUG ] connected to host: ceph-mon01 [ceph-mon01][DEBUG ] detect platform information from remote host [ceph-mon01][DEBUG ] detect machine type [ceph-mon01][DEBUG ] find the location of an executable [ceph_deploy.osd][INFO ] Distro info: CentOS Linux 7.9.2009 Core [ceph-mon01][DEBUG ] zeroing last few blocks of device [ceph-mon01][DEBUG ] find the location of an executable [ceph-mon01][INFO ] Running command: sudo /usr/sbin/ceph-volume lvm zap /dev/sdc [ceph-mon01][WARNIN] --> Zapping: /dev/sdc [ceph-mon01][WARNIN] --> --destroy was not specified, but zapping a whole device will remove the partition table [ceph-mon01][WARNIN] Running command: /bin/dd if=/dev/zero of=/dev/sdc bs=1M count=10 conv=fsync [ceph-mon01][WARNIN] stderr: 10+0 records in [ceph-mon01][WARNIN] 10+0 records out [ceph-mon01][WARNIN] 10485760 bytes (10 MB) copied [ceph-mon01][WARNIN] stderr: , 0.0849861 s, 123 MB/s [ceph-mon01][WARNIN] --> Zapping successful for: <Raw Device: /dev/sdc> [cephadm@ceph-admin ceph-cluster]$
提示：擦凈磁碟我們需要在後面接對應主機和磁碟；若設備上此前有數據，則可能需要在相應節點上以root用戶使用「ceph-volume lvm zap –destroy {DEVICE}」命令進行；
添加osd
查看 ceph-deploy osd幫助
[cephadm@ceph-admin ceph-cluster]$ ceph-deploy osd --help
usage: ceph-deploy osd [-h] {list,create} ...
Create OSDs from a data disk on a remote host:
ceph-deploy osd create {node} --data /path/to/device
For bluestore, optional devices can be used::
ceph-deploy osd create {node} --data /path/to/data --block-db /path/to/db-device
ceph-deploy osd create {node} --data /path/to/data --block-wal /path/to/wal-device
ceph-deploy osd create {node} --data /path/to/data --block-db /path/to/db-device --block-wal /path/to/wal-device
For filestore, the journal must be specified, as well as the objectstore::
ceph-deploy osd create {node} --filestore --data /path/to/data --journal /path/to/journal
For data devices, it can be an existing logical volume in the format of:
vg/lv, or a device. For other OSD components like wal, db, and journal, it
can be logical volume (in vg/lv format) or it must be a GPT partition.
positional arguments:
{list,create}
list List OSD info from remote host(s)
create Create new Ceph OSD daemon by preparing and activating a
device
optional arguments:
-h, --help show this help message and exit
[cephadm@ceph-admin ceph-cluster]$
提示：ceph-deploy osd有兩個子命令，list表示列出遠程主機上osd；create表示創建一個新的ceph osd守護進程設備；
查看ceph-deploy osd create 幫助
[cephadm@ceph-admin ceph-cluster]$ ceph-deploy osd create --help
usage: ceph-deploy osd create [-h] [--data DATA] [--journal JOURNAL]
[--zap-disk] [--fs-type FS_TYPE] [--dmcrypt]
[--dmcrypt-key-dir KEYDIR] [--filestore]
[--bluestore] [--block-db BLOCK_DB]
[--block-wal BLOCK_WAL] [--debug]
[HOST]
positional arguments:
HOST Remote host to connect
optional arguments:
-h, --help show this help message and exit
--data DATA The OSD data logical volume (vg/lv) or absolute path
to device
--journal JOURNAL Logical Volume (vg/lv) or path to GPT partition
--zap-disk DEPRECATED - cannot zap when creating an OSD
--fs-type FS_TYPE filesystem to use to format DEVICE (xfs, btrfs)
--dmcrypt use dm-crypt on DEVICE
--dmcrypt-key-dir KEYDIR
directory where dm-crypt keys are stored
--filestore filestore objectstore
--bluestore bluestore objectstore
--block-db BLOCK_DB bluestore block.db path
--block-wal BLOCK_WAL
bluestore block.wal path
--debug Enable debug mode on remote ceph-volume calls
[cephadm@ceph-admin
提示：create可以指定數據盤，日誌盤以及block-db盤和bluestore 日誌盤等資訊；
將ceph-mon01的/dev/sdb盤添加為集群osd
[cephadm@ceph-admin ceph-cluster]$ ceph-deploy osd create ceph-mon01 --data /dev/sdb
查看集群狀態
提示：可以看到現在集群osd有一個正常，存儲空間為80G；說明我們剛才添加到osd已經成功；後續其他主機上的osd也是上述過程，先擦凈磁碟，然後在添加為osd；
列出對應主機上的osd資訊
[cephadm@ceph-admin ceph-cluster]$ ceph-deploy osd list ceph-mon01 [ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadm/.cephdeploy.conf [ceph_deploy.cli][INFO ] Invoked (2.0.1): /bin/ceph-deploy osd list ceph-mon01 [ceph_deploy.cli][INFO ] ceph-deploy options: [ceph_deploy.cli][INFO ] username : None [ceph_deploy.cli][INFO ] verbose : False [ceph_deploy.cli][INFO ] debug : False [ceph_deploy.cli][INFO ] overwrite_conf : False [ceph_deploy.cli][INFO ] subcommand : list [ceph_deploy.cli][INFO ] quiet : False [ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f01148f9128> [ceph_deploy.cli][INFO ] cluster : ceph [ceph_deploy.cli][INFO ] host : ['ceph-mon01'] [ceph_deploy.cli][INFO ] func : <function osd at 0x7f011493d9b0> [ceph_deploy.cli][INFO ] ceph_conf : None [ceph_deploy.cli][INFO ] default_release : False [ceph-mon01][DEBUG ] connection detected need for sudo [ceph-mon01][DEBUG ] connected to host: ceph-mon01 [ceph-mon01][DEBUG ] detect platform information from remote host [ceph-mon01][DEBUG ] detect machine type [ceph-mon01][DEBUG ] find the location of an executable [ceph_deploy.osd][INFO ] Distro info: CentOS Linux 7.9.2009 Core [ceph_deploy.osd][DEBUG ] Listing disks on ceph-mon01... [ceph-mon01][DEBUG ] find the location of an executable [ceph-mon01][INFO ] Running command: sudo /usr/sbin/ceph-volume lvm list [ceph-mon01][DEBUG ] [ceph-mon01][DEBUG ] [ceph-mon01][DEBUG ] ====== osd.0 ======= [ceph-mon01][DEBUG ] [ceph-mon01][DEBUG ] [block] /dev/ceph-56cdba71-749f-4c01-8364-f5bdad0b8f8d/osd-block-538baff0-ed25-4e3f-9ed7-f228a7ca0086 [ceph-mon01][DEBUG ] [ceph-mon01][DEBUG ] block device /dev/ceph-56cdba71-749f-4c01-8364-f5bdad0b8f8d/osd-block-538baff0-ed25-4e3f-9ed7-f228a7ca0086 [ceph-mon01][DEBUG ] block uuid 40cRBg-53ZO-Dbho-wWo6-gNcJ-ZJJi-eZC6Vt [ceph-mon01][DEBUG ] cephx lockbox secret [ceph-mon01][DEBUG ] cluster fsid 7fd4a619-9767-4b46-9cee-78b9dfe88f34 [ceph-mon01][DEBUG ] cluster name ceph [ceph-mon01][DEBUG ] crush device class None [ceph-mon01][DEBUG ] encrypted 0 [ceph-mon01][DEBUG ] osd fsid 538baff0-ed25-4e3f-9ed7-f228a7ca0086 [ceph-mon01][DEBUG ] osd id 0 [ceph-mon01][DEBUG ] type block [ceph-mon01][DEBUG ] vdo 0 [ceph-mon01][DEBUG ] devices /dev/sdb [cephadm@ceph-admin ceph-cluster]$
提示：到此我們RADOS集群相關組件就都部署完畢了；
管理osd ceph命令查看osd相關資訊
1、查看osd狀態
[root@ceph-mon01 ~]# ceph osd stat 10 osds: 10 up, 10 in; epoch: e56
提示：osds表示現有集群里osd總數；up表示活動在線的osd數量，in表示在集群內的osd數量；
2、查看osd編號
[root@ceph-mon01 ~]# ceph osd ls 0 1 2 3 4 5 6 7 8 9 [root@ceph-mon01 ~]#
3、查看osd映射狀態
[root@ceph-mon01 ~]# ceph osd dump epoch 56 fsid 7fd4a619-9767-4b46-9cee-78b9dfe88f34 created 2022-09-24 00:36:13.639715 modified 2022-09-24 02:29:38.086464 flags sortbitwise,recovery_deletes,purged_snapdirs crush_version 25 full_ratio 0.95 backfillfull_ratio 0.9 nearfull_ratio 0.85 require_min_compat_client jewel min_compat_client jewel require_osd_release mimic pool 1 'testpool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 16 pgp_num 16 last_change 42 flags hashpspool stripe_width 0 max_osd 10 osd.0 up in weight 1 up_from 55 up_thru 0 down_at 0 last_clean_interval [0,0) 192.168.0.71:6800/52355 172.16.30.71:6800/52355 172.16.30.71:6801/52355 192.168.0.71:6801/52355 exists,up bf3649af-e3f4-41a2-a5ce-8f1a316d344e osd.1 up in weight 1 up_from 9 up_thru 42 down_at 0 last_clean_interval [0,0) 192.168.0.71:6802/49913 172.16.30.71:6802/49913 172.16.30.71:6803/49913 192.168.0.71:6803/49913 exists,up 7293a12a-7b4e-4c86-82dc-0acc15c3349e osd.2 up in weight 1 up_from 13 up_thru 42 down_at 0 last_clean_interval [0,0) 192.168.0.72:6800/48196 172.16.30.72:6800/48196 172.16.30.72:6801/48196 192.168.0.72:6801/48196 exists,up 96c437c5-8e82-4486-910f-9e98d195e4f9 osd.3 up in weight 1 up_from 17 up_thru 55 down_at 0 last_clean_interval [0,0) 192.168.0.72:6802/48679 172.16.30.72:6802/48679 172.16.30.72:6803/48679 192.168.0.72:6803/48679 exists,up 4659d2a9-09c7-49d5-bce0-4d2e65f5198c osd.4 up in weight 1 up_from 21 up_thru 55 down_at 0 last_clean_interval [0,0) 192.168.0.73:6800/48122 172.16.30.73:6800/48122 172.16.30.73:6801/48122 192.168.0.73:6801/48122 exists,up de019aa8-3d2a-4079-a99e-ec2da2d4edb9 osd.5 up in weight 1 up_from 25 up_thru 55 down_at 0 last_clean_interval [0,0) 192.168.0.73:6802/48601 172.16.30.73:6802/48601 172.16.30.73:6803/48601 192.168.0.73:6803/48601 exists,up 119c8748-af3b-4ac4-ac74-6171c90c82cc osd.6 up in weight 1 up_from 29 up_thru 55 down_at 0 last_clean_interval [0,0) 192.168.0.74:6801/58248 172.16.30.74:6800/58248 172.16.30.74:6801/58248 192.168.0.74:6802/58248 exists,up 08d8dd8b-cdfe-4338-83c0-b1e2b5c2a799 osd.7 up in weight 1 up_from 33 up_thru 55 down_at 0 last_clean_interval [0,0) 192.168.0.74:6803/58727 172.16.30.74:6802/58727 172.16.30.74:6803/58727 192.168.0.74:6804/58727 exists,up 9de6cbd0-bb1b-49e9-835c-3e714a867393 osd.8 up in weight 1 up_from 37 up_thru 42 down_at 0 last_clean_interval [0,0) 192.168.0.75:6800/48268 172.16.30.75:6800/48268 172.16.30.75:6801/48268 192.168.0.75:6801/48268 exists,up 63aaa0b8-4e52-4d74-82a8-fbbe7b48c837 osd.9 up in weight 1 up_from 41 up_thru 42 down_at 0 last_clean_interval [0,0) 192.168.0.75:6802/48751 172.16.30.75:6802/48751 172.16.30.75:6803/48751 192.168.0.75:6803/48751 exists,up 6bf3204a-b64c-4808-a782-434a93ac578c [root@ceph-mon01 ~]#
刪除osd
1、停用設備
提示：可以看到我們將0號osd停用以後，對應集群狀態里osd就只有9個在集群里；
2、停止進程
提示：停用進程需要在對應主機上停止ceph-osd@{osd-num}；停止進程以後，對應集群狀態就能看到對應只有9個osd進程處於up狀態；
3、移除設備
提示：可以看到移除osd以後，對應集群狀態里就只有9個osd了；若類似如下的OSD的配置資訊存在於ceph.conf配置文件中，管理員在刪除OSD之後手動將其刪除。
[osd.1] host = {hostname}
不過，對於Luminous之前的版本來說，管理員需要依次手動執行如下步驟刪除OSD設備：
1. 於CRUSH運行圖中移除設備：ceph osd crush remove {name}
2. 移除OSD的認證key：ceph auth del osd.{osd-num}
3. 最後移除OSD設備：ceph osd rm {osd-num}
測試上傳下載數據對象
1、創建存儲池並設置PG數量為16個
[root@ceph-mon01 ~]# ceph osd pool create testpool 16 16 pool 'testpool' created [root@ceph-mon01 ~]# ceph osd pool ls testpool [root@ceph-mon01 ~]#
2、上傳文件到testpool
[root@ceph-mon01 ~]# rados put test /etc/issue -p testpool [root@ceph-mon01 ~]# rados ls -p testpool test [root@ceph-mon01 ~]#
提示：可以看到我們上傳/etc/issue文件到testpool存儲池中並命名為test，對應文件已將在testpool存儲中存在；說明上傳沒有問題；
3、獲取存儲池中數據對象的具體位置資訊
[root@ceph-mon01 ~]# ceph osd map testpool test osdmap e44 pool 'testpool' (1) object 'test' -> pg 1.40e8aab5 (1.5) -> up ([4,0,6], p4) acting ([4,0,6], p4) [root@ceph-mon01 ~]#
提示：可以看到test文件在testpool存儲中被分別存放編號為4、0、6的osd上去了；
4、下載文件到本地
[root@ceph-mon01 ~]# ls [root@ceph-mon01 ~]# rados get test test-down -p testpool [root@ceph-mon01 ~]# ls test-down [root@ceph-mon01 ~]# cat test-down \S Kernel \r on an \m [root@ceph-mon01 ~]#
5、刪除數據對象
[root@ceph-mon01 ~]# rados rm test -p testpool [root@ceph-mon01 ~]# rados ls -p testpool [root@ceph-mon01 ~]#
6、刪除存儲池
[root@ceph-mon01 ~]# ceph osd pool rm testpool Error EPERM: WARNING: this will *PERMANENTLY DESTROY* all data stored in pool testpool. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. [root@ceph-mon01 ~]# ceph osd pool rm testpool --yes-i-really-really-mean-it. Error EPERM: WARNING: this will *PERMANENTLY DESTROY* all data stored in pool testpool. If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it. [root@ceph-mon01 ~]#
提示：刪除存儲池命令存在數據丟失的風險，Ceph於是默認禁止此類操作。管理員需要在ceph.conf配置文件中啟用支援刪除存儲池的操作後，方可使用類似上述命令刪除存儲池；
擴展ceph集群
擴展mon節點
Ceph存儲集群需要至少運行一個Ceph Monitor和一個Ceph Manager，生產環境中，為了實現高可用性，Ceph存儲集群通常運行多個監視器，以免單監視器整個存儲集群崩潰。Ceph使用Paxos演算法，該演算法需要半數以上的監視器大於n/2，其中n為總監視器數量）才能形成法定人數。儘管此非必需，但奇數個監視器往往更好。「ceph-deploy mon add {ceph-nodes}」命令可以一次添加一個監視器節點到集群中。
[cephadm@ceph-admin ceph-cluster]$ ceph-deploy mon add ceph-mon02 [cephadm@ceph-admin ceph-cluster]$ ceph-deploy mon add ceph-mon03
查看監視器及法定人數相關狀態
[root@ceph-mon01 ~]# ceph quorum_status --format json-pretty
{
"election_epoch": 12,
"quorum": [
0,
1,
2
],
"quorum_names": [
"ceph-mon01",
"ceph-mon02",
"ceph-mon03"
],
"quorum_leader_name": "ceph-mon01",
"monmap": {
"epoch": 3,
"fsid": "7fd4a619-9767-4b46-9cee-78b9dfe88f34",
"modified": "2022-09-24 01:56:24.196075",
"created": "2022-09-24 00:36:13.210155",
"features": {
"persistent": [
"kraken",
"luminous",
"mimic",
"osdmap-prune"
],
"optional": []
},
"mons": [
{
"rank": 0,
"name": "ceph-mon01",
"addr": "192.168.0.71:6789/0",
"public_addr": "192.168.0.71:6789/0"
},
{
"rank": 1,
"name": "ceph-mon02",
"addr": "192.168.0.72:6789/0",
"public_addr": "192.168.0.72:6789/0"
},
{
"rank": 2,
"name": "ceph-mon03",
"addr": "192.168.0.73:6789/0",
"public_addr": "192.168.0.73:6789/0"
}
]
}
}
[root@ceph-mon01 ~]#
提示：可以看到現在有3個mon節點，其中mon01為leader節點，總共有3個選票；
擴展mgr節點
Ceph Manager守護進程以「Active/Standby」模式運行，部署其它ceph-mgr守護程式可確保在Active節點或其上的ceph-mgr守護進程故障時，其中的一個Standby實例可以在不中斷服務的情況下接管其任務。「ceph-deploy mgr create {new-manager-nodes}」命令可以一次添加多個Manager節點。
把ceph-mon01節點添加為mgr節點
[cephadm@ceph-admin ceph-cluster]$ ceph-deploy mgr create ceph-mon01
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadm/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (2.0.1): /bin/ceph-deploy mgr create ceph-mon01
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] mgr : [('ceph-mon01', 'ceph-mon01')]
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] subcommand : create
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7fba72e66950>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] func : <function mgr at 0x7fba736df230>
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.mgr][DEBUG ] Deploying mgr, cluster ceph hosts ceph-mon01:ceph-mon01
[ceph-mon01][DEBUG ] connection detected need for sudo
[ceph-mon01][DEBUG ] connected to host: ceph-mon01
[ceph-mon01][DEBUG ] detect platform information from remote host
[ceph-mon01][DEBUG ] detect machine type
[ceph_deploy.mgr][INFO ] Distro info: CentOS Linux 7.9.2009 Core
[ceph_deploy.mgr][DEBUG ] remote host will use systemd
[ceph_deploy.mgr][DEBUG ] deploying mgr bootstrap to ceph-mon01
[ceph-mon01][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-mon01][WARNIN] mgr keyring does not exist yet, creating one
[ceph-mon01][DEBUG ] create a keyring file
[ceph-mon01][DEBUG ] create path recursively if it doesn't exist
[ceph-mon01][INFO ] Running command: sudo ceph --cluster ceph --name client.bootstrap-mgr --keyring /var/lib/ceph/bootstrap-mgr/ceph.keyring auth get-or-create mgr.ceph-mon01 mon allow profile mgr osd allow * mds allow * -o /var/lib/ceph/mgr/ceph-ceph-mon01/keyring
[ceph-mon01][INFO ] Running command: sudo systemctl enable ceph-mgr@ceph-mon01
[ceph-mon01][WARNIN] Created symlink from /etc/systemd/system/ceph-mgr.target.wants/[email protected] to /usr/lib/systemd/system/[email protected].
[ceph-mon01][INFO ] Running command: sudo systemctl start ceph-mgr@ceph-mon01
[ceph-mon01][INFO ] Running command: sudo systemctl enable ceph.target
[cephadm@ceph-admin ceph-cluster]$
查看集群狀態
提示：可以看到現在集群有3個mgr；對應我們剛才加的cehp-mon01以standby的形式運行著；到此一個完整的RADOS集群就搭建好了，該集群現在有3個mon節點，3個mgr節點，10個osd；
