容器編排系統K8s之crd資源
前文我們了解了k8s節點污點和pod的對節點污點容忍度相關話題，回顧請參考：//www.cnblogs.com/qiuhom-1874/p/14255486.html；今天我們來聊一下擴展k8s相關話題；
k8s上創建資源對象的過程
我們知道在k8s上，資源是有類型的，不同類型的資源，其定義方式和使用的欄位各不相同；用戶創建一個資源，實際上就是把k8s抽象的資源做實例化，即把k8s抽象出來的資源，通過資源清單賦值，創建出來的對象就是我們實例化對應類型資源的結果；用戶創建一個資源，首先會把請求發送給apiserver，通過apiserver的認證、授權、准入控制以後，對應創建資源的定義就存放在etcd中，控制器通過watch機制監視apiserver上的資源變動，通過對應資源變動事件觸發對應類型資源的控制器將對應資源創建出來，並通過控制器內部的和解循環監控著對應資源狀態是否和用戶定義的期望狀態一樣，如果發現不一樣，內部和解循環就會被觸發，對應控制器會向apiserver發起創建資源的請求，將對應資源重建，讓對應資源的狀態始終滿足用戶期望的狀態；從上述的過程來看，用戶創建一個資源分兩個步驟，第一步是將對應請求發送給apiserver，通過apiserver把對應資源定義的資訊存放在etcd中；第二個步驟是對應資源類型的控制器通過apiserver從etcd中讀取對應資源的定義，將其創建出來；對於etcd來說，它本是就一個kv資料庫，可以存儲任意類型的kv數據，但在在k8s上，apiserver將不同類型的資源定義抽象成不同的資源，使得用戶創建對應資源，必須是滿足對應類型資源定義的規範，然後將規範的定義存放在etcd中；簡單講apiserver就是把用戶存入etcd中的數據做了一層抽象，使得用戶不能夠隨意將任意數據存儲到etcd中，存入etcd中的數據必須是滿足對應apiserver介面定義的規範；這就好比我們在使用mysql資料庫時，必須遵守對應庫中的對應表的定義；
在k8s上創建自定義資源類型
在k8s上，資源的類型有很多，比如pod，service，PersistentVolume，PersistentVolumeClaim等等，這些都是一些基礎的資源類型；我們要創建某種資源，直接使用對應的資源類型，實例化一個對象即可；假如我們要在k8s上創建一個集群，我們是不是可以直接使用某種類型的資源，實例化一個集群對象呢？理論上是可以的，但是前提是對應k8s上有對應類型的資源；有對應類型的資源，用戶就可以把對應創建資源的定義存放在etcd中；除了有對應類型的資源，我們還需要有對應的控制器將對應資源創建出來；這樣一來對於不同集群或應用來說，其組織方式和邏輯都不同，使用的資源類型和控制器也有所不同；用戶要想實例化更高級的資源來，就必須自己手動實現定義其資源類型，將對應資源類型實例化為對象；除此之外必要時還需要自己實現對應資源的控制器；簡單講用戶想要實現更高級的資源類型，就必須擴展現有k8s的資源類型和控制器；
在k8s上擴展資源類型的方式有三種，第一種是crd，crd是k8s內建的資源類型，該類型資源主要用來創建用戶自定義資源類型的資源；即通過crd資源，可以將用戶自定義資源類型轉換為k8s上資源類型；第二種是自定義apiserver；這種方式要比第一種方式要複雜一點，需要用戶手動開發程式實現對應功能的apiserver，讓其用戶創建自定義類型資源能夠通過自定義apiserver實現；第三種方式就是修改現有k8sapiserver，讓其支援對應用戶自定義資源類型；
自定義控制器
自定義資源類型我們可以使用crd資源實現，也可以使用自定義apiserver或修改原有apiserver程式碼實現，但是只有資源類型是不能夠讓對應自定義類型資源實例化為一個自定義資源對象，只有自定義資源類型，用戶創建對應資源類型的資源對象時，只能把對應資源類型的定義資訊寫入到etcd中，它不能真正的跑起來，要想真正的跑起來，我們還需要一個自定義控制器，專門負責監聽對應的資源類型的資源變化，將對應資源實例化為對應k8s上的資源對象；當然不是所有的自定義類型的資源都需要自定義控制器，如果對應自定義類型資源調用了底層基礎控制器來管控對應自定義資源，那麼對應自定義類型資源就不需要使用自定義控制器；我們知道控制器是k8s上的一個重要組件，它的工作邏輯是註冊監聽在apiserver上對應類型的資源變動，如果對應資源狀態不滿足用戶期望狀態，它就會根據內部的和解循環來請求apiserver將對應類型資源的定義發送給它，然後根據資源定義來重建對應的資源，讓其狀態始終和用戶期望的狀態保持一致；自定義控制器也是同樣的邏輯，使用自定義控制器的目的也是讓對應自定義類型資源能夠被自定義控制器監聽，一旦對應資源發生變動，它能夠將其在k8s上創建出來，並一直保持和用戶期望的狀態吻合；自定義控制器和自定義資源類型可以分開實現，也可以合併在一起實現，即自定義控制器程式能夠自動創建crd資源，讓其對應自定義類型資源能夠被k8s識別並將其創建出來；具體是分開實現還是合併在一起實現，取決開發自定義控制器程式設計師；
crd資源定義幫助
[root@master01 ~]# kubectl explain crd
KIND: CustomResourceDefinition
VERSION: apiextensions.k8s.io/v1
DESCRIPTION:
CustomResourceDefinition represents a resource that should be exposed on
the API server. Its name MUST be in the format <.spec.name>.<.spec.group>.
FIELDS:
apiVersion <string>
APIVersion defines the versioned schema of this representation of an
object. Servers should convert recognized schemas to the latest internal
value, and may reject unrecognized values. More info:
//git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
kind <string>
Kind is a string value representing the REST resource this object
represents. Servers may infer this from the endpoint the client submits
requests to. Cannot be updated. In CamelCase. More info:
//git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
metadata <Object>
spec <Object> -required-
spec describes how the user wants the resources to appear
status <Object>
status indicates the actual state of the CustomResourceDefinition
[root@master01 ~]#
提示：crd資源是k8s上的標準資源之一，它的定義主要有apiVersion，kind，metadata，spec和status；其中kind類型為CustomResourceDefinition，apiVersion是apiextensions.k8s.io/v1；這兩個是固定格式；spec欄位是用來定義對應指定資源類型資源的相關屬性；
crd.spec欄位說明
[root@master01 ~]# kubectl explain crd.spec
KIND: CustomResourceDefinition
VERSION: apiextensions.k8s.io/v1
RESOURCE: spec <Object>
DESCRIPTION:
spec describes how the user wants the resources to appear
CustomResourceDefinitionSpec describes how a user wants their resource to
appear
FIELDS:
conversion <Object>
conversion defines conversion settings for the CRD.
group <string> -required-
group is the API group of the defined custom resource. The custom resources
are served under `/apis/<group>/...`. Must match the name of the
CustomResourceDefinition (in the form `<names.plural>.<group>`).
names <Object> -required-
names specify the resource and kind names for the custom resource.
preserveUnknownFields <boolean>
preserveUnknownFields indicates that object fields which are not specified
in the OpenAPI schema should be preserved when persisting to storage.
apiVersion, kind, metadata and known fields inside metadata are always
preserved. This field is deprecated in favor of setting
`x-preserve-unknown-fields` to true in
`spec.versions[*].schema.openAPIV3Schema`. See
//kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/#pruning-versus-preserving-unknown-fields
for details.
scope <string> -required-
scope indicates whether the defined custom resource is cluster- or
namespace-scoped. Allowed values are `Cluster` and `Namespaced`.
versions <[]Object> -required-
versions is the list of all API versions of the defined custom resource.
Version names are used to compute the order in which served versions are
listed in API discovery. If the version string is "kube-like", it will sort
above non "kube-like" version strings, which are ordered lexicographically.
"Kube-like" versions start with a "v", then are followed by a number (the
major version), then optionally the string "alpha" or "beta" and another
number (the minor version). These are sorted first by GA > beta > alpha
(where GA is a version with no suffix such as beta or alpha), and then by
comparing major version, then minor version. An example sorted list of
versions: v10, v2, v1, v11beta2, v10beta3, v3beta1, v12alpha1, v11alpha2,
foo1, foo10.
[root@master01 ~]#
提示：crd.spec中group欄位使用來描述對應自定義類型資源的群組名稱，其值為字元串；names欄位是用來描述自定義類型資源對應類型，名稱等等，其值為一個對象；scope欄位用來定義對應自定義資源是那個級別的資源；該欄位的值只能為Cluster或Namespaced；versions欄位是用來指定對應自定義資源的版本，以及對應類型資源的屬性欄位等資訊，該欄位為一個列表對象；
示例：定義一個自定義類型資源
[root@master01 ~]# cat crontab-crd.yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
# 名字必需與下面的 spec 欄位匹配，並且格式為 '<名稱的複數形式>.<組名>'
name: crontabs.stable.example.com
spec:
# 組名稱，用於 REST API: /apis/<組>/<版本>
group: stable.example.com
# 列舉此 CustomResourceDefinition 所支援的版本
versions:
- name: v1
# 每個版本都可以通過 served 標誌來獨立啟用或禁止
served: true
# 其中一個且只有一個版本必需被標記為存儲版本
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
cronSpec:
type: string
image:
type: string
replicas:
type: integer
# 可以是 Namespaced 或 Cluster
scope: Namespaced
names:
# 名稱的複數形式，用於 URL：/apis/<組>/<版本>/<名稱的複數形式>
plural: crontabs
# 名稱的單數形式，作為命令行使用時和顯示時的別名
singular: crontab
# kind 通常是單數形式的駝峰編碼（CamelCased）形式。你的資源清單會使用這一形式。
kind: CronTab
# shortNames 允許你在命令行使用較短的字元串來匹配資源
shortNames:
- ct
[root@master01 ~]#
應用資源清單前使用kubectl get crontab
[root@master01 ~]# kubectl get crontab error: the server doesn't have a resource type "crontab" [root@master01 ~]#
提示：在沒有應用資源清單前使用kubectl get crontab，它會提示我們所沒有類型為crontab的資源
應用資源清單
[root@master01 ~]# kubectl apply -f crontab-crd.yaml customresourcedefinition.apiextensions.k8s.io/crontabs.stable.example.com created [root@master01 ~]# kubectl get crontab No resources found in default namespace. [root@master01 ~]#
提示：應用資源清單後，再次使用kubectl get crontab就沒有報錯了，只是提示在default名稱空間沒有對應類型資源；
查看crd
[root@master01 ~]# kubectl get crd NAME CREATED AT bgpconfigurations.crd.projectcalico.org 2021-01-03T15:49:21Z bgppeers.crd.projectcalico.org 2021-01-03T15:49:21Z blockaffinities.crd.projectcalico.org 2021-01-03T15:49:21Z clusterinformations.crd.projectcalico.org 2021-01-03T15:49:21Z crontabs.stable.example.com 2021-01-12T12:39:00Z felixconfigurations.crd.projectcalico.org 2021-01-03T15:49:21Z globalnetworkpolicies.crd.projectcalico.org 2021-01-03T15:49:21Z globalnetworksets.crd.projectcalico.org 2021-01-03T15:49:21Z hostendpoints.crd.projectcalico.org 2021-01-03T15:49:21Z ipamblocks.crd.projectcalico.org 2021-01-03T15:49:21Z ipamconfigs.crd.projectcalico.org 2021-01-03T15:49:21Z ipamhandles.crd.projectcalico.org 2021-01-03T15:49:21Z ippools.crd.projectcalico.org 2021-01-03T15:49:21Z kubecontrollersconfigurations.crd.projectcalico.org 2021-01-03T15:49:21Z networkpolicies.crd.projectcalico.org 2021-01-03T15:49:21Z networksets.crd.projectcalico.org 2021-01-03T15:49:22Z [root@master01 ~]# kubectl get crd/crontabs.stable.example.com NAME CREATED AT crontabs.stable.example.com 2021-01-12T12:39:00Z [root@master01 ~]#
查看詳情
[root@master01 ~]# kubectl get crd/crontabs.stable.example.com
NAME CREATED AT
crontabs.stable.example.com 2021-01-12T12:39:00Z
[root@master01 ~]# kubectl describe crd/crontabs.stable.example.com
Name: crontabs.stable.example.com
Namespace:
Labels: <none>
Annotations: <none>
API Version: apiextensions.k8s.io/v1
Kind: CustomResourceDefinition
Metadata:
Creation Timestamp: 2021-01-12T12:39:00Z
Generation: 1
Managed Fields:
API Version: apiextensions.k8s.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
f:acceptedNames:
f:kind:
f:listKind:
f:plural:
f:shortNames:
f:singular:
f:conditions:
Manager: kube-apiserver
Operation: Update
Time: 2021-01-12T12:39:00Z
API Version: apiextensions.k8s.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:spec:
f:conversion:
.:
f:strategy:
f:group:
f:names:
f:kind:
f:listKind:
f:plural:
f:shortNames:
f:singular:
f:scope:
f:versions:
f:status:
f:storedVersions:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2021-01-12T12:39:00Z
Resource Version: 805506
UID: b92a90f4-c953-4876-a496-030c9ba023fd
Spec:
Conversion:
Strategy: None
Group: stable.example.com
Names:
Kind: CronTab
List Kind: CronTabList
Plural: crontabs
Short Names:
ct
Singular: crontab
Scope: Namespaced
Versions:
Name: v1
Schema:
openAPIV3Schema:
Properties:
Spec:
Properties:
Cron Spec:
Type: string
Image:
Type: string
Replicas:
Type: integer
Type: object
Type: object
Served: true
Storage: true
Status:
Accepted Names:
Kind: CronTab
List Kind: CronTabList
Plural: crontabs
Short Names:
ct
Singular: crontab
Conditions:
Last Transition Time: 2021-01-12T12:39:00Z
Message: no conflicts found
Reason: NoConflicts
Status: True
Type: NamesAccepted
Last Transition Time: 2021-01-12T12:39:00Z
Message: the initial names have been accepted
Reason: InitialNamesAccepted
Status: True
Type: Established
Stored Versions:
v1
Events: <none>
[root@master01 ~]#
使用自定義資源類型crontab創建資源
[root@master01 ~]# cat my-crontab.yaml apiVersion: "stable.example.com/v1" kind: CronTab metadata: name: my-new-cron-object spec: cronSpec: "* * * * */5" image: my-awesome-cron-image [root@master01 ~]#
提示：上述資源清單表示創建一個類型為Crontab的資源，該資源的群組版本為stable.example.com/v1;
應用資源清單
[root@master01 ~]# kubectl apply -f my-crontab.yaml
crontab.stable.example.com/my-new-cron-object created
[root@master01 ~]# kubectl get ct
NAME AGE
my-new-cron-object 5s
[root@master01 ~]# kubectl describe ct/my-new-cron-object
Name: my-new-cron-object
Namespace: default
Labels: <none>
Annotations: <none>
API Version: stable.example.com/v1
Kind: CronTab
Metadata:
Creation Timestamp: 2021-01-12T12:45:29Z
Generation: 1
Managed Fields:
API Version: stable.example.com/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:spec:
.:
f:cronSpec:
f:image:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2021-01-12T12:45:29Z
Resource Version: 806182
UID: 31a88a3d-fa99-42b8-80f6-3e4559efdc40
Spec:
Cron Spec: * * * * */5
Image: my-awesome-cron-image
Events: <none>
[root@master01 ~]#
提示：可以看到對應類型資源已經創建成功；以上示例只是單純的crd的使用示例，沒有任何實質的作用；
部署自定義控制器
示例：部署mongodb-aperator
1、克隆項目
[root@master01 ~]# git clone //github.com/mongodb/mongodb-kubernetes-operator.git Cloning into 'mongodb-kubernetes-operator'... remote: Enumerating objects: 95, done. remote: Counting objects: 100% (95/95), done. remote: Compressing objects: 100% (74/74), done. remote: Total 4506 (delta 30), reused 60 (delta 15), pack-reused 4411 Receiving objects: 100% (4506/4506), 18.04 MiB | 183.00 KiB/s, done. Resolving deltas: 100% (2621/2621), done. [root@master01 ~]#
2、創建名稱空間mongodb，並進入到mongodb-kubernetes-operator目錄應用crd資源，創建自定義資源類型
[root@master01 mongodb-kubernetes-operator]# kubectl create ns mongodb namespace/mongodb created [root@master01 mongodb-kubernetes-operator]# kubectl get ns NAME STATUS AGE default Active 35d ingress-nginx Active 22d kube-node-lease Active 35d kube-public Active 35d kube-system Active 35d kubernetes-dashboard Active 11d mongodb Active 4s [root@master01 mongodb-kubernetes-operator]# ls agent build deploy docs go.sum pkg release.json scripts testdata version APACHE2 cmd dev_notes go.mod LICENSE.md README.md requirements.txt test tools.go [root@master01 mongodb-kubernetes-operator]# kubectl apply -f deploy/crds/mongodb.com_mongodb_crd.yaml -n mongodb Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition customresourcedefinition.apiextensions.k8s.io/mongodb.mongodb.com created [root@master01 mongodb-kubernetes-operator]#
驗證：查看mongodb類型資源是否已經創建成功？
[root@master01 mongodb-kubernetes-operator]# kubectl get crd NAME CREATED AT bgpconfigurations.crd.projectcalico.org 2021-01-03T15:49:21Z bgppeers.crd.projectcalico.org 2021-01-03T15:49:21Z blockaffinities.crd.projectcalico.org 2021-01-03T15:49:21Z clusterinformations.crd.projectcalico.org 2021-01-03T15:49:21Z crontabs.stable.example.com 2021-01-12T12:39:00Z felixconfigurations.crd.projectcalico.org 2021-01-03T15:49:21Z globalnetworkpolicies.crd.projectcalico.org 2021-01-03T15:49:21Z globalnetworksets.crd.projectcalico.org 2021-01-03T15:49:21Z hostendpoints.crd.projectcalico.org 2021-01-03T15:49:21Z ipamblocks.crd.projectcalico.org 2021-01-03T15:49:21Z ipamconfigs.crd.projectcalico.org 2021-01-03T15:49:21Z ipamhandles.crd.projectcalico.org 2021-01-03T15:49:21Z ippools.crd.projectcalico.org 2021-01-03T15:49:21Z kubecontrollersconfigurations.crd.projectcalico.org 2021-01-03T15:49:21Z mongodb.mongodb.com 2021-01-13T06:38:22Z networkpolicies.crd.projectcalico.org 2021-01-03T15:49:21Z networksets.crd.projectcalico.org 2021-01-03T15:49:22Z [root@master01 mongodb-kubernetes-operator]# kubectl get crd/mongodb.mongodb.com NAME CREATED AT mongodb.mongodb.com 2021-01-13T06:38:22Z [root@master01 mongodb-kubernetes-operator]#
3、安裝operator
[root@master01 mongodb-kubernetes-operator]# kubectl apply -f deploy/operator/ -n mongodb deployment.apps/mongodb-kubernetes-operator created role.rbac.authorization.k8s.io/mongodb-kubernetes-operator created rolebinding.rbac.authorization.k8s.io/mongodb-kubernetes-operator created serviceaccount/mongodb-kubernetes-operator created [root@master01 mongodb-kubernetes-operator]#
提示：mongodb-kubernetes-operator這個項目是將自定義控制器和自定義資源類型分開實現的；其operator只負責創建和監聽對應資源類型的變化，在資源有變化時，實例化為對應資源對象，並保持對應資源對象狀態吻合用戶期望狀態；上述四個清單中主要是創建了一個sa賬戶，並對對應的sa用戶授權；
operator.yaml內容
[root@master01 mongodb-kubernetes-operator]# cat deploy/operator/operator.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: mongodb-kubernetes-operator
spec:
replicas: 1
selector:
matchLabels:
name: mongodb-kubernetes-operator
template:
metadata:
labels:
name: mongodb-kubernetes-operator
spec:
serviceAccountName: mongodb-kubernetes-operator
containers:
- name: mongodb-kubernetes-operator
image: quay.io/mongodb/mongodb-kubernetes-operator:0.3.0
command:
- mongodb-kubernetes-operator
imagePullPolicy: Always
env:
- name: WATCH_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: OPERATOR_NAME
value: "mongodb-kubernetes-operator"
- name: AGENT_IMAGE # The MongoDB Agent the operator will deploy to manage MongoDB deployments
value: quay.io/mongodb/mongodb-agent:10.19.0.6562-1
- name: VERSION_UPGRADE_HOOK_IMAGE
value: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.2
- name: MONGODB_IMAGE
value: "library/mongo"
- name: MONGODB_REPO_URL
value: "registry.hub.docker.com"
[root@master01 mongodb-kubernetes-operator]#
提示：上述資源主要用deploy控制器運行對應自定義控制器為一個pod；
驗證：查看operator是否正常運行
[root@master01 mongodb-kubernetes-operator]# kubectl get pods -n mongodb NAME READY STATUS RESTARTS AGE mongodb-kubernetes-operator-7d557bcc95-th8js 1/1 Running 0 26s [root@master01 mongodb-kubernetes-operator]#
提示：能夠看到operator正常運行，就表示operator已經安裝成功；
驗證：使用自定義資源類型創建一個mongodb 副本集集群
[root@master01 mongodb-kubernetes-operator]# cat deploy/crds/mongodb.com_v1_mongodb_cr.yaml
---
apiVersion: mongodb.com/v1
kind: MongoDB
metadata:
name: example-mongodb
spec:
members: 3
type: ReplicaSet
version: "4.2.6"
security:
authentication:
modes: ["SCRAM"]
users:
- name: my-user
db: admin
passwordSecretRef: # a reference to the secret that will be used to generate the user's password
name: my-user-password
roles:
- name: clusterAdmin
db: admin
- name: userAdminAnyDatabase
db: admin
scramCredentialsSecretName: my-scram
# the user credentials will be generated from this secret
# once the credentials are generated, this secret is no longer required
---
apiVersion: v1
kind: Secret
metadata:
name: my-user-password
type: Opaque
stringData:
password: 58LObjiMpxcjP1sMDW
[root@master01 mongodb-kubernetes-operator]# kubectl apply -f deploy/crds/mongodb.com_v1_mongodb_cr.yaml
mongodb.mongodb.com/example-mongodb created
secret/my-user-password created
[root@master01 mongodb-kubernetes-operator]#
應用清單
[root@master01 mongodb-kubernetes-operator]# kubectl apply -f deploy/crds/mongodb.com_v1_mongodb_cr.yaml -n mongodb mongodb.mongodb.com/example-mongodb created secret/my-user-password created [root@master01 mongodb-kubernetes-operator]# kubectl get pods -n mongodb NAME READY STATUS RESTARTS AGE example-mongodb-0 0/2 Pending 0 9s mongodb-kubernetes-operator-7d557bcc95-th8js 1/1 Running 0 88s [root@master01 mongodb-kubernetes-operator]#
提示：這裡可以看到對應pod處於pending狀態；
查看pod詳細資訊
[root@master01 mongodb-kubernetes-operator]# kubectl describe pod/example-mongodb-0 -n mongodb|grep -A 10 "Events" Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 66s (x2 over 66s) default-scheduler 0/5 nodes are available: 5 pod has unbound immediate PersistentVolumeClaims. [root@master01 mongodb-kubernetes-operator]#
提示：這裡提示沒有可以用的pvc；
刪除mongodb名稱空間下pvc
[root@master01 mongodb-kubernetes-operator]# kubectl get pvc -n mongodb NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE data-volume-example-mongodb-0 Pending 92s [root@master01 mongodb-kubernetes-operator]# kubectl delete pvc --all -n mongodb persistentvolumeclaim "data-volume-example-mongodb-0" deleted [root@master01 mongodb-kubernetes-operator]# kubectl get pvc -n mongodb No resources found in mongodb namespace. [root@master01 mongodb-kubernetes-operator]#
創建pv和pvc
[root@master01 mongodb-kubernetes-operator]# cd
[root@master01 ~]# cat pv-demo.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: nfs-pv-v1
labels:
app: example-mongodb-svc
spec:
capacity:
storage: 1Gi
volumeMode: Filesystem
accessModes: ["ReadWriteOnce","ReadWriteMany","ReadOnlyMany"]
persistentVolumeReclaimPolicy: Retain
mountOptions:
- hard
- nfsvers=4.1
nfs:
path: /data/v1
server: 192.168.0.99
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: nfs-pv-v2
labels:
app: example-mongodb-svc
spec:
capacity:
storage: 1Gi
volumeMode: Filesystem
accessModes: ["ReadWriteOnce","ReadWriteMany","ReadOnlyMany"]
persistentVolumeReclaimPolicy: Retain
mountOptions:
- hard
- nfsvers=4.1
nfs:
path: /data/v2
server: 192.168.0.99
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: nfs-pv-v3
labels:
app: example-mongodb-svc
spec:
capacity:
storage: 1Gi
volumeMode: Filesystem
accessModes: ["ReadWriteOnce","ReadWriteMany","ReadOnlyMany"]
persistentVolumeReclaimPolicy: Retain
mountOptions:
- hard
- nfsvers=4.1
nfs:
path: /data/v3
server: 192.168.0.99
[root@master01 ~]#
應用清單創建pv
[root@master01 ~]# kubectl apply -f pv-demo.yaml persistentvolume/nfs-pv-v1 created persistentvolume/nfs-pv-v2 created persistentvolume/nfs-pv-v3 created [root@master01 ~]# kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE nfs-pv-v1 1Gi RWO,ROX,RWX Retain Available 3s nfs-pv-v2 1Gi RWO,ROX,RWX Retain Available 3s nfs-pv-v3 1Gi RWO,ROX,RWX Retain Available 3s [root@master01 ~]#
創建pvc清單
[root@master01 ~]# cat pvc-demo.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: data-volume-example-mongodb-0
namespace: mongodb
spec:
accessModes:
- ReadWriteMany
volumeMode: Filesystem
resources:
requests:
storage: 500Mi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: data-volume-example-mongodb-1
namespace: mongodb
spec:
accessModes:
- ReadWriteMany
volumeMode: Filesystem
resources:
requests:
storage: 500Mi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: data-volume-example-mongodb-2
namespace: mongodb
spec:
accessModes:
- ReadWriteMany
volumeMode: Filesystem
resources:
requests:
storage: 500Mi
[root@master01 ~]#
應用清單創建pvc
[root@master01 ~]# kubectl get pvc -n mongodb No resources found in mongodb namespace. [root@master01 ~]# kubectl apply -f pvc-demo.yaml persistentvolumeclaim/data-volume-example-mongodb-0 created persistentvolumeclaim/data-volume-example-mongodb-1 created persistentvolumeclaim/data-volume-example-mongodb-2 created [root@master01 ~]# kubectl get pvc -n mongodb NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE data-volume-example-mongodb-0 Bound nfs-pv-v1 1Gi RWO,ROX,RWX 6s data-volume-example-mongodb-1 Bound nfs-pv-v2 1Gi RWO,ROX,RWX 6s data-volume-example-mongodb-2 Bound nfs-pv-v3 1Gi RWO,ROX,RWX 6s [root@master01 ~]# kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE nfs-pv-v1 1Gi RWO,ROX,RWX Retain Bound mongodb/data-volume-example-mongodb-0 102s nfs-pv-v2 1Gi RWO,ROX,RWX Retain Bound mongodb/data-volume-example-mongodb-1 102s nfs-pv-v3 1Gi RWO,ROX,RWX Retain Bound mongodb/data-volume-example-mongodb-2 102s [root@master01 ~]#
提示：可以看到對應pvc和pv已經綁定好了；
驗證：查看mongodb副本集集群是否跑起來了？
[root@master01 ~]# kubectl get pods -n mongodb NAME READY STATUS RESTARTS AGE example-mongodb-0 2/2 Running 0 6m19s example-mongodb-1 0/2 PodInitializing 0 10s mongodb-kubernetes-operator-7d557bcc95-th8js 1/1 Running 0 7m38s [root@master01 ~]# kubectl get pods -n mongodb -w NAME READY STATUS RESTARTS AGE example-mongodb-0 2/2 Running 0 6m35s example-mongodb-1 1/2 Running 0 26s mongodb-kubernetes-operator-7d557bcc95-th8js 1/1 Running 0 7m54s example-mongodb-1 2/2 Running 0 43s example-mongodb-2 0/2 Pending 0 0s example-mongodb-2 0/2 Pending 0 0s example-mongodb-2 0/2 Init:0/1 0 0s example-mongodb-2 0/2 Init:0/1 0 1s example-mongodb-2 0/2 Terminating 0 4s example-mongodb-2 0/2 Terminating 0 6s example-mongodb-2 0/2 Terminating 0 20s example-mongodb-2 0/2 Terminating 0 20s example-mongodb-2 0/2 Pending 0 0s example-mongodb-2 0/2 Pending 0 0s example-mongodb-2 0/2 Init:0/1 0 0s example-mongodb-2 0/2 Init:0/1 0 1s example-mongodb-2 0/2 PodInitializing 0 7s example-mongodb-2 1/2 Running 0 14s example-mongodb-2 2/2 Running 0 36s ^C[root@master01 ~]# kubectl get pods -n mongodb NAME READY STATUS RESTARTS AGE example-mongodb-0 2/2 Running 0 8m example-mongodb-1 2/2 Running 0 111s example-mongodb-2 2/2 Running 0 48s mongodb-kubernetes-operator-7d557bcc95-th8js 1/1 Running 0 9m19s [root@master01 ~]#
提示：可以看到對應的pod已經正常跑起來了；
驗證：使用mongo 連接mongodbpod，看看對應副本集集群是否工作正常？
[root@master01 ~]# kubectl get pods -n mongodb -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
example-mongodb-0 2/2 Running 0 9m12s 10.244.4.101 node04.k8s.org <none> <none>
example-mongodb-1 2/2 Running 0 3m3s 10.244.2.130 node02.k8s.org <none> <none>
example-mongodb-2 2/2 Running 0 2m 10.244.1.130 node01.k8s.org <none> <none>
mongodb-kubernetes-operator-7d557bcc95-th8js 1/1 Running 0 10m 10.244.3.116 node03.k8s.org <none> <none>
[root@master01 ~]# mongo 10.244.4.101
MongoDB shell version v4.4.3
connecting to: mongodb://10.244.4.101:27017/test?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("b9d16fe9-6a74-4638-96e6-70aaf3c83bfa") }
MongoDB server version: 4.2.6
WARNING: shell and server versions do not match
example-mongodb:PRIMARY> show dbs
example-mongodb:PRIMARY> db.auth('my-user','58LObjiMpxcjP1sMDW')
Error: Authentication failed.
0
example-mongodb:PRIMARY> use admin
switched to db admin
example-mongodb:PRIMARY> db.auth('my-user','58LObjiMpxcjP1sMDW')
1
example-mongodb:PRIMARY> show dbs
admin 0.000GB
config 0.000GB
local 0.000GB
example-mongodb:PRIMARY> db.isMaster()
{
"hosts" : [
"example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017",
"example-mongodb-1.example-mongodb-svc.mongodb.svc.cluster.local:27017",
"example-mongodb-2.example-mongodb-svc.mongodb.svc.cluster.local:27017"
],
"setName" : "example-mongodb",
"setVersion" : 1,
"ismaster" : true,
"secondary" : false,
"primary" : "example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017",
"me" : "example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017",
"electionId" : ObjectId("7fffffff0000000000000003"),
"lastWrite" : {
"opTime" : {
"ts" : Timestamp(1610520741, 1),
"t" : NumberLong(3)
},
"lastWriteDate" : ISODate("2021-01-13T06:52:21Z"),
"majorityOpTime" : {
"ts" : Timestamp(1610520741, 1),
"t" : NumberLong(3)
},
"majorityWriteDate" : ISODate("2021-01-13T06:52:21Z")
},
"maxBsonObjectSize" : 16777216,
"maxMessageSizeBytes" : 48000000,
"maxWriteBatchSize" : 100000,
"localTime" : ISODate("2021-01-13T06:52:27.873Z"),
"logicalSessionTimeoutMinutes" : 30,
"connectionId" : 153,
"minWireVersion" : 0,
"maxWireVersion" : 8,
"readOnly" : false,
"ok" : 1,
"$clusterTime" : {
"clusterTime" : Timestamp(1610520741, 1),
"signature" : {
"hash" : BinData(0,"EcWzL7O9Ue9kmm6cQ4FumkcIP6g="),
"keyId" : NumberLong("6917119940596072451")
}
},
"operationTime" : Timestamp(1610520741, 1)
}
example-mongodb:PRIMARY>
提示：可以看到三個mongodb pod是副本集關係；其中example-mongodb-0是主節點，其他兩個是從節點；
最後還是說一下，我在做上面的實驗中，雖然mongodb operator已經正常工作，但是用mongo這個客戶端工具連接到主節點沒法寫數據，提示沒有許可權；但對應的用戶在對應庫是有讀寫許可權的；在admin庫下創建用戶能提示用戶添加成功，但是過幾秒查詢用戶資訊，發現用戶不存在，不知道什麼原因，有知道的朋友麻煩告知下（部落客郵箱：[email protected]），部落客將感激不盡。。
