容器編排系統K8s之crd資源

  前文我們了解了k8s節點污點和pod的對節點污點容忍度相關話題,回顧請參考://www.cnblogs.com/qiuhom-1874/p/14255486.html;今天我們來聊一下擴展k8s相關話題;

  k8s上創建資源對象的過程

  我們知道在k8s上,資源是有類型的,不同類型的資源,其定義方式和使用的欄位各不相同;用戶創建一個資源,實際上就是把k8s抽象的資源做實例化,即把k8s抽象出來的資源,通過資源清單賦值,創建出來的對象就是我們實例化對應類型資源的結果;用戶創建一個資源,首先會把請求發送給apiserver,通過apiserver的認證、授權、准入控制以後,對應創建資源的定義就存放在etcd中,控制器通過watch機制監視apiserver上的資源變動,通過對應資源變動事件觸發對應類型資源的控制器將對應資源創建出來,並通過控制器內部的和解循環監控著對應資源狀態是否和用戶定義的期望狀態一樣,如果發現不一樣,內部和解循環就會被觸發,對應控制器會向apiserver發起創建資源的請求,將對應資源重建,讓對應資源的狀態始終滿足用戶期望的狀態;從上述的過程來看,用戶創建一個資源分兩個步驟,第一步是將對應請求發送給apiserver,通過apiserver把對應資源定義的資訊存放在etcd中;第二個步驟是對應資源類型的控制器通過apiserver從etcd中讀取對應資源的定義,將其創建出來;對於etcd來說,它本是就一個kv資料庫,可以存儲任意類型的kv數據,但在在k8s上,apiserver將不同類型的資源定義抽象成不同的資源,使得用戶創建對應資源,必須是滿足對應類型資源定義的規範,然後將規範的定義存放在etcd中;簡單講apiserver就是把用戶存入etcd中的數據做了一層抽象,使得用戶不能夠隨意將任意數據存儲到etcd中,存入etcd中的數據必須是滿足對應apiserver介面定義的規範;這就好比我們在使用mysql資料庫時,必須遵守對應庫中的對應表的定義;

  在k8s上創建自定義資源類型

  在k8s上,資源的類型有很多,比如pod,service,PersistentVolume,PersistentVolumeClaim等等,這些都是一些基礎的資源類型;我們要創建某種資源,直接使用對應的資源類型,實例化一個對象即可;假如我們要在k8s上創建一個集群,我們是不是可以直接使用某種類型的資源,實例化一個集群對象呢?理論上是可以的,但是前提是對應k8s上有對應類型的資源;有對應類型的資源,用戶就可以把對應創建資源的定義存放在etcd中;除了有對應類型的資源,我們還需要有對應的控制器將對應資源創建出來;這樣一來對於不同集群或應用來說,其組織方式和邏輯都不同,使用的資源類型和控制器也有所不同;用戶要想實例化更高級的資源來,就必須自己手動實現定義其資源類型,將對應資源類型實例化為對象;除此之外必要時還需要自己實現對應資源的控制器;簡單講用戶想要實現更高級的資源類型,就必須擴展現有k8s的資源類型和控制器;

  在k8s上擴展資源類型的方式有三種,第一種是crd,crd是k8s內建的資源類型,該類型資源主要用來創建用戶自定義資源類型的資源;即通過crd資源,可以將用戶自定義資源類型轉換為k8s上資源類型;第二種是自定義apiserver;這種方式要比第一種方式要複雜一點,需要用戶手動開發程式實現對應功能的apiserver,讓其用戶創建自定義類型資源能夠通過自定義apiserver實現;第三種方式就是修改現有k8sapiserver,讓其支援對應用戶自定義資源類型;

  自定義控制器

  自定義資源類型我們可以使用crd資源實現,也可以使用自定義apiserver或修改原有apiserver程式碼實現,但是只有資源類型是不能夠讓對應自定義類型資源實例化為一個自定義資源對象,只有自定義資源類型,用戶創建對應資源類型的資源對象時,只能把對應資源類型的定義資訊寫入到etcd中,它不能真正的跑起來,要想真正的跑起來,我們還需要一個自定義控制器,專門負責監聽對應的資源類型的資源變化,將對應資源實例化為對應k8s上的資源對象;當然不是所有的自定義類型的資源都需要自定義控制器,如果對應自定義類型資源調用了底層基礎控制器來管控對應自定義資源,那麼對應自定義類型資源就不需要使用自定義控制器;我們知道控制器是k8s上的一個重要組件,它的工作邏輯是註冊監聽在apiserver上對應類型的資源變動,如果對應資源狀態不滿足用戶期望狀態,它就會根據內部的和解循環來請求apiserver將對應類型資源的定義發送給它,然後根據資源定義來重建對應的資源,讓其狀態始終和用戶期望的狀態保持一致;自定義控制器也是同樣的邏輯,使用自定義控制器的目的也是讓對應自定義類型資源能夠被自定義控制器監聽,一旦對應資源發生變動,它能夠將其在k8s上創建出來,並一直保持和用戶期望的狀態吻合;自定義控制器和自定義資源類型可以分開實現,也可以合併在一起實現,即自定義控制器程式能夠自動創建crd資源,讓其對應自定義類型資源能夠被k8s識別並將其創建出來;具體是分開實現還是合併在一起實現,取決開發自定義控制器程式設計師;

  crd資源定義幫助

[root@master01 ~]# kubectl explain crd
KIND:     CustomResourceDefinition
VERSION:  apiextensions.k8s.io/v1

DESCRIPTION:
     CustomResourceDefinition represents a resource that should be exposed on
     the API server. Its name MUST be in the format <.spec.name>.<.spec.group>.

FIELDS:
   apiVersion   <string>
     APIVersion defines the versioned schema of this representation of an
     object. Servers should convert recognized schemas to the latest internal
     value, and may reject unrecognized values. More info:
     //git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources

   kind <string>
     Kind is a string value representing the REST resource this object
     represents. Servers may infer this from the endpoint the client submits
     requests to. Cannot be updated. In CamelCase. More info:
     //git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds

   metadata     <Object>

   spec <Object> -required-
     spec describes how the user wants the resources to appear

   status       <Object>
     status indicates the actual state of the CustomResourceDefinition

[root@master01 ~]#

  提示:crd資源是k8s上的標準資源之一,它的定義主要有apiVersion,kind,metadata,spec和status;其中kind類型為CustomResourceDefinition,apiVersion是apiextensions.k8s.io/v1;這兩個是固定格式;spec欄位是用來定義對應指定資源類型資源的相關屬性;

  crd.spec欄位說明

[root@master01 ~]# kubectl explain crd.spec
KIND:     CustomResourceDefinition
VERSION:  apiextensions.k8s.io/v1

RESOURCE: spec <Object>

DESCRIPTION:
     spec describes how the user wants the resources to appear

     CustomResourceDefinitionSpec describes how a user wants their resource to
     appear

FIELDS:
   conversion   <Object>
     conversion defines conversion settings for the CRD.

   group        <string> -required-
     group is the API group of the defined custom resource. The custom resources
     are served under `/apis/<group>/...`. Must match the name of the
     CustomResourceDefinition (in the form `<names.plural>.<group>`).

   names        <Object> -required-
     names specify the resource and kind names for the custom resource.

   preserveUnknownFields        <boolean>
     preserveUnknownFields indicates that object fields which are not specified
     in the OpenAPI schema should be preserved when persisting to storage.
     apiVersion, kind, metadata and known fields inside metadata are always
     preserved. This field is deprecated in favor of setting
     `x-preserve-unknown-fields` to true in
     `spec.versions[*].schema.openAPIV3Schema`. See
     //kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/#pruning-versus-preserving-unknown-fields
     for details.

   scope        <string> -required-
     scope indicates whether the defined custom resource is cluster- or
     namespace-scoped. Allowed values are `Cluster` and `Namespaced`.

   versions     <[]Object> -required-
     versions is the list of all API versions of the defined custom resource.
     Version names are used to compute the order in which served versions are
     listed in API discovery. If the version string is "kube-like", it will sort
     above non "kube-like" version strings, which are ordered lexicographically.
     "Kube-like" versions start with a "v", then are followed by a number (the
     major version), then optionally the string "alpha" or "beta" and another
     number (the minor version). These are sorted first by GA > beta > alpha
     (where GA is a version with no suffix such as beta or alpha), and then by
     comparing major version, then minor version. An example sorted list of
     versions: v10, v2, v1, v11beta2, v10beta3, v3beta1, v12alpha1, v11alpha2,
     foo1, foo10.

[root@master01 ~]# 

  提示:crd.spec中group欄位使用來描述對應自定義類型資源的群組名稱,其值為字元串;names欄位是用來描述自定義類型資源對應類型,名稱等等,其值為一個對象;scope欄位用來定義對應自定義資源是那個級別的資源;該欄位的值只能為Cluster或Namespaced;versions欄位是用來指定對應自定義資源的版本,以及對應類型資源的屬性欄位等資訊,該欄位為一個列表對象;

  示例:定義一個自定義類型資源

[root@master01 ~]# cat crontab-crd.yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  # 名字必需與下面的 spec 欄位匹配,並且格式為 '<名稱的複數形式>.<組名>'
  name: crontabs.stable.example.com
spec:
  # 組名稱,用於 REST API: /apis/<組>/<版本>
  group: stable.example.com
  # 列舉此 CustomResourceDefinition 所支援的版本
  versions:
    - name: v1
      # 每個版本都可以通過 served 標誌來獨立啟用或禁止
      served: true
      # 其中一個且只有一個版本必需被標記為存儲版本
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                cronSpec:
                  type: string
                image:
                  type: string
                replicas:
                  type: integer
  # 可以是 Namespaced 或 Cluster
  scope: Namespaced
  names:
    # 名稱的複數形式,用於 URL:/apis/<組>/<版本>/<名稱的複數形式>
    plural: crontabs
    # 名稱的單數形式,作為命令行使用時和顯示時的別名
    singular: crontab
    # kind 通常是單數形式的駝峰編碼(CamelCased)形式。你的資源清單會使用這一形式。
    kind: CronTab
    # shortNames 允許你在命令行使用較短的字元串來匹配資源
    shortNames:
    - ct
[root@master01 ~]# 

  應用資源清單前使用kubectl get crontab

[root@master01 ~]# kubectl get crontab
error: the server doesn't have a resource type "crontab"
[root@master01 ~]# 

  提示:在沒有應用資源清單前使用kubectl get crontab,它會提示我們所沒有類型為crontab的資源

  應用資源清單

[root@master01 ~]# kubectl apply -f crontab-crd.yaml
customresourcedefinition.apiextensions.k8s.io/crontabs.stable.example.com created
[root@master01 ~]# kubectl get crontab
No resources found in default namespace.
[root@master01 ~]# 

  提示:應用資源清單後,再次使用kubectl get crontab就沒有報錯了,只是提示在default名稱空間沒有對應類型資源;

  查看crd

[root@master01 ~]# kubectl get crd
NAME                                                  CREATED AT
bgpconfigurations.crd.projectcalico.org               2021-01-03T15:49:21Z
bgppeers.crd.projectcalico.org                        2021-01-03T15:49:21Z
blockaffinities.crd.projectcalico.org                 2021-01-03T15:49:21Z
clusterinformations.crd.projectcalico.org             2021-01-03T15:49:21Z
crontabs.stable.example.com                           2021-01-12T12:39:00Z
felixconfigurations.crd.projectcalico.org             2021-01-03T15:49:21Z
globalnetworkpolicies.crd.projectcalico.org           2021-01-03T15:49:21Z
globalnetworksets.crd.projectcalico.org               2021-01-03T15:49:21Z
hostendpoints.crd.projectcalico.org                   2021-01-03T15:49:21Z
ipamblocks.crd.projectcalico.org                      2021-01-03T15:49:21Z
ipamconfigs.crd.projectcalico.org                     2021-01-03T15:49:21Z
ipamhandles.crd.projectcalico.org                     2021-01-03T15:49:21Z
ippools.crd.projectcalico.org                         2021-01-03T15:49:21Z
kubecontrollersconfigurations.crd.projectcalico.org   2021-01-03T15:49:21Z
networkpolicies.crd.projectcalico.org                 2021-01-03T15:49:21Z
networksets.crd.projectcalico.org                     2021-01-03T15:49:22Z
[root@master01 ~]# kubectl get crd/crontabs.stable.example.com
NAME                          CREATED AT
crontabs.stable.example.com   2021-01-12T12:39:00Z
[root@master01 ~]# 

  查看詳情

[root@master01 ~]# kubectl get crd/crontabs.stable.example.com
NAME                          CREATED AT
crontabs.stable.example.com   2021-01-12T12:39:00Z
[root@master01 ~]# kubectl describe crd/crontabs.stable.example.com
Name:         crontabs.stable.example.com
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  apiextensions.k8s.io/v1
Kind:         CustomResourceDefinition
Metadata:
  Creation Timestamp:  2021-01-12T12:39:00Z
  Generation:          1
  Managed Fields:
    API Version:  apiextensions.k8s.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        f:acceptedNames:
          f:kind:
          f:listKind:
          f:plural:
          f:shortNames:
          f:singular:
        f:conditions:
    Manager:      kube-apiserver
    Operation:    Update
    Time:         2021-01-12T12:39:00Z
    API Version:  apiextensions.k8s.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        f:conversion:
          .:
          f:strategy:
        f:group:
        f:names:
          f:kind:
          f:listKind:
          f:plural:
          f:shortNames:
          f:singular:
        f:scope:
        f:versions:
      f:status:
        f:storedVersions:
    Manager:         kubectl-client-side-apply
    Operation:       Update
    Time:            2021-01-12T12:39:00Z
  Resource Version:  805506
  UID:               b92a90f4-c953-4876-a496-030c9ba023fd
Spec:
  Conversion:
    Strategy:  None
  Group:       stable.example.com
  Names:
    Kind:       CronTab
    List Kind:  CronTabList
    Plural:     crontabs
    Short Names:
      ct
    Singular:  crontab
  Scope:       Namespaced
  Versions:
    Name:  v1
    Schema:
      openAPIV3Schema:
        Properties:
          Spec:
            Properties:
              Cron Spec:
                Type:  string
              Image:
                Type:  string
              Replicas:
                Type:  integer
            Type:      object
        Type:          object
    Served:            true
    Storage:           true
Status:
  Accepted Names:
    Kind:       CronTab
    List Kind:  CronTabList
    Plural:     crontabs
    Short Names:
      ct
    Singular:  crontab
  Conditions:
    Last Transition Time:  2021-01-12T12:39:00Z
    Message:               no conflicts found
    Reason:                NoConflicts
    Status:                True
    Type:                  NamesAccepted
    Last Transition Time:  2021-01-12T12:39:00Z
    Message:               the initial names have been accepted
    Reason:                InitialNamesAccepted
    Status:                True
    Type:                  Established
  Stored Versions:
    v1
Events:  <none>
[root@master01 ~]#

  使用自定義資源類型crontab創建資源

[root@master01 ~]# cat my-crontab.yaml
apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
  name: my-new-cron-object
spec:
  cronSpec: "* * * * */5"
  image: my-awesome-cron-image
[root@master01 ~]# 

  提示:上述資源清單表示創建一個類型為Crontab的資源,該資源的群組版本為stable.example.com/v1;

  應用資源清單

[root@master01 ~]# kubectl apply -f my-crontab.yaml
crontab.stable.example.com/my-new-cron-object created
[root@master01 ~]# kubectl get ct
NAME                 AGE
my-new-cron-object   5s
[root@master01 ~]# kubectl describe ct/my-new-cron-object
Name:         my-new-cron-object
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  stable.example.com/v1
Kind:         CronTab
Metadata:
  Creation Timestamp:  2021-01-12T12:45:29Z
  Generation:          1
  Managed Fields:
    API Version:  stable.example.com/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:cronSpec:
        f:image:
    Manager:         kubectl-client-side-apply
    Operation:       Update
    Time:            2021-01-12T12:45:29Z
  Resource Version:  806182
  UID:               31a88a3d-fa99-42b8-80f6-3e4559efdc40
Spec:
  Cron Spec:  * * * * */5
  Image:      my-awesome-cron-image
Events:       <none>
[root@master01 ~]# 

  提示:可以看到對應類型資源已經創建成功;以上示例只是單純的crd的使用示例,沒有任何實質的作用;

  部署自定義控制器

  示例:部署mongodb-aperator

  1、克隆項目

[root@master01 ~]# git clone //github.com/mongodb/mongodb-kubernetes-operator.git
Cloning into 'mongodb-kubernetes-operator'...
remote: Enumerating objects: 95, done.
remote: Counting objects: 100% (95/95), done.
remote: Compressing objects: 100% (74/74), done.
remote: Total 4506 (delta 30), reused 60 (delta 15), pack-reused 4411
Receiving objects: 100% (4506/4506), 18.04 MiB | 183.00 KiB/s, done.
Resolving deltas: 100% (2621/2621), done.
[root@master01 ~]# 

  2、創建名稱空間mongodb,並進入到mongodb-kubernetes-operator目錄應用crd資源,創建自定義資源類型

[root@master01 mongodb-kubernetes-operator]# kubectl create ns mongodb
namespace/mongodb created
[root@master01 mongodb-kubernetes-operator]# kubectl get ns
NAME                   STATUS   AGE
default                Active   35d
ingress-nginx          Active   22d
kube-node-lease        Active   35d
kube-public            Active   35d
kube-system            Active   35d
kubernetes-dashboard   Active   11d
mongodb                Active   4s
[root@master01 mongodb-kubernetes-operator]# ls
agent    build  deploy     docs    go.sum      pkg        release.json      scripts  testdata  version
APACHE2  cmd    dev_notes  go.mod  LICENSE.md  README.md  requirements.txt  test     tools.go
[root@master01 mongodb-kubernetes-operator]# kubectl apply -f deploy/crds/mongodb.com_mongodb_crd.yaml -n mongodb
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
customresourcedefinition.apiextensions.k8s.io/mongodb.mongodb.com created
[root@master01 mongodb-kubernetes-operator]# 

  驗證:查看mongodb類型資源是否已經創建成功?

[root@master01 mongodb-kubernetes-operator]# kubectl get crd
NAME                                                  CREATED AT
bgpconfigurations.crd.projectcalico.org               2021-01-03T15:49:21Z
bgppeers.crd.projectcalico.org                        2021-01-03T15:49:21Z
blockaffinities.crd.projectcalico.org                 2021-01-03T15:49:21Z
clusterinformations.crd.projectcalico.org             2021-01-03T15:49:21Z
crontabs.stable.example.com                           2021-01-12T12:39:00Z
felixconfigurations.crd.projectcalico.org             2021-01-03T15:49:21Z
globalnetworkpolicies.crd.projectcalico.org           2021-01-03T15:49:21Z
globalnetworksets.crd.projectcalico.org               2021-01-03T15:49:21Z
hostendpoints.crd.projectcalico.org                   2021-01-03T15:49:21Z
ipamblocks.crd.projectcalico.org                      2021-01-03T15:49:21Z
ipamconfigs.crd.projectcalico.org                     2021-01-03T15:49:21Z
ipamhandles.crd.projectcalico.org                     2021-01-03T15:49:21Z
ippools.crd.projectcalico.org                         2021-01-03T15:49:21Z
kubecontrollersconfigurations.crd.projectcalico.org   2021-01-03T15:49:21Z
mongodb.mongodb.com                                   2021-01-13T06:38:22Z
networkpolicies.crd.projectcalico.org                 2021-01-03T15:49:21Z
networksets.crd.projectcalico.org                     2021-01-03T15:49:22Z
[root@master01 mongodb-kubernetes-operator]# kubectl get crd/mongodb.mongodb.com
NAME                  CREATED AT
mongodb.mongodb.com   2021-01-13T06:38:22Z
[root@master01 mongodb-kubernetes-operator]# 

  3、安裝operator

[root@master01 mongodb-kubernetes-operator]# kubectl apply -f deploy/operator/ -n mongodb
deployment.apps/mongodb-kubernetes-operator created
role.rbac.authorization.k8s.io/mongodb-kubernetes-operator created
rolebinding.rbac.authorization.k8s.io/mongodb-kubernetes-operator created
serviceaccount/mongodb-kubernetes-operator created
[root@master01 mongodb-kubernetes-operator]# 

  提示:mongodb-kubernetes-operator這個項目是將自定義控制器和自定義資源類型分開實現的;其operator只負責創建和監聽對應資源類型的變化,在資源有變化時,實例化為對應資源對象,並保持對應資源對象狀態吻合用戶期望狀態;上述四個清單中主要是創建了一個sa賬戶,並對對應的sa用戶授權;

  operator.yaml內容

[root@master01 mongodb-kubernetes-operator]# cat deploy/operator/operator.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mongodb-kubernetes-operator
spec:
  replicas: 1
  selector:
    matchLabels:
      name: mongodb-kubernetes-operator
  template:
    metadata:
      labels:
        name: mongodb-kubernetes-operator
    spec:
      serviceAccountName: mongodb-kubernetes-operator
      containers:
        - name: mongodb-kubernetes-operator
          image: quay.io/mongodb/mongodb-kubernetes-operator:0.3.0
          command:
            - mongodb-kubernetes-operator
          imagePullPolicy: Always
          env:
            - name: WATCH_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: OPERATOR_NAME
              value: "mongodb-kubernetes-operator"
            - name: AGENT_IMAGE # The MongoDB Agent the operator will deploy to manage MongoDB deployments
              value: quay.io/mongodb/mongodb-agent:10.19.0.6562-1
            - name: VERSION_UPGRADE_HOOK_IMAGE
              value: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.2
            - name: MONGODB_IMAGE
              value: "library/mongo"
            - name: MONGODB_REPO_URL
              value: "registry.hub.docker.com"
[root@master01 mongodb-kubernetes-operator]# 

  提示:上述資源主要用deploy控制器運行對應自定義控制器為一個pod;

  驗證:查看operator是否正常運行

[root@master01 mongodb-kubernetes-operator]# kubectl get pods -n mongodb
NAME                                           READY   STATUS    RESTARTS   AGE
mongodb-kubernetes-operator-7d557bcc95-th8js   1/1     Running   0          26s
[root@master01 mongodb-kubernetes-operator]# 

  提示:能夠看到operator正常運行,就表示operator已經安裝成功;

  驗證:使用自定義資源類型創建一個mongodb 副本集集群

[root@master01 mongodb-kubernetes-operator]# cat deploy/crds/mongodb.com_v1_mongodb_cr.yaml 
---
apiVersion: mongodb.com/v1
kind: MongoDB
metadata:
  name: example-mongodb
spec:
  members: 3
  type: ReplicaSet
  version: "4.2.6"
  security:
    authentication:
      modes: ["SCRAM"]
  users:
    - name: my-user
      db: admin
      passwordSecretRef: # a reference to the secret that will be used to generate the user's password
        name: my-user-password
      roles:
        - name: clusterAdmin
          db: admin
        - name: userAdminAnyDatabase
          db: admin
      scramCredentialsSecretName: my-scram

# the user credentials will be generated from this secret
# once the credentials are generated, this secret is no longer required
---
apiVersion: v1
kind: Secret
metadata:
  name: my-user-password
type: Opaque
stringData:
  password: 58LObjiMpxcjP1sMDW
[root@master01 mongodb-kubernetes-operator]# kubectl apply -f deploy/crds/mongodb.com_v1_mongodb_cr.yaml
mongodb.mongodb.com/example-mongodb created
secret/my-user-password created
[root@master01 mongodb-kubernetes-operator]# 

  應用清單

[root@master01 mongodb-kubernetes-operator]# kubectl apply -f deploy/crds/mongodb.com_v1_mongodb_cr.yaml -n mongodb
mongodb.mongodb.com/example-mongodb created
secret/my-user-password created
[root@master01 mongodb-kubernetes-operator]# kubectl get pods -n mongodb
NAME                                           READY   STATUS    RESTARTS   AGE
example-mongodb-0                              0/2     Pending   0          9s
mongodb-kubernetes-operator-7d557bcc95-th8js   1/1     Running   0          88s
[root@master01 mongodb-kubernetes-operator]# 

  提示:這裡可以看到對應pod處於pending狀態;

  查看pod詳細資訊

[root@master01 mongodb-kubernetes-operator]# kubectl describe pod/example-mongodb-0 -n mongodb|grep -A 10 "Events"
Events:
  Type     Reason            Age                From               Message
  ----     ------            ----               ----               -------
  Warning  FailedScheduling  66s (x2 over 66s)  default-scheduler  0/5 nodes are available: 5 pod has unbound immediate PersistentVolumeClaims.
[root@master01 mongodb-kubernetes-operator]# 

  提示:這裡提示沒有可以用的pvc;

  刪除mongodb名稱空間下pvc

[root@master01 mongodb-kubernetes-operator]# kubectl get pvc -n mongodb
NAME                            STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-volume-example-mongodb-0   Pending                                                     92s
[root@master01 mongodb-kubernetes-operator]# kubectl delete pvc --all -n mongodb
persistentvolumeclaim "data-volume-example-mongodb-0" deleted
[root@master01 mongodb-kubernetes-operator]# kubectl get pvc -n mongodb
No resources found in mongodb namespace.
[root@master01 mongodb-kubernetes-operator]# 

  創建pv和pvc

[root@master01 mongodb-kubernetes-operator]# cd
[root@master01 ~]# cat pv-demo.yaml 
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv-v1
  labels:
    app: example-mongodb-svc
spec:
  capacity:
    storage: 1Gi
  volumeMode: Filesystem
  accessModes: ["ReadWriteOnce","ReadWriteMany","ReadOnlyMany"]
  persistentVolumeReclaimPolicy: Retain
  mountOptions:
  - hard
  - nfsvers=4.1
  nfs:
    path: /data/v1
    server: 192.168.0.99
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv-v2
  labels:
    app: example-mongodb-svc
spec:
  capacity:
    storage: 1Gi
  volumeMode: Filesystem
  accessModes: ["ReadWriteOnce","ReadWriteMany","ReadOnlyMany"]
  persistentVolumeReclaimPolicy: Retain
  mountOptions:
  - hard
  - nfsvers=4.1
  nfs:
    path: /data/v2
    server: 192.168.0.99
---

apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv-v3
  labels:
    app: example-mongodb-svc
spec:
  capacity:
    storage: 1Gi
  volumeMode: Filesystem
  accessModes: ["ReadWriteOnce","ReadWriteMany","ReadOnlyMany"]
  persistentVolumeReclaimPolicy: Retain
  mountOptions:
  - hard
  - nfsvers=4.1
  nfs:
    path: /data/v3
    server: 192.168.0.99
[root@master01 ~]# 

  應用清單創建pv

[root@master01 ~]# kubectl apply -f pv-demo.yaml
persistentvolume/nfs-pv-v1 created
persistentvolume/nfs-pv-v2 created
persistentvolume/nfs-pv-v3 created
[root@master01 ~]# kubectl get pv
NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
nfs-pv-v1   1Gi        RWO,ROX,RWX    Retain           Available                                   3s
nfs-pv-v2   1Gi        RWO,ROX,RWX    Retain           Available                                   3s
nfs-pv-v3   1Gi        RWO,ROX,RWX    Retain           Available                                   3s
[root@master01 ~]# 

  創建pvc清單

[root@master01 ~]# cat pvc-demo.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-volume-example-mongodb-0 
  namespace: mongodb
spec:
  accessModes:
    - ReadWriteMany
  volumeMode: Filesystem
  resources:
    requests:
      storage: 500Mi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-volume-example-mongodb-1
  namespace: mongodb
spec:
  accessModes:
    - ReadWriteMany
  volumeMode: Filesystem
  resources:
    requests:
      storage: 500Mi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-volume-example-mongodb-2
  namespace: mongodb
spec:
  accessModes:
    - ReadWriteMany
  volumeMode: Filesystem
  resources:
    requests:
      storage: 500Mi
[root@master01 ~]# 

  應用清單創建pvc

[root@master01 ~]# kubectl get pvc -n mongodb
No resources found in mongodb namespace.
[root@master01 ~]# kubectl apply -f pvc-demo.yaml
persistentvolumeclaim/data-volume-example-mongodb-0 created
persistentvolumeclaim/data-volume-example-mongodb-1 created
persistentvolumeclaim/data-volume-example-mongodb-2 created
[root@master01 ~]# kubectl get pvc -n mongodb    
NAME                            STATUS   VOLUME      CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-volume-example-mongodb-0   Bound    nfs-pv-v1   1Gi        RWO,ROX,RWX                   6s
data-volume-example-mongodb-1   Bound    nfs-pv-v2   1Gi        RWO,ROX,RWX                   6s
data-volume-example-mongodb-2   Bound    nfs-pv-v3   1Gi        RWO,ROX,RWX                   6s
[root@master01 ~]# kubectl get pv
NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                   STORAGECLASS   REASON   AGE
nfs-pv-v1   1Gi        RWO,ROX,RWX    Retain           Bound    mongodb/data-volume-example-mongodb-0                           102s
nfs-pv-v2   1Gi        RWO,ROX,RWX    Retain           Bound    mongodb/data-volume-example-mongodb-1                           102s
nfs-pv-v3   1Gi        RWO,ROX,RWX    Retain           Bound    mongodb/data-volume-example-mongodb-2                           102s
[root@master01 ~]# 

  提示:可以看到對應pvc和pv已經綁定好了;

  驗證:查看mongodb副本集集群是否跑起來了?

[root@master01 ~]# kubectl get pods -n mongodb
NAME                                           READY   STATUS            RESTARTS   AGE
example-mongodb-0                              2/2     Running           0          6m19s
example-mongodb-1                              0/2     PodInitializing   0          10s
mongodb-kubernetes-operator-7d557bcc95-th8js   1/1     Running           0          7m38s
[root@master01 ~]# kubectl get pods -n mongodb -w
NAME                                           READY   STATUS    RESTARTS   AGE
example-mongodb-0                              2/2     Running   0          6m35s
example-mongodb-1                              1/2     Running   0          26s
mongodb-kubernetes-operator-7d557bcc95-th8js   1/1     Running   0          7m54s
example-mongodb-1                              2/2     Running   0          43s
example-mongodb-2                              0/2     Pending   0          0s
example-mongodb-2                              0/2     Pending   0          0s
example-mongodb-2                              0/2     Init:0/1   0          0s
example-mongodb-2                              0/2     Init:0/1   0          1s
example-mongodb-2                              0/2     Terminating   0          4s
example-mongodb-2                              0/2     Terminating   0          6s
example-mongodb-2                              0/2     Terminating   0          20s
example-mongodb-2                              0/2     Terminating   0          20s
example-mongodb-2                              0/2     Pending       0          0s
example-mongodb-2                              0/2     Pending       0          0s
example-mongodb-2                              0/2     Init:0/1      0          0s
example-mongodb-2                              0/2     Init:0/1      0          1s
example-mongodb-2                              0/2     PodInitializing   0          7s
example-mongodb-2                              1/2     Running           0          14s
example-mongodb-2                              2/2     Running           0          36s
^C[root@master01 ~]# kubectl get pods -n mongodb 
NAME                                           READY   STATUS    RESTARTS   AGE
example-mongodb-0                              2/2     Running   0          8m
example-mongodb-1                              2/2     Running   0          111s
example-mongodb-2                              2/2     Running   0          48s
mongodb-kubernetes-operator-7d557bcc95-th8js   1/1     Running   0          9m19s
[root@master01 ~]# 

  提示:可以看到對應的pod已經正常跑起來了;

  驗證:使用mongo 連接mongodbpod,看看對應副本集集群是否工作正常?

[root@master01 ~]# kubectl get pods -n mongodb -o wide
NAME                                           READY   STATUS    RESTARTS   AGE     IP             NODE             NOMINATED NODE   READINESS GATES
example-mongodb-0                              2/2     Running   0          9m12s   10.244.4.101   node04.k8s.org   <none>           <none>
example-mongodb-1                              2/2     Running   0          3m3s    10.244.2.130   node02.k8s.org   <none>           <none>
example-mongodb-2                              2/2     Running   0          2m      10.244.1.130   node01.k8s.org   <none>           <none>
mongodb-kubernetes-operator-7d557bcc95-th8js   1/1     Running   0          10m     10.244.3.116   node03.k8s.org   <none>           <none>
[root@master01 ~]# mongo 10.244.4.101
MongoDB shell version v4.4.3
connecting to: mongodb://10.244.4.101:27017/test?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("b9d16fe9-6a74-4638-96e6-70aaf3c83bfa") }
MongoDB server version: 4.2.6
WARNING: shell and server versions do not match
example-mongodb:PRIMARY> show dbs
example-mongodb:PRIMARY> db.auth('my-user','58LObjiMpxcjP1sMDW')
Error: Authentication failed.
0
example-mongodb:PRIMARY> use admin
switched to db admin
example-mongodb:PRIMARY> db.auth('my-user','58LObjiMpxcjP1sMDW')
1
example-mongodb:PRIMARY> show dbs
admin   0.000GB
config  0.000GB
local   0.000GB
example-mongodb:PRIMARY> db.isMaster()
{
        "hosts" : [
                "example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017",
                "example-mongodb-1.example-mongodb-svc.mongodb.svc.cluster.local:27017",
                "example-mongodb-2.example-mongodb-svc.mongodb.svc.cluster.local:27017"
        ],
        "setName" : "example-mongodb",
        "setVersion" : 1,
        "ismaster" : true,
        "secondary" : false,
        "primary" : "example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017",
        "me" : "example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017",
        "electionId" : ObjectId("7fffffff0000000000000003"),
        "lastWrite" : {
                "opTime" : {
                        "ts" : Timestamp(1610520741, 1),
                        "t" : NumberLong(3)
                },
                "lastWriteDate" : ISODate("2021-01-13T06:52:21Z"),
                "majorityOpTime" : {
                        "ts" : Timestamp(1610520741, 1),
                        "t" : NumberLong(3)
                },
                "majorityWriteDate" : ISODate("2021-01-13T06:52:21Z")
        },
        "maxBsonObjectSize" : 16777216,
        "maxMessageSizeBytes" : 48000000,
        "maxWriteBatchSize" : 100000,
        "localTime" : ISODate("2021-01-13T06:52:27.873Z"),
        "logicalSessionTimeoutMinutes" : 30,
        "connectionId" : 153,
        "minWireVersion" : 0,
        "maxWireVersion" : 8,
        "readOnly" : false,
        "ok" : 1,
        "$clusterTime" : {
                "clusterTime" : Timestamp(1610520741, 1),
                "signature" : {
                        "hash" : BinData(0,"EcWzL7O9Ue9kmm6cQ4FumkcIP6g="),
                        "keyId" : NumberLong("6917119940596072451")
                }
        },
        "operationTime" : Timestamp(1610520741, 1)
}
example-mongodb:PRIMARY> 

  提示:可以看到三個mongodb pod是副本集關係;其中example-mongodb-0是主節點,其他兩個是從節點;

  最後還是說一下,我在做上面的實驗中,雖然mongodb operator已經正常工作,但是用mongo這個客戶端工具連接到主節點沒法寫數據,提示沒有許可權;但對應的用戶在對應庫是有讀寫許可權的;在admin庫下創建用戶能提示用戶添加成功,但是過幾秒查詢用戶資訊,發現用戶不存在,不知道什麼原因,有知道的朋友麻煩告知下(部落客郵箱:[email protected]),部落客將感激不盡。。