kubernetes之數據管理

 volume

emptyDir


[machangwei@mcwk8s-master ~]$ kubectl apply -f mcwVolume1.yml  #部署emptydir
pod/producer-consumer created
[machangwei@mcwk8s-master ~]$ cat mcwVolume1.yml 
apiVersion: v1
kind: Pod
metadata:
  name: producer-consumer
spec:
  containers:
  - image: busybox
    name: producer
    volumeMounts:
    - mountPath: /producer_dir
      name: shared-volume
    args:
    - /bin/sh
    - -c
    - echo "hello world" >/producer_dir/hello ; sleep 30000

  - image: busybox
    name: consumer
    volumeMounts:
    - mountPath: /consumer_dir
      name: shared-volume
    args:
    - /bin/sh
    - -c
    - cat /consumer_dir/hello ; sleep 30000

  volumes:
  - name: shared-volume
    emptyDir: {}

######定義一個volumes，使用空字典代表它是個空目錄，起個名字。規格容器下，有兩個鏡像代表兩個容器。這是pod種類資源。
#元數據名稱就是運行的pod的名稱。pod下兩個容器，下面這行結果，ready下應該是兩個容器，但是暫時是0個容器準備好。容器正在創建中
#容器下有鏡像。每個鏡像下做配置，一個鏡像的配置就是一個容器。鏡像下的名稱就是容器的名稱。指定邏輯卷掛載，指定這個容器的邏輯卷
#掛載路徑是容器里哪個目錄，需要掛載的邏輯卷名稱是哪個。後面定義邏輯卷的時候有定義它的名稱。然後這個容器的args參數，就是啟動容器後，
#要運行的命令，這裡生產者是每隔睡眠時間個單位，就寫入數據到掛載目錄下的某個文件。消費者就是每個睡眠個時間單位，去查看（消費）被掛載目錄的文件
#由於兩者是同一個volume掛載上去的，所以這個pod中的兩個容器共享這一個volume。而這個volume實質就是一個目錄，請往下看
[machangwei@mcwk8s-master ~]$ kubectl get pod
NAME                READY   STATUS              RESTARTS   AGE
producer-consumer   0/2     ContainerCreating   0          84s
[machangwei@mcwk8s-master ~]$ kubectl describe pod producer-consumer
Name:         producer-consumer
Namespace:    default
Priority:     0
Node:         mcwk8s-node1/10.0.0.5
Start Time:   Fri, 18 Feb 2022 20:20:38 +0800
Labels:       <none>
Annotations:  <none>
Status:       Running
IP:           10.244.1.15
IPs:
  IP:  10.244.1.15
Containers:
  producer:
    Container ID:  docker://e9a06c83f73861d15a115cc95665d19d8b6ef024546d339cc049d1571840cad7
    Image:         busybox
    Image ID:      docker-pullable://busybox@sha256:5acba83a746c7608ed544dc1533b87c737a0b0fb730301639a0179f9344b1678
    Port:          <none>
    Host Port:     <none>
    Args:
      /bin/sh
      -c
      echo "hello world" >/producer_dir/hello ; sleep 30000
    State:          Running
      Started:      Fri, 18 Feb 2022 20:21:54 +0800
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /producer_dir from shared-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-spkgq (ro)
  consumer:
    Container ID:  docker://634dc8fe053cc6ea1ee0f5f818cac6a7babbd5f84804579386382e2d2c6d1b40
    Image:         busybox
    Image ID:      docker-pullable://busybox@sha256:5acba83a746c7608ed544dc1533b87c737a0b0fb730301639a0179f9344b1678
    Port:          <none>
    Host Port:     <none>
    Args:
      /bin/sh
      -c
      cat /consumer_dir/hello ; sleep 30000
    State:          Running
      Started:      Fri, 18 Feb 2022 20:22:11 +0800
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /consumer_dir from shared-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-spkgq (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  shared-volume:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-spkgq:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  2m39s  default-scheduler  Successfully assigned default/producer-consumer to mcwk8s-node1
  Normal  Pulling    105s   kubelet            Pulling image "busybox"
  Normal  Pulled     89s    kubelet            Successfully pulled image "busybox" in 16.349119305s
  Normal  Created    89s    kubelet            Created container producer
  Normal  Started    83s    kubelet            Started container producer
  Normal  Pulling    83s    kubelet            Pulling image "busybox"
  Normal  Pulled     67s    kubelet            Successfully pulled image "busybox" in 16.122373015s
  Normal  Created    67s    kubelet            Created container consumer
  Normal  Started    66s    kubelet            Started container consumer
[machangwei@mcwk8s-master ~]$ kubectl get pod
NAME                READY   STATUS    RESTARTS   AGE
producer-consumer   2/2     Running   0          2m55s
[machangwei@mcwk8s-master ~]$ kubectl logs producer-consumer

error: a container name must be specified for pod producer-consumer, choose one of: [producer consumer]
[machangwei@mcwk8s-master ~]$ 
[machangwei@mcwk8s-master ~]$ kubectl logs producer-consumer consumer
hello world
[machangwei@mcwk8s-master ~]$ kubectl get pod -o wide #查看pod所在的host
NAME                READY   STATUS    RESTARTS   AGE     IP            NODE           NOMINATED NODE   READINESS GATES
producer-consumer   2/2     Running   0          6m19s   10.244.1.15   mcwk8s-node1   <none>           <none>


[root@mcwk8s-node1 ~]$ docker ps|grep producer-consumer  #pod部署主機上查看跟這個pod有關的容器。兩個容器和pod在容一個host
634dc8fe053c   busybox                                              "/bin/sh -c 'cat /co…"    13 minutes ago   Up 13 minutes             k8s_consumer_producer-consume_default_d32da9de-8574-458a-97c7-31cf9107f0a1_0
e9a06c83f738   busybox                                              "/bin/sh -c 'echo \"h…"   13 minutes ago   Up 13 minutes             k8s_producer_producer-consume_default_d32da9de-8574-458a-97c7-31cf9107f0a1_0
173864534fcb   registry.aliyuncs.com/google_containers/pause:3.6    "/pause"                  14 minutes ago   Up 13 minutes             k8s_POD_producer-consumer_default_d32da9de-8574-458a-97c7-31cf9107f0a1_0
#它們的命名時部署配置中鏡像下的名字是容器名，pod的POD,然後三者後面都是接pod名稱producer-consumer
#看下面的，可以看到生產這容器和消費者容器源目錄都是一樣的，掛載的目標目錄是各自容器中指定的目錄。有的時候，可能容器改變，可能源目錄和目標目錄
發生了變化，當容器好了後，應該就都是如下類似。的
[root@mcwk8s-node1 ~]$ docker inspect e9a06c8|grep  -iA 8 "mounts"
        "Mounts": [ 
            {
                "Type": "bind",
                "Source": "/var/lib/kubelet/pods/d32da9de-8574-458a-97c7-31cf9107f0a1/volumes/kubernetes.io~empty-dir/shared-volume",
                "Destination": "/producer_dir",
                "Mode": "Z",
                "RW": true,
                "Propagation": "rprivate"
            },
[root@mcwk8s-node1 ~]$ docker inspect 634dc8|grep  -iA 8 "mounts"
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/var/lib/kubelet/pods/d32da9de-8574-458a-97c7-31cf9107f0a1/volumes/kubernetes.io~empty-dir/shared-volume",
                "Destination": "/consumer_dir",
                "Mode": "Z",
                "RW": true,
                "Propagation": "rprivate"
            },
[root@mcwk8s-node1 ~]$

查看邏輯卷在物理機中的實際位置。以及兩個容器操作後的數據

[root@mcwk8s-node1 ~]$ ls /var/lib/kubelet/pods/d32da9de-8574-458a-97c7-31cf9107f0a1/volumes/kubernetes.io~empty-dir/shared-volume
hello
[root@mcwk8s-node1 ~]$ cat /var/lib/kubelet/pods/d32da9de-8574-458a-97c7-31cf9107f0a1/volumes/kubernetes.io~empty-dir/shared-volume/hello
hello world
[root@mcwk8s-node1 ~]$ ls /var/lib/kubelet/pods/d32da9de-8574-458a-97c7-31cf9107f0a1/volumes/
kubernetes.io~empty-dir kubernetes.io~projected

[root@mcwk8s-node1 ~]$ docker inspect 634dc8
[
    {
        "Id": "634dc8fe053cc6ea1ee0f5f818cac6a7babbd5f84804579386382e2d2c6d1b40",
        "Created": "2022-02-18T12:22:10.65373133Z",
        "Path": "/bin/sh",
        "Args": [
            "-c",
            "cat /consumer_dir/hello ; sleep 30000"
        ],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 91494,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2022-02-18T12:22:11.920497543Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:beae173ccac6ad749f76713cf4440fe3d21d1043fe616dfbe30775815d1d0f6a",
        "ResolvConfPath": "/var/lib/docker/containers/173864534fcb27ce32342a5b5ba9edc4adc3731c6a629f4175e3bedf173b1dc2/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/173864534fcb27ce32342a5b5ba9edc4adc3731c6a629f4175e3bedf173b1dc2/hostname",
        "HostsPath": "/var/lib/kubelet/pods/d32da9de-8574-458a-97c7-31cf9107f0a1/etc-hosts",
        "LogPath": "/var/lib/docker/containers/634dc8fe053cc6ea1ee0f5f818cac6a7babbd5f84804579386382e2d2c6d1b40/634dc8fe053cc6ea1ee0f5f818cac6a7babbd5f84804579386382e2d2c6d1b40-json.log",
        "Name": "/k8s_consumer_producer-consumer_default_d32da9de-8574-458a-97c7-31cf9107f0a1_0",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/var/lib/kubelet/pods/d32da9de-8574-458a-97c7-31cf9107f0a1/volumes/kubernetes.io~empty-dir/shared-volume:/consumer_dir:Z",
                "/var/lib/kubelet/pods/d32da9de-8574-458a-97c7-31cf9107f0a1/volumes/kubernetes.io~projected/kube-api-access-spkgq:/var/run/secrets/kubernetes.io/serviceaccount:ro,Z",
                "/var/lib/kubelet/pods/d32da9de-8574-458a-97c7-31cf9107f0a1/etc-hosts:/etc/hosts:Z",
                "/var/lib/kubelet/pods/d32da9de-8574-458a-97c7-31cf9107f0a1/containers/consumer/61a05b66:/dev/termination-log:Z"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "container:173864534fcb27ce32342a5b5ba9edc4adc3731c6a629f4175e3bedf173b1dc2",
            "PortBindings": null,
            "RestartPolicy": {
                "Name": "no",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "CgroupnsMode": "host",
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "container:173864534fcb27ce32342a5b5ba9edc4adc3731c6a629f4175e3bedf173b1dc2",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 1000,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": [
                "seccomp=unconfined"
            ],
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 2,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "kubepods-besteffort-podd32da9de_8574_458a_97c7_31cf9107f0a1.slice",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 100000,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "KernelMemory": 0,
            "KernelMemoryTCP": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": [
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/asound",
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/b45ab6cc1d2f691b9529c5dc413b8bcaa62350a62a43d9b2d45f2f2cdc7fb63d-init/diff:/var/lib/docker/overlay2/5c9e2add80e4eb9b40376cc60407679fdc4b510a0c146356397e8a769c1a307c/diff",
                "MergedDir": "/var/lib/docker/overlay2/b45ab6cc1d2f691b9529c5dc413b8bcaa62350a62a43d9b2d45f2f2cdc7fb63d/merged",
                "UpperDir": "/var/lib/docker/overlay2/b45ab6cc1d2f691b9529c5dc413b8bcaa62350a62a43d9b2d45f2f2cdc7fb63d/diff",
                "WorkDir": "/var/lib/docker/overlay2/b45ab6cc1d2f691b9529c5dc413b8bcaa62350a62a43d9b2d45f2f2cdc7fb63d/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/var/lib/kubelet/pods/d32da9de-8574-458a-97c7-31cf9107f0a1/volumes/kubernetes.io~empty-dir/shared-volume",
                "Destination": "/consumer_dir",
                "Mode": "Z",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/var/lib/kubelet/pods/d32da9de-8574-458a-97c7-31cf9107f0a1/volumes/kubernetes.io~projected/kube-api-access-spkgq",
                "Destination": "/var/run/secrets/kubernetes.io/serviceaccount",
                "Mode": "ro,Z",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/var/lib/kubelet/pods/d32da9de-8574-458a-97c7-31cf9107f0a1/etc-hosts",
                "Destination": "/etc/hosts",
                "Mode": "Z",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/var/lib/kubelet/pods/d32da9de-8574-458a-97c7-31cf9107f0a1/containers/consumer/61a05b66",
                "Destination": "/dev/termination-log",
                "Mode": "Z",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "producer-consumer",
            "Domainname": "",
            "User": "0",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "KUBERNETES_PORT_443_TCP_PROTO=tcp",
                "KUBERNETES_PORT_443_TCP_PORT=443",
                "KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1",
                "KUBERNETES_SERVICE_HOST=10.96.0.1",
                "KUBERNETES_SERVICE_PORT=443",
                "KUBERNETES_SERVICE_PORT_HTTPS=443",
                "KUBERNETES_PORT=tcp://10.96.0.1:443",
                "KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": [
                "/bin/sh",
                "-c",
                "cat /consumer_dir/hello ; sleep 30000"
            ],
            "Healthcheck": {
                "Test": [
                    "NONE"
                ]
            },
            "Image": "busybox@sha256:5acba83a746c7608ed544dc1533b87c737a0b0fb730301639a0179f9344b1678",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {
                "annotation.io.kubernetes.container.hash": "2c893f27",
                "annotation.io.kubernetes.container.restartCount": "0",
                "annotation.io.kubernetes.container.terminationMessagePath": "/dev/termination-log",
                "annotation.io.kubernetes.container.terminationMessagePolicy": "File",
                "annotation.io.kubernetes.pod.terminationGracePeriod": "30",
                "io.kubernetes.container.logpath": "/var/log/pods/default_producer-consumer_d32da9de-8574-458a-97c7-31cf9107f0a1/consumer/0.log",
                "io.kubernetes.container.name": "consumer",
                "io.kubernetes.docker.type": "container",
                "io.kubernetes.pod.name": "producer-consumer",
                "io.kubernetes.pod.namespace": "default",
                "io.kubernetes.pod.uid": "d32da9de-8574-458a-97c7-31cf9107f0a1",
                "io.kubernetes.sandbox.id": "173864534fcb27ce32342a5b5ba9edc4adc3731c6a629f4175e3bedf173b1dc2"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {}
        }
    }
]
[root@mcwk8s-node1 ~]$

描述某個容器的詳情

[root@mcwk8s-node1 ~]$ docker ps|grep producer-consumer  
634dc8fe053c   busybox                                              "/bin/sh -c 'cat /co…"    3 hours ago   Up 3 hours             k8s_consumer_producer-consumer_default_d32da9de-8574-458a-97c7-31cf9107f0a1_0
e9a06c83f738   busybox                                              "/bin/sh -c 'echo \"h…"   3 hours ago   Up 3 hours             k8s_producer_producer-consumer_default_d32da9de-8574-458a-97c7-31cf9107f0a1_0
173864534fcb   registry.aliyuncs.com/google_containers/pause:3.6    "/pause"                  3 hours ago   Up 3 hours             k8s_POD_producer-consumer_default_d32da9de-8574-458a-97c7-31cf9107f0a1_0
[root@mcwk8s-node1 ~]$ 
[root@mcwk8s-node1 ~]$ 
[root@mcwk8s-node1 ~]$ docker inspect 634dc8|grep  -iA 8 "mounts"
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/var/lib/kubelet/pods/d32da9de-8574-458a-97c7-31cf9107f0a1/volumes/kubernetes.io~empty-dir/shared-volume",
                "Destination": "/consumer_dir",
                "Mode": "Z",
                "RW": true,
                "Propagation": "rprivate"
            },
[root@mcwk8s-node1 ~]$ docker exec -it 634d cat /consumer_dir/hello 
hello world
[root@mcwk8s-node1 ~]$ ls /var/lib/kubelet/pods/d32da9de-8574-458a-97c7-31cf9107f0a1/volumes/kubernetes.io~empty-dir/shared-volume
hello
[root@mcwk8s-node1 ~]$ cat /var/lib/kubelet/pods/d32da9de-8574-458a-97c7-31cf9107f0a1/volumes/kubernetes.io~empty-dir/shared-volume/hello 
hello world
[root@mcwk8s-node1 ~]$ echo -e "   mcw">>/var/lib/kubelet/pods/d32da9de-8574-458a-97c7-31cf9107f0a1/volumes/kubernetes.io~empty-dir/shared-volume/hello
[root@mcwk8s-node1 ~]$ docker exec -it 634d cat /consumer_dir/hello  #修改host目錄，容器中顯示被修改後的
hello world
   mcw
[root@mcwk8s-node1 ~]$ docker exec -it 634d touch /consumer_dir/mcw.txt #容器中修改，host中也是修改後的
[root@mcwk8s-node1 ~]$ ls /var/lib/kubelet/pods/d32da9de-8574-458a-97c7-31cf9107f0a1/volumes/kubernetes.io~empty-dir/shared-volume
hello  mcw.txt
[root@mcwk8s-node1 ~]$

hostPath

[machangwei@mcwk8s-master ~]$ kubectl get pod --all-namespaces|grep apiserver  
kube-system   kube-apiserver-mcwk8s-master            1/1     Running   15 (110m ago)   28d
[machangwei@mcwk8s-master ~]$ kubectl  edit --namespace=kube-system pod kube-apiserver-mcwk8s-master
......
    volumeMounts: #我們查看apiserver服務的配置。 
    - mountPath: /etc/ssl/certs #可以看到掛載容器中的三個目錄，使用某個名稱的volume，是否只讀掛載
      name: ca-certs  #下面有定義用來掛載的volume
      readOnly: true
    - mountPath: /etc/pki
      name: etc-pki
      readOnly: true
    - mountPath: /etc/kubernetes/pki
      name: k8s-certs
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  hostNetwork: true
  nodeName: mcwk8s-master
  preemptionPolicy: PreemptLowerPriority
  priority: 2000001000
  priorityClassName: system-node-critical
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext:
    seccompProfile:
      type: RuntimeDefault
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    operator: Exists
  volumes:
  - hostPath:   #host主機path目錄被映射到使用這個容器的目錄中，起個名字後面通過名字使用這個host上的目錄
      path: /etc/ssl/certs
      type: DirectoryOrCreate
    name: ca-certs
  - hostPath:
      path: /etc/pki
      type: DirectoryOrCreate
    name: etc-pki
  - hostPath:
      path: /etc/kubernetes/pki
      type: DirectoryOrCreate
    name: k8s-certs

外部Storage Provider

雲硬碟存儲

PersistentVolume & PersistentVolumeClaim

persistent 英[pəˈsɪstənt] 美[pərˈsɪstənt]

 

adj. 持久的; 持續的; 堅持不懈的; 執著的; 不屈不撓的; 連綿的; 反覆出現的;

 

claim 英[kleɪm] 美[kleɪm]
v. 宣稱; 聲稱; 斷言; 要求(擁有); 索取; 認領; 索要; 引起（注意）; 獲得; 奪走，奪去（生命）;
n. 聲明; 宣稱; 斷言; (尤指對財產、土地等要求擁有的)所有權; (尤指向公司、政府等)索款，索賠;

PersistentVolume & PersistentVolumeClaim   pv &pvc

pv持久化volume，pvc持久化volume申請。

NFS PersistentVolume

nfs伺服器

參考：//www.cnblogs.com/machangwei-8/articles/15487295.html#_label5

[root@mcwk8s-master ~]$ showmount -e 10.0.0.4  #先部署好nfs伺服器，這裡nfs的共享目錄是/nfsdata
Export list for 10.0.0.4:
/nfsdata *
[root@mcwk8s-master ~]$

要使用外部存儲，這裡使用nfs掛載存儲，得有nfs伺服器。上面已經部署好了。

pv

[machangwei@mcwk8s-master ~]$ vim nfs-pv1.yml
[machangwei@mcwk8s-master ~]$ kubectl apply -f nfs-pv1.yml 
persistentvolume/mypv1 created
[machangwei@mcwk8s-master ~]$ cat nfs-pv1.yml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: mypv1
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Recycle
  storageClassName: nfs
  nfs:
    path: /nfsdata/pv1
    server: 10.0.0.4
[machangwei@mcwk8s-master ~]$ kubectl get pv
NAME    CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
mypv1   1Gi        RWO            Recycle          Available           nfs                     21s

配置pv，配置api版本，種類是持久化邏輯卷。元數據名稱就是邏輯卷名稱。

規格中容量存儲設置，訪問模式是什麼，這裡設置該pv只能讀寫模式掛載一個節點。模式還有ReadWriteMany 能以讀寫模式掛載到多個節點；ReadOnlyMany 能以只讀模式掛載到多個節點。

定義持久化邏輯卷重新聲明策略，即回收策略。Reain 需要管理員手工回；Recycle清除PV中的數據，；Delete刪除存儲提供者上對應的資源

定義存儲類名稱，這裡是使用nfs存儲。nfs的資訊需要設置，提供的掛載path是什麼，訪問服務地址是什麼

pvc

種類是持久化邏輯卷聲明；api版本多少，元數據名稱設置邏輯卷名稱；規則里設置訪問模式，這裡是以讀寫模式可以掛載到單個節點；資源是多少，請求存儲大小是多少設置。存儲類的名稱是nfs

[machangwei@mcwk8s-master ~]$ cat nfs-pvc1.yml 
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: mypvc1
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: nfs
[machangwei@mcwk8s-master ~]$ kubectl apply -f nfs-pvc1.yml 
persistentvolumeclaim/mypvc1 created
[machangwei@mcwk8s-master ~]$ kubectl get pvc
NAME     STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
mypvc1   Bound    mypv1    1Gi        RWO            nfs            8s
[machangwei@mcwk8s-master ~]$ kubectl get pvc -o wide  #查看到pvc綁定的pv，聲明之後就綁定了邏輯卷
NAME     STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE   VOLUMEMODE
mypvc1   Bound    mypv1    1Gi        RWO            nfs            18s   Filesystem
[machangwei@mcwk8s-master ~]$ kubectl get pv -o wide  #查看到pv被pvc綁定了，
NAME    CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM            STORAGECLASS   REASON   AGE     VOLUMEMODE
mypv1   1Gi        RWO            Recycle          Bound    default/mypvc1   nfs                     2m40s   Filesystem

pod還有報錯了

[machangwei@mcwk8s-master ~]$ cat pod1.yml 
kind: Pod
apiVersion: v1
metadata:
  name: mypod1
spec:
  containers:
    - name: mypod1
      image: busybox
      args:
      - /bin/sh
      - -c
      - sleep 30000
      volumeMounts:
      - mountPath: "/mydata"
        name: mydata
  volumes:
    - name: mydata
      persistentVolumeClaim:
        claimName: mypvc1

#pod中使用創建的pvc。種類是pod，元數據名稱設置pod名稱。規格里設置容器資訊。容器邏輯卷和鏡像是同一級下的，掛載容器中指定路徑是什麼，使用哪個邏輯卷名稱；下面定義了這個
#使用的邏輯卷，這個邏輯卷怎麼定義的？邏輯卷定義和容器同級，定義邏輯卷名稱，持久化邏輯卷聲明的名字是哪個，就是之前創建好的哪個pvc。這樣修改容器中的那個目錄，容器對應該主機的
#某個邏輯卷目錄，而該目錄又被掛載nfs掛載著，這樣容器就間接訪問到了nfs上的目錄
[machangwei@mcwk8s-master ~]$ kubectl apply -f pod1.yml 
pod/mypod1 created
[machangwei@mcwk8s-master ~]$ kubectl get pod -o wide
NAME     READY   STATUS              RESTARTS   AGE   IP       NODE           NOMINATED NODE   READINESS GATES
mypod1   0/1     ContainerCreating   0          16s   <none>   mcwk8s-node1   <none>           <none>
[machangwei@mcwk8s-master ~]$ kubectl get pod -o wide
NAME     READY   STATUS              RESTARTS   AGE   IP       NODE           NOMINATED NODE   READINESS GATES
mypod1   0/1     ContainerCreating   0          33s   <none>   mcwk8s-node1   <none>           <none>
[machangwei@mcwk8s-master ~]$ kubectl get pod -o wide
NAME     READY   STATUS              RESTARTS   AGE   IP       NODE           NOMINATED NODE   READINESS GATES
mypod1   0/1     ContainerCreating   0          62s   <none>   mcwk8s-node1   <none>           <none>
[machangwei@mcwk8s-master ~]$ kubectl describe pod mypod1
Name:         mypod1
Namespace:    default
Priority:     0
Node:         mcwk8s-node1/10.0.0.5
Start Time:   Sat, 19 Feb 2022 01:21:32 +0800
Labels:       <none>
Annotations:  <none>
Status:       Pending
IP:           
IPs:          <none>
Containers:
  mypod1:
    Container ID:  
    Image:         busybox
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Args:
      /bin/sh
      -c
      sleep 30000
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /mydata from mydata (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-28s6l (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  mydata:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  mypvc1
    ReadOnly:   false
  kube-api-access-28s6l:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age                From               Message
  ----     ------       ----               ----               -------
  Normal   Scheduled    80s                default-scheduler  Successfully assigned default/mypod1 to mcwk8s-node1
  Warning  FailedMount  13s (x8 over 77s)  kubelet            MountVolume.SetUp failed for volume "mypv1" : mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t nfs 10.0.0.4:/nfsdata/pv1 /var/lib/kubelet/pods/d1ba5034-28ad-4898-a7aa-3551bc330a4b/volumes/kubernetes.io~nfs/mypv1
Output: mount: wrong fs type, bad option, bad superblock on 10.0.0.4:/nfsdata/pv1,
       missing codepage or helper program, or other error
       (for several filesystems (e.g. nfs, cifs) you might
       need a /sbin/mount.<type> helper program)

       In some cases useful info is found in syslog - try
       dmesg | tail or so.
[machangwei@mcwk8s-master ~]$   #由上可知，就是將網路存儲掛載到host本地某個目錄

解決上面的pod部署掛載問題

缺少包

上面的原因是，由於部署pod在節點上，從節點上把nfs服務(在主上)掛載到節點1，
由於缺少包，無法掛載，導致pod無法在節點上做掛載。
在節點1無法掛載nfs
[root@mcwk8s-node1 ~]$ mount -t nfs 10.0.0.4:/nfsdata/ /root/mcw/
mount: wrong fs type, bad option, bad superblock on 10.0.0.4:/nfsdata/,
       missing codepage or helper program, or other error
       (for several filesystems (e.g. nfs, cifs) you might
       need a /sbin/mount.<type> helper program)

       In some cases useful info is found in syslog - try
       dmesg | tail or so.
原因，缺少包，安裝一下。
[root@mcwk8s-node1 ~]$ yum install nfs-utils ^C
[root@mcwk8s-node1 ~]$ mount -t nfs 10.0.0.4:/nfsdata/ /root/mcw/
[root@mcwk8s-node1 ~]$ df -h|tail -1 #可以正常掛載了
10.0.0.4:/nfsdata         19G  3.9G   15G  22% /root/mcw
[root@mcwk8s-node1 ~]$ df -h  #過一會看pv1已經掛載上了
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   19G  2.6G   16G  14% /
devtmpfs                 478M     0  478M   0% /dev
tmpfs                    489M     0  489M   0% /dev/shm
tmpfs                    489M   51M  438M  11% /run
tmpfs                    489M     0  489M   0% /sys/fs/cgroup
/dev/sda1                797M  125M  673M  16% /boot
tmpfs                    877M   12K  877M   1% /var/lib/kubelet/pods/9b74614a-6d0b-439c-8ea5-cf1583be7a5a/volumes/kubernetes.io~projected/kube-api-access-6hk5l
tmpfs                    877M   12K  877M   1% /var/lib/kubelet/pods/57f73623-751b-41a3-883e-62c025c49f92/volumes/kubernetes.io~projected/kube-api-access-wv8cc
overlay                   19G  2.6G   16G  14% /var/lib/docker/overlay2/c3a132cb506fe3db901260f4e8e40ab73c20b6dbb8904c545bd6c12e45af2ce9/merged
overlay                   19G  2.6G   16G  14% /var/lib/docker/overlay2/9c92a061f975e8f573fc9b895664adeeec375d7609c6e00fd5facada9a525a13/merged
shm                       64M     0   64M   0% /var/lib/docker/containers/fc176c43633f8e6362d47f6bbec6f2f556685bbe51564fe189cf58cd9bf69fa5/mounts/shm
shm                       64M     0   64M   0% /var/lib/docker/containers/3bbd8916f1ae8819e27eb1269afa524eca1c60a5d0b8df56dbbf314217c063a0/mounts/shm
overlay                   19G  2.6G   16G  14% /var/lib/docker/overlay2/914fc6440a30892ef51129fb445349c082d79ffb0fd79d62769384c7920e983a/merged
overlay                   19G  2.6G   16G  14% /var/lib/docker/overlay2/fe20b74227debccec19071237760ba6c37be72eadccc3999490a8aeae432919a/merged
tmpfs                     98M     0   98M   0% /run/user/0
tmpfs                    877M   12K  877M   1% /var/lib/kubelet/pods/d1ba5034-28ad-4898-a7aa-3551bc330a4b/volumes/kubernetes.io~projected/kube-api-access-28s6l
10.0.0.4:/nfsdata/pv1     19G  3.9G   15G  22% /var/lib/kubelet/pods/d1ba5034-28ad-4898-a7aa-3551bc330a4b/volumes/kubernetes.io~nfs/mypv1
overlay                   19G  2.6G   16G  14% /var/lib/docker/overlay2/b316299270c3541708659a9bbd6c65bb6d1fe4fa713460ad298866e1096adb0a/merged
shm                       64M     0   64M   0% /var/lib/docker/containers/b35f15c56b2e6f635dd346a9352fa0cbcb77ec9ba3f9a53b62f7bb53cb249e30/mounts/shm
overlay                   19G  2.6G   16G  14% /var/lib/docker/overlay2/bda202921f17c4ec866e22f2517af01a6df6434e4107b05588f4a001d8b5ec49/merged


但是驗證的時候，報錯只讀
[machangwei@mcwk8s-master ~]$ kubectl exec mypod1 touch /mydata/hello
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
touch: /mydata/hello: Read-only file system
command terminated with exit code 1

去節點上查看pod寫入不了
[root@mcwk8s-node1 ~]$ docker exec -it e09 touch /mydata/mcw.txt
touch: /mydata/mcw.txt: Read-only file system


這個無法從容器中寫入文件到nfs存儲里，跟容器有關嗎。下面直接通過掛載的方式，將某個目錄掛載上去了，但是寫入文件容易顯示只讀。說明這是nfs服務端的問題。

[root@mcwk8s-node1 ~]$ mount -t nfs 10.0.0.4:/nfsdata/pv1 /root/mcw/
[root@mcwk8s-node1 ~]$ df -h|grep mcw
10.0.0.4:/nfsdata/pv1 19G 3.9G 15G 22% /root/mcw
[root@mcwk8s-node1 ~]$ touch /root/mcw/test.txt
touch: cannot touch 『/root/mcw/test.txt』: Read-only file system
[root@mcwk8s-node1 ~]$

nfs問題導致的容器無法寫入數據到外部存儲的問題解決

這個無法從容器中寫入文件到nfs存儲里，跟容器有關嗎。下面直接通過掛載的方式，
將某個目錄掛載上去了，但是寫入文件容易顯示只讀。說明這是nfs服務端的問題。
[root@mcwk8s-master ~]$ cat /etc/exports
/nfsdata *
/nfsdata/pv1 *
[root@mcwk8s-master ~]$ vim /etc/exports
[root@mcwk8s-master ~]$ cat /etc/exports
/nfsdata * (rw,sync,all_squash)
/nfsdata/pv1 * (rw,sync,all_squash)
[root@mcwk8s-master ~]$ systemctl restart nfs
[root@mcwk8s-master ~]$ showmount -e 10.0.0.4
Export list for 10.0.0.4:
/nfsdata/pv1 *
/nfsdata     *

然後再去節點1上掛載發現還是不能寫入文件
[root@mcwk8s-node1 ~]$ mount  -t nfs 10.0.0.4:/nfsdata/pv1 /root/mcw/
[root@mcwk8s-node1 ~]$ df -h|grep mcw
10.0.0.4:/nfsdata/pv1     19G  3.9G   15G  22% /root/mcw
[root@mcwk8s-node1 ~]$ touch /root/mcw/test.txt
touch: cannot touch 『/root/mcw/test.txt』: Read-only file system

再次修改nfs配置，*代表所有ip網段，後面將小括弧里的設置，弄成和ip間是沒有空格的，這樣才生效了
[root@mcwk8s-master ~]$ vim /etc/exports
[root@mcwk8s-master ~]$ cat /etc/exports
/nfsdata *(rw,sync,all_squash)
/nfsdata/pv1 *(rw,sync,all_squash)
[root@mcwk8s-master ~]$ systemctl restart nfs
[root@mcwk8s-master ~]$ showmount -e 10.0.0.4
Export list for 10.0.0.4:
/nfsdata/pv1 *
/nfsdata     *
[root@mcwk8s-master ~]$ 

再次去節點1上訪問，發現成功創建
[root@mcwk8s-node1 ~]$ touch /root/mcw/test.txt
[root@mcwk8s-node1 ~]$ ls /root/mcw/
test.txt
[root@mcwk8s-node1 ~]$ df -h|grep mcw
10.0.0.4:/nfsdata/pv1     19G  3.9G   15G  22% /root/mcw

主節點上也可以看到這個文件
[root@mcwk8s-master ~]$ ls /nfsdata/pv1/
test.txt


也可以正常從容器中寫入文件到nfs伺服器
[machangwei@mcwk8s-master ~]$ kubectl get pod
NAME     READY   STATUS    RESTARTS   AGE
mypod1   1/1     Running   0          177m
[machangwei@mcwk8s-master ~]$ kubectl exec mypod1 ls /mydata
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
test.txt
[machangwei@mcwk8s-master ~]$ kubectl exec mypod1 touch /mydata/mcw.txt
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
[machangwei@mcwk8s-master ~]$ ls /nfsdata/pv1/
mcw.txt  test.txt


節點1上看掛載目錄，首先是10.0.0.4:/nfsdata/pv1這個目錄重新被掛載了，
雖然df查看只顯示/root/mcw，但是也是可以通過之前那個mypv1目錄去訪問到nfs伺服器上的文件的
當把/root/mcw卸載後，那麼之前那個mypv1就顯示出來了。並且可以正常使用
[root@mcwk8s-node1 ~]$ df -h|grep pv1
10.0.0.4:/nfsdata/pv1     19G  3.9G   15G  22% /root/mcw
[root@mcwk8s-node1 ~]$ ls /var/lib/kubelet/pods/d1ba5034-28ad-4898-a7aa-3551bc330a4b/volumes/kubernetes.io~nfs/mypv1
mcw.txt  test.txt
[root@mcwk8s-node1 ~]$ unmout /root/mcw/
-bash: unmout: command not found
[root@mcwk8s-node1 ~]$ unmount /root/mcw/
-bash: unmount: command not found
[root@mcwk8s-node1 ~]$ unmount /root/mcw/
unalias        unexpand       unicode_stop   unix2dos       unix_chkpwd    unlink         unshare        unxz           
uname          unicode_start  uniq           unix2mac       unix_update    unset          until          
[root@mcwk8s-node1 ~]$ umount /root/mcw/
[root@mcwk8s-node1 ~]$ df -h|grep pv1
10.0.0.4:/nfsdata/pv1     19G  3.9G   15G  22% /var/lib/kubelet/pods/d1ba5034-28ad-4898-a7aa-3551bc330a4b/volumes/kubernetes.io~nfs/mypv1
[root@mcwk8s-node1 ~]$ ls /var/lib/kubelet/pods/d1ba5034-28ad-4898-a7aa-3551bc330a4b/volumes/kubernetes.io~nfs/mypv1
mcw.txt  test.txt

回收PV

會刪除nfs數據的回收pv

[machangwei@mcwk8s-master ~]$ kubectl get pvc
NAME     STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
mypvc1   Bound    mypv1    1Gi        RWO            nfs            3h31m
[machangwei@mcwk8s-master ~]$ kubectl get pv
NAME    CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM            STORAGECLASS   REASON   AGE
mypv1   1Gi        RWO            Recycle          Bound    default/mypvc1   nfs                     3h33m
[machangwei@mcwk8s-master ~]$ kubectl delete pvc mypvc1 #刪除pvc失敗，pvc狀態也變成終止，而不是綁定
persistentvolumeclaim "mypvc1" deleted

^C
[machangwei@mcwk8s-master ~]$ kubectl delete pvc mypvc1
persistentvolumeclaim "mypvc1" deleted
^C
[machangwei@mcwk8s-master ~]$ 
[machangwei@mcwk8s-master ~]$ kubectl get pod  #刪除pod，再刪除pvc,pvc成功刪除，網上說是刪除pod，再刪pvc,pv,應該就是這個順序吧
NAME     READY   STATUS    RESTARTS   AGE
mypod1   1/1     Running   0          3h28m
[machangwei@mcwk8s-master ~]$ kubectl delete pod mypod1
pod "mypod1" deleted
[machangwei@mcwk8s-master ~]$ kubectl get pvc
No resources found in default namespace.
[machangwei@mcwk8s-master ~]$ kubectl get pod  #刪除pvc的時候創建了回收pv的容器
NAME                 READY   STATUS              RESTARTS   AGE
recycler-for-mypv1   0/1     ContainerCreating   0          33s
[machangwei@mcwk8s-master ~]$ kubectl get pv  #狀態現在是空閑，pv可以重新被其它pvc聲明綁定。在刪除pvc的過程中應該是被釋放的狀態，那時是不能被其它pvc綁定
NAME    CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
mypv1   1Gi        RWO            Recycle          Available           nfs                     3h41m
[machangwei@mcwk8s-master ~]$ 
[machangwei@mcwk8s-master ~]$ ls /nfsdata/pv1/  #刪除pvc後，pv雖然空閑了，但是pv對應的
[machangwei@mcwk8s-master ~]$ #nfs伺服器存儲數據都被刪除了。這是回收策略決定的，所以當我們要
[machangwei@mcwk8s-master ~]$ #刪除pvc時，一定要知道pv使用的什麼回收策略，防止本來不想刪除pv數據的，但是被刪掉的情況發生
[machangwei@mcwk8s-master ~]$ #刪除pvc，但是不刪除外部存儲nfs上的數據，得用Retain的回收策略

不會刪除nfs服務端數據的回收pv

[machangwei@mcwk8s-master ~]$ cat nfs-pvc1.yml 
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: mypvc1
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: nfs
[machangwei@mcwk8s-master ~]$ cat nfs-pv1.yml  #將pv的回收策略修改為Retain，這樣刪除pvc就不會刪除掉nfs服務端數據了
apiVersion: v1
kind: PersistentVolume
metadata:
  name: mypv1
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: nfs
  nfs:
    path: /nfsdata/pv1
    server: 10.0.0.4
[machangwei@mcwk8s-master ~]$ 
[machangwei@mcwk8s-master ~]$ 
[machangwei@mcwk8s-master ~]$ kubectl apply -f nfs-pv1.yml 
persistentvolume/mypv1 configured
[machangwei@mcwk8s-master ~]$ kubectl get pv  #查看可知，現在回收策略已經修改了
NAME    CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
mypv1   1Gi        RWO            Retain           Available           nfs                     3h56m
[machangwei@mcwk8s-master ~]$ kubectl apply -f nfs-pvc1.yml 

persistentvolumeclaim/mypvc1 created
[machangwei@mcwk8s-master ~]$ 
[machangwei@mcwk8s-master ~]$ kubectl get pvc
NAME     STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
mypvc1   Bound    mypv1    1Gi        RWO            nfs            10s
[machangwei@mcwk8s-master ~]$ cat pod1.yml 
kind: Pod
apiVersion: v1
metadata:
  name: mypod1
spec:
  containers:
    - name: mypod1
      image: busybox
      args:
      - /bin/sh
      - -c
      - sleep 30000
      volumeMounts:
      - mountPath: "/mydata"
        name: mydata
  volumes:
    - name: mydata
      persistentVolumeClaim:
        claimName: mypvc1
[machangwei@mcwk8s-master ~]$ kubectl apply -f pod1.yml #部署pod
pod/mypod1 created
[machangwei@mcwk8s-master ~]$ kubectl get pod -o wide  
NAME     READY   STATUS    RESTARTS   AGE     IP            NODE           NOMINATED NODE   READINESS GATES
mypod1   1/1     Running   0          3m16s   10.244.2.16   mcwk8s-node2   <none>           <none>
[machangwei@mcwk8s-master ~]$ kubectl exec mypod1 touch ls /mydata  
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
[machangwei@mcwk8s-master ~]$ ls /nfsdata/pv1/  #查看nfs服務端沒有數據
[machangwei@mcwk8s-master ~]$ kubectl exec mypod1 touch  /mydata/mcw.txt #執行容器名字，後接命令。這樣就相當於進入容器執行命令
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
[machangwei@mcwk8s-master ~]$ ls /nfsdata/pv1/  #容器中創建了文件，可以看到nfs服務端已經存在這個文件了
mcw.txt
[machangwei@mcwk8s-master ~]$ 
[machangwei@mcwk8s-master ~]$ kubectl get pvc
NAME     STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
mypvc1   Bound    mypv1    1Gi        RWO            nfs            7m10s
[machangwei@mcwk8s-master ~]$ kubectl delete pvc mypvc1 #此時刪除pvc，卡住很長時間不動


再開一個xshell會話，將使用pvc的pod刪除掉，
[machangwei@mcwk8s-master ~]$ kubectl get pv
NAME    CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM            STORAGECLASS   REASON   AGE
mypv1   1Gi        RWO            Retain           Bound    default/mypvc1   nfs                     4h5m
[machangwei@mcwk8s-master ~]$ kubectl get pod
NAME     READY   STATUS    RESTARTS   AGE
mypod1   1/1     Running   0          8m31s
[machangwei@mcwk8s-master ~]$ kubectl delete pod mypod1 #當刪除掉pod後，pvc也立即刪除了，pv不再是綁定狀態，暫時是被釋放的狀態
pod "mypod1" deleted
[machangwei@mcwk8s-master ~]$ 

pod刪除後,pvc也被刪除了，應該是需要刪除pod，才能刪除掉pvc吧，目前看是這個樣子。
[machangwei@mcwk8s-master ~]$ kubectl delete pvc mypvc1
persistentvolumeclaim "mypvc1" deleted
[machangwei@mcwk8s-master ~]$ kubectl get pv #刪除掉pvc後，發現狀態一直是被釋放的，只有狀態是空閑的，才能重新部署pvc使用這個pv，也就是才能被其它pvc申請
NAME    CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM            STORAGECLASS   REASON   AGE
mypv1   1Gi        RWO            Retain           Released   default/mypvc1   nfs                     4h7m
[machangwei@mcwk8s-master ~]$ ls /nfsdata/pv1/  #查看到即使刪除了pvc,但是外部存儲數據nfs服務端的數據還是存在的。
mcw.txt
[machangwei@mcwk8s-master ~]$

讓pv中的數據重新被使用

[machangwei@mcwk8s-master ~]$ #retain回收策略，被釋放的狀態，對應的pvc不存在了，
[machangwei@mcwk8s-master ~]$ kubectl get pv  #pv下面的數據還想被重新使用，
NAME    CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM            STORAGECLASS   REASON   AGE
mypv1   1Gi        RWO            Retain           Released   default/mypvc1   nfs                     4h16m
[machangwei@mcwk8s-master ~]$ kubectl get pvc
No resources found in default namespace.
[machangwei@mcwk8s-master ~]$ kubectl delete pv mypv1  #那麼需要刪除pv,

persistentvolume "mypv1" deleted
[machangwei@mcwk8s-master ~]$ 
[machangwei@mcwk8s-master ~]$ ls /nfsdata/pv1/  #這種情況的pv刪除是不會將外部數據刪除掉的
mcw.txt
[machangwei@mcwk8s-master ~]$ kubectl apply -f nfs-pv1.yml  #然後重新部署pv,pvc,pod
persistentvolume/mypv1 created
[machangwei@mcwk8s-master ~]$ kubectl get pv   #重新部署了pv，pv空閑可被pvc聲明使用
NAME    CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
mypv1   1Gi        RWO            Retain           Available           nfs                     12s
[machangwei@mcwk8s-master ~]$ kubectl apply -f nfs-pvc1.yml 
persistentvolumeclaim/mypvc1 created
[machangwei@mcwk8s-master ~]$ kubectl apply -f pod1.yml 
pod/mypod1 created
[machangwei@mcwk8s-master ~]$ kubectl get pod 
NAME     READY   STATUS    RESTARTS   AGE
mypod1   1/1     Running   0          4m26s
[machangwei@mcwk8s-master ~]$ kubectl exec mypod1 ls /mydata  #然後發現pod下容器重新使用這個數據了。也就是這種情況下數據是能保留下來的
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
mcw.txt
[machangwei@mcwk8s-master ~]$ ls /nfsdata/pv1/
mcw.txt
[machangwei@mcwk8s-master ~]$

PV動態供給

 AWS EBS 支援pv動態供給，以後補充

//kubernetes.io/docs/concepts/storage/storage-classes/#provisioner

mysql資料庫持久化邏輯卷使用案例(nfs)

查看部署配置文件

[machangwei@mcwk8s-master ~]$ cat mysql-pv.yml 
apiVersion: v1
kind: PersistentVolume
metadata:
  name: mysql-pv
spec:
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 1Gi
  persistentVolumeReclaimPolicy: Retain
  storageClassName: nfs
  nfs:
    path: /nfsdata/mysql-pv
    server: 10.0.0.4
[machangwei@mcwk8s-master ~]$ cat mysql-pvc.yml 
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: mysql-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: nfs
[machangwei@mcwk8s-master ~]$ cat mysql.yml 
apiVersion: v1
kind: Service
metadata:
  name: mysql
spec:
  ports:
  - port: 3306
  selector:
    app: mysql
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql
spec:
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - image: mysql:5.6
        name: mysql
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: password
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: mysql-persistent-storage
          mountPath: /var/lib/mysql
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: mysql-pvc   
[machangwei@mcwk8s-master ~]$

詳解配置文件

[machangwei@mcwk8s-master ~]$ cat mysql-pv.yml 
apiVersion: v1
kind: PersistentVolume
metadata:
  name: mysql-pv
spec:
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 1Gi
  persistentVolumeReclaimPolicy: Retain  #使用Reatin回收策略，即使刪除pv,pvc，數據也不清除
  storageClassName: nfs
  nfs:
    path: /nfsdata/mysql-pv
    server: 10.0.0.4
[machangwei@mcwk8s-master ~]$ cat mysql-pvc.yml 
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: mysql-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: nfs
[machangwei@mcwk8s-master ~]$ cat mysql.yml 
apiVersion: v1
kind: Service  #服務種類
metadata:
  name: mysql #元數據名稱是服務名稱
spec:  #查看規則
  ports:  #服務使用埠是什麼
  - port: 3306
  selector:  #使用哪個選擇器這裡是使用app mysql的選擇器。應該是創建app mysql 標籤，後面選擇器匹配的時候用把
    app: mysql
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql
spec:
  selector:  #選擇器匹配標籤帶有app mysql 的
    matchLabels:
      app: mysql
  template:  #查看模板
    metadata:  元數據標籤是app mysql 模板規則如下
      labels:
        app: mysql
    spec:
      containers:
      - image: mysql:5.6  #容器使用哪個鏡像
        name: mysql    #容器的名稱是什麼
        env:
        - name: MYSQL_ROOT_PASSWORD  #容器內環境變數咋樣，這裡定義了mysql用戶密碼
          value: password
        ports:
        - containerPort: 3306 #埠下，容器埠用的是哪個，並且是什麼名字
          name: mysql
        volumeMounts:
        - name: mysql-persistent-storage #容器邏輯卷掛載嗎，和容器同級。
          mountPath: /var/lib/mysql  #容器內部掛載目錄
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: mysql-pvc   
[machangwei@mcwk8s-master ~]$

部署pv，pvc ,service，deployment

[machangwei@mcwk8s-master ~]$ kubectl apply -f  mysql-pv.yml
persistentvolume/mysql-pv unchanged
[machangwei@mcwk8s-master ~]$ kubectl apply -f  mysql-pvc.yml
persistentvolumeclaim/mysql-pvc created
[machangwei@mcwk8s-master ~]$ kubectl get pv,pvc
NAME                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM               STORAGECLASS   REASON   AGE
persistentvolume/mysql-pv   1Gi        RWO            Retain           Bound    default/mysql-pvc   nfs                     4m22s

NAME                              STATUS   VOLUME     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/mysql-pvc   Bound    mysql-pv   1Gi        RWO            nfs            11s

[machangwei@mcwk8s-master ~]$ kubectl apply -f mysql.yml 
service/mysql unchanged
deployment.apps/mysql created
[machangwei@mcwk8s-master ~]$ kubectl get service
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
kubernetes   ClusterIP   10.96.0.1      <none>        443/TCP    29d
mysql        ClusterIP   10.97.19.121   <none>        3306/TCP   9m22s
[machangwei@mcwk8s-master ~]$ kubectl get pod
NAME                   READY   STATUS              RESTARTS   AGE
mysql-dbffc69d-trvvb   0/1     ContainerCreating   0          8m25s

查看是卡住了，mysq-pv目錄沒有，掛載上又卸載不下來了
  Normal   Scheduled    8m7s                            default-scheduler  Successfully assigned default/mysql-dbffc69d-trvvb to mcwk8s-node2
  Warning  FailedMount  <invalid> (x11 over <invalid>)  kubelet            MountVolume.SetUp failed for volume "mysql-pv" : mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t nfs 10.0.0.4:/nfsdata/mysql-pv /var/lib/kubelet/pods/6392d0d9-15be-4ba9-a338-61f290e6f999/volumes/kubernetes.io~nfs/mysql-pv
Output: mount.nfs: mounting 10.0.0.4:/nfsdata/mysql-pv failed, reason given by server: No such file or directory
  Warning  FailedMount  <invalid> (x3 over <invalid>)  kubelet  Unable to attach or mount volumes: unmounted volumes=[mysql-persistent-storage], unattached volumes=[mysql-persistent-storage kube-api-access-v6lfr]: timed out waiting for the condition
我刪除這個pod也卡在刪不掉，找到pod所在節點，然後把這個掛載上的目錄，手動執行命令umount卸載掉，然後就把這pod刪除了，隨後生成新的pod，運行起來了
[machangwei@mcwk8s-master ~]$ kubectl delete pod mysql-dbffc69d-trvvb
pod "mysql-dbffc69d-trvvb" deleted

後面重新部署

[machangwei@mcwk8s-master ~]$ ls
mysql-pvc.yml  mysql-pv.yml  mysql.yml
[machangwei@mcwk8s-master ~]$ kubectl apply -f mysql-pv.yml 
persistentvolume/mysql-pv created
[machangwei@mcwk8s-master ~]$ kubectl apply -f mysql-pvc.yml 
persistentvolumeclaim/mysql-pvc created
[machangwei@mcwk8s-master ~]$ kubectl apply -f mysql.yml 
service/mysql created
deployment.apps/mysql created

報錯
  ----     ------     ----                  ----               -------
  Normal   Scheduled  13m                   default-scheduler  Successfully assigned default/mysql-dbffc69d-292lp to mcwk8s-node2
  Normal   Pulling    13m                   kubelet            Pulling image "mysql:5.6"
  Normal   Pulled     10m                   kubelet            Successfully pulled image "mysql:5.6" in 2m17.834341538s
  Normal   Created    9m24s (x5 over 10m)   kubelet            Created container mysql
  Normal   Started    9m24s (x5 over 10m)   kubelet            Started container mysql
  Normal   Pulled     9m24s (x4 over 10m)   kubelet            Container image "mysql:5.6" already present on machine
  Warning  BackOff    3m13s (x37 over 10m)  kubelet            Back-off restarting failed container

  
  [root@mcwk8s-node2 ~]$ docker ps -a
CONTAINER ID   IMAGE          COMMAND    CREATED      STATUS                  PORTS     NAMES
75829eceaae7 dd3b2a5dcb48  "docker-entrypoint.s…" About a minute ago   Exited (1) About a minute ago  k8s_mysql_mysql-dbffc69d-292lp_default_e9825a68-47b0-471d-8280-b95bd51c4068_6

  去節點上查看MySQL容器日誌，報錯
  [root@mcwk8s-node2 ~]$ docker logs 758
2022-02-19 11:16:19+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.6.51-1debian9 started.
chown: changing ownership of '/var/lib/mysql/': Operation not permitted

解決方法，給nfs添加配置no_root_squash
[root@mcwk8s-master ~]$ cat /etc/exports
/nfsdata *(rw,sync)
[root@mcwk8s-master ~]$ vim /etc/exports
[root@mcwk8s-master ~]$ cat /etc/exports
/nfsdata *(rw,sync,no_root_squash)
[root@mcwk8s-master ~]$ systemctl restart nfs
[root@mcwk8s-master ~]$ showmount -e localhost
Export list for localhost:
/nfsdata *
[root@mcwk8s-master ~]$

進入MySQL並創建數據

[machangwei@mcwk8s-master ~]$ kubectl run -it  --rm --image=mysql:5.6 --restart=Never mysql-mcwclient -- mysql -h 10.103.171.207  -P3306 -ppassword
If you don't see a command prompt, try pressing enter.
mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
+--------------------+
3 rows in set (0.03 sec)

mysql> create database mcwtest;
Query OK, 1 row affected (0.08 sec)

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mcwtest            |
| mysql              |
| performance_schema |
+--------------------+
4 rows in set (0.03 sec)

mysql> use mcwtest;
Database changed
mysql> create table my_id(id int(4))
    -> ;
Query OK, 0 rows affected (0.25 sec)

mysql> insert my_i values(111);
ERROR 1146 (42S02): Table 'mcwtest.my_i' doesn't exist
mysql> insert my_id values(111);
Query OK, 1 row affected (0.05 sec)

mysql> select * from my_id;
+------+
| id   |
+------+
|  111 |
+------+
1 row in set (0.00 sec)

mysql> \q
Bye
pod "mysql-mcwclient" deleted
[machangwei@mcwk8s-master ~]$ kubectl get pod -o wide
NAME                   READY   STATUS    RESTARTS   AGE   IP           NODE           NOMINATED NODE   READINESS GATES
mysql-dbffc69d-jhvvz   1/1     Running   0          27m   10.244.1.4   mcwk8s-node1   <none>           <none>
[machangwei@mcwk8s-master ~]$

關閉node1模擬故障

[root@mcwk8s-node1 ~]$ shutdown now

Connection closed by foreign host.

Disconnected from remote host(mcw05) at 20:19:00.

Type `help' to learn how to use Xshell prompt.
[c:\~]$

驗證服務故障轉移

 

[machangwei@mcwk8s-master ~]$ kubectl get pod -o wide
NAME                   READY   STATUS    RESTARTS   AGE   IP           NODE           NOMINATED NODE   READINESS GATES
mysql-dbffc69d-jhvvz   1/1     Running   0          38m   10.244.1.4   mcwk8s-node1   <none>           <none>
[machangwei@mcwk8s-master ~]$ kubectl describe pod mysql-dbffc69d-jhvvz
.....
Warning  NodeNotReady            111s               node-controller    Node is not ready

  Warning  NodeNotReady            5m29s              node-controller    Node is not ready
[machangwei@mcwk8s-master ~]$ 當發現節點5分鐘左右沒準備好的時候，就開始運行新的pod,並部署mysql服務的pod在可用節點上。
[machangwei@mcwk8s-master ~]$ kubectl get pod -o wide
NAME                   READY   STATUS        RESTARTS   AGE    IP            NODE           NOMINATED NODE   READINESS GATES
mysql-dbffc69d-ch2vg   1/1     Running       0          2m9s   10.244.2.18   mcwk8s-node2   <none>           <none>
mysql-dbffc69d-jhvvz   1/1     Terminating   0          43m    10.244.1.4    mcwk8s-node1   <none>           <none>
[machangwei@mcwk8s-master ~]$ #還是用之前的命令來連接。--後面應該是連接命令吧。-h指定集群服務ip,指定埠和密碼
[machangwei@mcwk8s-master ~]$ kubectl get service  #--rm是退出後就刪的pod.
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP    29d
mysql        ClusterIP   10.103.171.207   <none>        3306/TCP   79m
[machangwei@mcwk8s-master ~]$ kubectl run -it  --rm --image=mysql:5.6 --restart=Never mysql-mcwclient -- mysql -h 10.103.171.207  -P3306 -ppassword
If you don't see a command prompt, try pressing enter. #當我們要進入MySQL命令行，需要按enter。不用一直等待，因為等著沒用

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mcwtest            |
| mysql              |
| performance_schema |
+--------------------+
4 rows in set (0.02 sec)

mysql> use mcwtest;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> show tables;
+-------------------+
| Tables_in_mcwtest |
+-------------------+
| my_id             |
+-------------------+
1 row in set (0.01 sec)

mysql> select * from my_id;
+------+
| id   |
+------+
|  111 |
+------+
1 row in set (0.02 sec)

mysql> \q
Bye
pod "mysql-mcwclient" deleted

到root下查看資料庫的數據存放地址，也就是nfs服務端上。我們設置了即使刪除pv,pvc也是不刪除數據的。如上可知，當資料庫所在主機down之後，那麼過一段時間，會重新從其它節點上運行新的pod,舊的pod還存在，只是狀態是終止。
[root@mcwk8s-master ~]$
[root@mcwk8s-master ~]$ ls /nfsdata/mysql-pv/
auto.cnf  ibdata1  ib_logfile0  ib_logfile1  mcwtest  mysql  performance_schema
[root@mcwk8s-master ~]$ ls /nfsdata/mysql-pv/mcwtest/
db.opt  my_id.frm  my_id.ibd
[root@mcwk8s-master ~]$ ls /nfsdata/mysql-pv/mcwtest/ -lh
total 112K
-rw-rw----. 1 polkitd ssh_keys   65 Feb 19 20:08 db.opt
-rw-rw----. 1 polkitd ssh_keys 8.4K Feb 19 20:09 my_id.frm
-rw-rw----. 1 polkitd ssh_keys  96K Feb 19 20:09 my_id.ibd
[root@mcwk8s-master ~]$ ls /nfsdata/mysql-pv/ -lh
total 109M
-rw-rw----. 1 polkitd ssh_keys   56 Feb 19 19:23 auto.cnf
-rw-rw----. 1 polkitd ssh_keys  12M Feb 19 20:24 ibdata1
-rw-rw----. 1 polkitd ssh_keys  48M Feb 19 20:24 ib_logfile0
-rw-rw----. 1 polkitd ssh_keys  48M Feb 19 19:21 ib_logfile1
drwx------. 2 polkitd ssh_keys   54 Feb 19 20:09 mcwtest
drwx------. 2 polkitd ssh_keys 4.0K Feb 19 19:25 mysql
drwx------. 2 polkitd ssh_keys 4.0K Feb 19 19:21 performance_schema

 

 

參考學習書籍：每天5分鐘玩轉kubernetes  —cloudman

 

Tags:

VirMach 便宜 VPS

QNews

QNews