Operator介紹

2022 年 6 月 26 日
筆記
Kubernetes, Operator

一、Operator簡介

在Kubernetes中我們經常使用Deployment、DaemonSet、Service、ConfigMap等資源，這些資源都是Kubernetes的內置資源，他們的創建、更新、刪除等均有Controller Manager負責管理。

二、Operator組成

Operator(Controller+CRD），Operator是由Kubernetes自定義資源（CRD）和控制器（Controller）構成的元原生擴展服務，其中CRD定義了每個Operator需要創建和管理的自定義資源對象，底層實際就是通過APIServer介面在ETCD中註冊一種新的資源類型，註冊完成後就可以創建該資源類型的對象了。但是僅註冊資源和創建資源對象是沒有任何實際意義的，CRD最重要的是需要配合對應的Controller來實現自定義資源的功能達到自定義資源期望的狀態，比如內置的Deployment Controller用來控制Department資源對象的功能，根據配置生成特定數量的Pod監控其狀態，並根據事件做出相應的動作。

三、Operator使用

用戶想為自己的自定義資源構建一個Kubernetes Operator，有很多工具可供選擇比如Operator SDK、Kubebuilder，甚至可以使用Operator SDK（HELM、Ansible、Go）。這些工具創建Kubernetes Operator用來監控自定義資源，並且根據資源的變化調整資源狀態。

Operator作為自定義擴展資源以Deployment的方式部署到Kubernetes中，通過List-Watch方式監聽對應資源的變化，當用戶修改自定義資源中的任何內容，Operator會監控資源的更改，並根據更改內容執行特定的操作，這些操作通常會對Kubernetes API中某些資源進行調用。

四、Operator應用案例

Kubebuilder介紹

Kubebuilder是一個用Go語言構建Kubenetes API控制器和CRD的腳手架工具，通過使用Kubebuilder,用戶可以遵循一套簡單的編程框架，編寫Operator使用實例。

kubenetes: v1.23.8
go

tar xf go1.18.3.linux-amd64.tar.gz
mv go /usr/local/
vim /etc/profile.d/go183.sh
export GO111MODULE=on
export GOROOT=/usr/local/go 
export GOPATH=/root/gopath
export PATH=$PATH:$GOROOT/bin:$GOPATH/bin

source /etc/profile.d/go183.sh

kubebuilder: 3.5.0

yum -y install gcc
curl -L -o kubebuilder //go.kubebuilder.io/dl/latest/$(go env GOOS)/$(go env GOARCH)
chmod +x kubebuilder && mv kubebuilder /usr/local/bin/

kubebuilder version

案例

　　Welcome案例主要實現使用Operator和CRD部署一套完整的應用環境，可以實現根據自定義類型創建資源，通過創建一個Welcome類型的資源，後台自動創建Deployment和Service，通過Web頁面訪問Service呈現應用部署，通過自定義控制器方式進行控制管理

　　我們西藥創建Welcome自定義資源及對應的Controllers，最終我們可以通過類似如下程式碼的YAML文件部署簡單的Web應用

apiVersion: webapp.demo.welcome.domain/v1
kind: Welcome
metadata:
  name: welcome-sample
spec:
  name: myfriends

Web應用介紹

本案例中，我們使用Go語言HTTP模組創建一個Web服務，用戶訪問頁面後會自動載入NAME及PORT環境變數並渲染index.html靜態文件中，程式碼如下：

package main

import (
	"fmt"
	"net/http"
	"os"
)

func main() {
	name := os.Getenv("NAME")
	hello := fmt.Sprintf("Hello %s", name)
	http.Handle("/hello/", http.StripPrefix("/hello/", http.FileServer(http.Dir("static"))))
	f, err := os.OpenFile("./static/index.html", os.O_APPEND|os.O_WRONLY|os.O_CREATE, 0600)
	if err != nil {
		panic(err)
	}
	defer f.Close()
	if _, err = f.WriteString(hello); err != nil {
		panic(err)
	}
	port := os.Getenv("PORT")
	if port == "" {
		port = "8080"
	}
}

其中，NAME環境變數通過我們在Welcome中定義的name欄位獲取，我們在下面的控制器編寫中會詳細介紹獲取欄位的詳細方法。我們將index.html放在Static文件夾下，並將工程文件打包為Docker鏡像，Dockerfile如下：

FROM golang:1.17 as builder

WORKDIR /
COPY . .
COPY static

RUN CGO_ENABLED=0 GOOS=linux go build -v -o main
FROM alpine
RUN apk add --no-cache ca-certificates
COPY --from=builder /main /usr/local/main
COPY --from=builder static /static
CMD ["/usr/local/main"]

項目初始化

使用Kubebuilder命令鏡像項目初始化

mkdir demo
cd demo
go mod init welcome_demo.domain
kubebuilder init --domain demo.welcome.domain

初始化項目後，kubebuilder生成項目介面如下

[root@k8s-01 demo]# tree .
.
├── config
│   ├── default
│   │   ├── kustomization.yaml
│   │   ├── manager_auth_proxy_patch.yaml
│   │   └── manager_config_patch.yaml
│   ├── manager
│   │   ├── controller_manager_config.yaml
│   │   ├── kustomization.yaml
│   │   └── manager.yaml
│   ├── prometheus
│   │   ├── kustomization.yaml
│   │   └── monitor.yaml
│   └── rbac
│       ├── auth_proxy_client_clusterrole.yaml
│       ├── auth_proxy_role_binding.yaml
│       ├── auth_proxy_role.yaml
│       ├── auth_proxy_service.yaml
│       ├── kustomization.yaml
│       ├── leader_election_role_binding.yaml
│       ├── leader_election_role.yaml
│       ├── role_binding.yaml
│       └── service_account.yaml
├── Dockerfile
├── go.mod
├── go.sum
├── hack
│   └── boilerplate.go.txt
├── main.go
├── Makefile
├── PROJECT
└── README.md

6 directories, 25 files

創建`Welcome` Kind和其對應的控制器

[root@k8s-01 demo]# kubebuilder create api --group webapp --kind Welcome --version v1
Create Resource [y/n]
y
Create Controller [y/n]
y

輸入兩次y, Kubebuilder 分別創建了資源和控制器的模板，此處的group、version、kind這3個屬性組合起來標識一個k8s的CRD，創建完成後，Kubebuilder添加文件如下：

[root@k8s-01 demo]# tree .
.
├── api
│   └── v1
│       ├── groupversion_info.go
│       ├── welcome_types.go
│       └── zz_generated.deepcopy.go
├── bin
│   └── controller-gen
├── config
│   ├── crd
│   │   ├── kustomization.yaml
│   │   ├── kustomizeconfig.yaml
│   │   └── patches
│   │       ├── cainjection_in_welcomes.yaml
│   │       └── webhook_in_welcomes.yaml
│   ├── default
│   │   ├── kustomization.yaml
│   │   ├── manager_auth_proxy_patch.yaml
│   │   └── manager_config_patch.yaml
│   ├── manager
│   │   ├── controller_manager_config.yaml
│   │   ├── kustomization.yaml
│   │   └── manager.yaml
│   ├── prometheus
│   │   ├── kustomization.yaml
│   │   └── monitor.yaml
│   ├── rbac
│   │   ├── auth_proxy_client_clusterrole.yaml
│   │   ├── auth_proxy_role_binding.yaml
│   │   ├── auth_proxy_role.yaml
│   │   ├── auth_proxy_service.yaml
│   │   ├── kustomization.yaml
│   │   ├── leader_election_role_binding.yaml
│   │   ├── leader_election_role.yaml
│   │   ├── role_binding.yaml
│   │   ├── service_account.yaml
│   │   ├── welcome_editor_role.yaml
│   │   └── welcome_viewer_role.yaml
│   └── samples
│       └── webapp_v1_welcome.yaml
├── controllers
│   ├── suite_test.go
│   └── welcome_controller.go
├── Dockerfile
├── go.mod
├── go.sum
├── hack
│   └── boilerplate.go.txt
├── main.go
├── Makefile
├── PROJECT
└── README.md

13 directories, 38 files

後續需要執行兩步操作：

修改Resource Type
修改Controller 邏輯

修改Resource Type

此處Resource Type為需要定義的資源欄位，用於在Yaml文件中進行聲明，本案例中需要新增name欄位用於「Welcome」Kind中的Web應用，程式碼如下：

vim api/v1/welcome_types.go

// WelcomeSpec defines the desired state of Welcome
type WelcomeSpec struct {
        // INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
        // Important: Run "make" to regenerate code after modifying this file

        // Foo is an example field of Welcome. Edit welcome_types.go to remove/update
        // Foo string `json:"foo,omitempty"`
        Name string `json:"name,omitempty"`
}

修改Controller邏輯

在Controller中需要通過Reconcile方法完成Deployment和Service部署，並最終達到期望的狀態。

vim controllers/welcome_controller.go

//+kubebuilder:rbac:groups=webapp.demo.welcome.domain,resources=welcomes,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=webapp.demo.welcome.domain,resources=welcomes/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=webapp.demo.welcome.domain,resources=welcomes/finalizers,verbs=update

// Reconcile is part of the main kubernetes reconciliation loop which aims to
// move the current state of the cluster closer to the desired state.
// TODO(user): Modify the Reconcile function to compare the state specified by
// the Welcome object against the actual cluster state, and then
// perform operations to make the cluster state reflect the state specified by
// the user.
//
// For more details, check Reconcile and its Result here:
// - //pkg.go.dev/sigs.k8s.io/[email protected]/pkg/reconcile
func (r *WelcomeReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
        // _ = log.FromContext(ctx)

        // TODO(user): your logic here
        log := r.Log.WithValues("welcome", req.NamespacedName)
        log.Info("reconcilling welcome")
        return ctrl.Result{}, nil
}

最終程式碼如下

/*
Copyright 2022.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    //www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package controllers

import (
	"context"
	"fmt"
	appsv1 "k8s.io/api/apps/v1"
	"k8s.io/apimachinery/pkg/api/resource"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/util/intstr"
	"sigs.k8s.io/controller-runtime/pkg/log"

	corev1 "k8s.io/api/core/v1"
	"k8s.io/apimachinery/pkg/runtime"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/client"
	webappv1 "welcome_demo.domain/api/v1"
)

// WelcomeReconciler reconciles a Welcome object
type WelcomeReconciler struct {
	client.Client
	Scheme *runtime.Scheme
}

//+kubebuilder:rbac:groups=webapp.demo.welcome.domain,resources=welcomes,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=webapp.demo.welcome.domain,resources=welcomes/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=webapp.demo.welcome.domain,resources=welcomes/finalizers,verbs=update

// Reconcile is part of the main kubernetes reconciliation loop which aims to
// move the current state of the cluster closer to the desired state.
// TODO(user): Modify the Reconcile function to compare the state specified by
// the Welcome object against the actual cluster state, and then
// perform operations to make the cluster state reflect the state specified by
// the user.
//
// For more details, check Reconcile and its Result here:
// - //pkg.go.dev/sigs.k8s.io/[email protected]/pkg/reconcile
func (r *WelcomeReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	_ = log.FromContext(ctx)

	// TODO(user): your logic here
	welcome := &webappv1.Welcome{}
	if err := r.Client.Get(ctx, req.NamespacedName, welcome); err != nil {
		return ctrl.Result{}, err
	}
	deployment, err := r.createWelcomeDeployment(welcome)
	if err != nil {
		return ctrl.Result{}, err
	}
	fmt.Println("create deployment success!")
	svc, err := r.createService(welcome)
	if err != nil {
		return ctrl.Result{}, err
	}
	fmt.Println("create service success!")
	applyOpts := []client.PatchOption{client.ForceOwnership, client.FieldOwner("welcome_controller")}
	err = r.Patch(ctx, &deployment, client.Apply, applyOpts...)
	if err != nil {
		return ctrl.Result{}, err
	}
	err = r.Patch(ctx, &svc, client.Apply, applyOpts...)
	if err != nil {
		return ctrl.Result{}, err
	}
	return ctrl.Result{}, nil
}

func (r *WelcomeReconciler) createWelcomeDeployment(welcome *webappv1.Welcome) (appsv1.Deployment, error) {
	defOne := int32(1)
	name := welcome.Spec.Name
	if name == "" {
		name = "world"
	}
	depl := appsv1.Deployment{
		TypeMeta: metav1.TypeMeta{
			APIVersion: appsv1.SchemeGroupVersion.String(),
			Kind:       "Deployment",
		},
		ObjectMeta: metav1.ObjectMeta{
			Name:      welcome.Name,
			Namespace: welcome.Namespace,
		},
		Spec: appsv1.DeploymentSpec{
			Replicas: &defOne,
			Selector: &metav1.LabelSelector{
				MatchLabels: map[string]string{"welcome": welcome.Name},
			},
			Template: corev1.PodTemplateSpec{
				ObjectMeta: metav1.ObjectMeta{
					Labels: map[string]string{"welcome": welcome.Name},
				},
				Spec: corev1.PodSpec{
					Containers: []corev1.Container{
						{
							Name: "welcome",
							Env: []corev1.EnvVar{
								{Name: "NAME", Value: name},
							},
							Ports: []corev1.ContainerPort{
								{ContainerPort: 8080,
									Name:     "http",
									Protocol: "TCP",
								},
							},
							Image: "sdfcdwefe/operatordemo:v1",
							Resources: corev1.ResourceRequirements{
								Requests: corev1.ResourceList{
									corev1.ResourceCPU:    *resource.NewMilliQuantity(100, resource.DecimalSI),
									corev1.ResourceMemory: *resource.NewMilliQuantity(100000, resource.BinarySI),
								},
							},
						},
					},
				},
			},
		},
	}

	return depl, nil
}

func (r *WelcomeReconciler) createService(welcome *webappv1.Welcome) (corev1.Service, error) {
	svc := corev1.Service{
		TypeMeta: metav1.TypeMeta{
			APIVersion: corev1.SchemeGroupVersion.String(),
			Kind:       "Service"},
		ObjectMeta: metav1.ObjectMeta{
			Name:      welcome.Name,
			Namespace: welcome.Namespace,
		},
		Spec: corev1.ServiceSpec{
			Ports: []corev1.ServicePort{
				{Name: "http",
					Port:       8080,
					Protocol:   "TCP",
					TargetPort: intstr.FromString("http")},
			},
			Selector: map[string]string{"welcome": welcome.Name},
			Type:     corev1.ServiceTypeLoadBalancer,
		},
	}
	return svc, nil
}

// SetupWithManager sets up the controller with the Manager.
func (r *WelcomeReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&webappv1.Welcome{}).
		Complete(r)
}

Welcome應部署

生成CRD資源

[root@k8s-01 demo]# make manifests
/root/demo/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases

[root@k8s-01 demo]# tree .
.
├── api
│   └── v1
│       ├── groupversion_info.go
│       ├── welcome_types.go
│       └── zz_generated.deepcopy.go
├── bin
│   └── controller-gen
├── config
│   ├── crd
│   │   ├── bases
│   │   │   └── webapp.demo.welcome.domain_welcomes.yaml
│   │   ├── kustomization.yaml
│   │   ├── kustomizeconfig.yaml
│   │   └── patches
│   │       ├── cainjection_in_welcomes.yaml
│   │       └── webhook_in_welcomes.yaml
│   ├── default
│   │   ├── kustomization.yaml
│   │   ├── manager_auth_proxy_patch.yaml
│   │   └── manager_config_patch.yaml
│   ├── manager
│   │   ├── controller_manager_config.yaml
│   │   ├── kustomization.yaml
│   │   └── manager.yaml
│   ├── prometheus
│   │   ├── kustomization.yaml
│   │   └── monitor.yaml
│   ├── rbac
│   │   ├── auth_proxy_client_clusterrole.yaml
│   │   ├── auth_proxy_role_binding.yaml
│   │   ├── auth_proxy_role.yaml
│   │   ├── auth_proxy_service.yaml
│   │   ├── kustomization.yaml
│   │   ├── leader_election_role_binding.yaml
│   │   ├── leader_election_role.yaml
│   │   ├── role_binding.yaml
│   │   ├── role.yaml
│   │   ├── service_account.yaml
│   │   ├── welcome_editor_role.yaml
│   │   └── welcome_viewer_role.yaml
│   └── samples
│       └── webapp_v1_welcome.yaml
├── controllers
│   ├── suite_test.go
│   └── welcome_controller.go
├── Dockerfile
├── go.mod
├── go.sum
├── hack
│   └── boilerplate.go.txt
├── main.go
├── Makefile
├── PROJECT
└── README.md

14 directories, 40 files


[root@k8s-01 demo]# cat config/crd/bases/webapp.demo.welcome.domain_welcomes.yaml 
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.9.0
  creationTimestamp: null
  name: welcomes.webapp.demo.welcome.domain
spec:
  group: webapp.demo.welcome.domain
  names:
    kind: Welcome
    listKind: WelcomeList
    plural: welcomes
    singular: welcome
  scope: Namespaced
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        description: Welcome is the Schema for the welcomes API
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: //git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: //git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: WelcomeSpec defines the desired state of Welcome
            properties:
              name:
                description: Foo is an example field of Welcome. Edit welcome_types.go
                  to remove/update Foo string `json:"foo,omitempty"`
                type: string
            type: object
          status:
            description: WelcomeStatus defines the observed state of Welcome
            type: object
        type: object
    served: true
    storage: true
    subresources:
      status: {}

創建Welcome類型資源

[root@k8s-01 demo]# kubectl create -f config/crd/bases/
customresourcedefinition.apiextensions.k8s.io/welcomes.webapp.demo.welcome.domain created
[root@k8s-01 demo]# kubectl create -f config/samples/webapp_v1_welcome.yaml 
welcome.webapp.demo.welcome.domain/welcome-sample created

使用`kubectl get crd`命令查看自定義對象

[root@k8s-01 demo]# kubectl get crd | grep welcome
welcomes.webapp.demo.welcome.domain          2022-06-25T09:10:37Z

通過kubectl get welcome命令可以看到創建的welcome對象

[root@k8s-01 demo]# kubectl get welcome
NAME             AGE
welcome-sample   2m20s

此時CRD並不會完成任何工作，只是在ETCD中創建了一條記錄，我們需要運行Controller才能幫助我們完成工作並最終達到welcome定義的狀態。

[root@k8s-01 demo]# make run
/root/demo/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
/root/demo/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
api/v1/welcome_types.go
go vet ./...
go run ./main.go
1.6561485987622015e+09	INFO	controller-runtime.metrics	Metrics server is starting to listen	{"addr": ":8080"}
1.6561485987624757e+09	INFO	setup	starting manager
1.6561485987638762e+09	INFO	Starting server	{"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
1.656148598763948e+09	INFO	Starting server	{"kind": "health probe", "addr": "[::]:8081"}
1.656148598764167e+09	INFO	Starting EventSource	{"controller": "welcome", "controllerGroup": "webapp.demo.welcome.domain", "controllerKind": "Welcome", "source": "kind source: *v1.Welcome"}
1.6561485987641926e+09	INFO	Starting Controller	{"controller": "welcome", "controllerGroup": "webapp.demo.welcome.domain", "controllerKind": "Welcome"}
1.6561485988653958e+09	INFO	Starting workers	{"controller": "welcome", "controllerGroup": "webapp.demo.welcome.domain", "controllerKind": "Welcome", "worker count": 1}
create deployment success!
create service success!

以上方式在本地啟動控制器，方便調試和驗證

查看創建的deployment、service

[root@k8s-01 demo]# kubectl get deploy
NAME             READY   UP-TO-DATE   AVAILABLE   AGE
welcome-sample   1/1     1            1           34s
[root@k8s-01 demo]# kubectl get svc
NAME             TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
kubernetes       ClusterIP      10.96.0.1       <none>        443/TCP          4d2h
welcome-sample   LoadBalancer   10.106.36.129   <pending>     8080:30037/TCP   39s

Tags: Kubernetes Operator

Operator介紹

一、Operator簡介

二、Operator組成

三、Operator使用

四、Operator應用案例

Kubebuilder介紹

案例

Web應用介紹

項目初始化

修改Resource Type

修改Controller邏輯

Welcome應部署

VirMach 便宜 VPS

QNews

Operator介紹

一、Operator簡介

二、Operator組成

三、Operator使用

四、Operator應用案例

Kubebuilder介紹

案例

Web應用介紹

項目初始化

修改Resource Type

修改Controller邏輯

Welcome應部署

分享此文：

Related Posts

Redis中是如何實現分散式鎖的？

細談設計稿還原

4.48億！廣東彩票史上第一大獎誕生 僅次于山東4.97億

02 CSS塊級元素和行內元素

VirMach 便宜 VPS

QNews

熱門搜尋

4.48億！廣東彩票史上第一大獎誕生僅次于山東4.97億