kubebuilder實戰之二：初次體驗kubebuilder

2021 年 8 月 26 日
筆記

歡迎訪問我的GitHub

//github.com/zq2599/blog_demos

內容：所有原創文章分類匯總及配套源碼，涉及Java、Docker、Kubernetes、DevOPS等；

系列文章鏈接

本篇概覽

本文是《kubebuilder實戰》系列的第二篇，前文將kubebuilder環境準備完畢，今天咱們在此環境創建CRD和Controller，再部署到kubernetes環境並且驗證是否生效，整篇文章由以下內容組成：

創建API(CRD和Controller)
構建和部署CRD
編譯和運行controller
創建CRD對應的實例
刪除實例並停止controller
將controller製作成docker鏡像
卸載和清理

創建helloworld項目

執行以下命令，創建helloworld項目：

mkdir -p $GOPATH/src/helloworld
cd $GOPATH/src/helloworld
kubebuilder init --domain com.bolingcavalry

控制台輸出類似以下內容：

[root@kubebuilder helloworld]# kubebuilder init --domain com.bolingcavalry
Writing scaffold for you to edit...
Get controller runtime:
$ go get sigs.k8s.io/[email protected]
Update go.mod:
$ go mod tidy
Running make:
$ make
/root/gopath/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
go build -o bin/manager main.go
Next: define a resource with:
$ kubebuilder create api

等待數分鐘後創建完成，在$GOPATH/src/helloworld目錄下新增以下內容，可見這是個標準的go module工程：

[root@kubebuilder ~]# tree $GOPATH/src/helloworld
/root/gopath/src/helloworld
├── bin
│   └── manager
├── config
│   ├── certmanager
│   │   ├── certificate.yaml
│   │   ├── kustomization.yaml
│   │   └── kustomizeconfig.yaml
│   ├── default
│   │   ├── kustomization.yaml
│   │   ├── manager_auth_proxy_patch.yaml
│   │   ├── manager_webhook_patch.yaml
│   │   └── webhookcainjection_patch.yaml
│   ├── manager
│   │   ├── kustomization.yaml
│   │   └── manager.yaml
│   ├── prometheus
│   │   ├── kustomization.yaml
│   │   └── monitor.yaml
│   ├── rbac
│   │   ├── auth_proxy_client_clusterrole.yaml
│   │   ├── auth_proxy_role_binding.yaml
│   │   ├── auth_proxy_role.yaml
│   │   ├── auth_proxy_service.yaml
│   │   ├── kustomization.yaml
│   │   ├── leader_election_role_binding.yaml
│   │   ├── leader_election_role.yaml
│   │   └── role_binding.yaml
│   └── webhook
│       ├── kustomization.yaml
│       ├── kustomizeconfig.yaml
│       └── service.yaml
├── Dockerfile
├── go.mod
├── go.sum
├── hack
│   └── boilerplate.go.txt
├── main.go
├── Makefile
└── PROJECT

9 directories, 30 files

創建API(CRD和Controller)

接下來要要創建資源相關的內容了，group/version/kind這三部分可以確定資源的唯一身份，命令如下：

cd $GOPATH/src/helloworld
kubebuilder create api \
--group webapp \
--version v1 \
--kind Guestbook

控制台會提醒是否創建資源(Create Resource [y/n])，輸入y
接下來控制台會提醒是否創建控制器(Create Controller [y/n])，輸入y
kubebuilder會根據上述命令新增多個文件，如下圖紅框所示：

構建和部署CRD

kubebuilder提供的Makefile將構建和部署工作大幅度簡化，執行以下命令會將最新構建的CRD部署在kubernetes上：

cd $GOPATH/src/helloworld
make install

控制台輸出如下內容，提示部署成功：

[root@kubebuilder helloworld]# make install
/root/gopath/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
kustomize build config/crd | kubectl apply -f -
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
customresourcedefinition.apiextensions.k8s.io/guestbooks.webapp.com.bolingcavalry created

編譯和運行controller

kubebuilder自動生成的controller源碼地址是：$GOPATH/src/helloworld/controllers/guestbook_controller.go ，內容如下：

package controllers

import (
	"context"

	"github.com/go-logr/logr"
	"k8s.io/apimachinery/pkg/runtime"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/client"

	webappv1 "helloworld/api/v1"
)

// GuestbookReconciler reconciles a Guestbook object
type GuestbookReconciler struct {
	client.Client
	Log    logr.Logger
	Scheme *runtime.Scheme
}

// +kubebuilder:rbac:groups=webapp.com.bolingcavalry,resources=guestbooks,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=webapp.com.bolingcavalry,resources=guestbooks/status,verbs=get;update;patch

func (r *GuestbookReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
	_ = context.Background()
	_ = r.Log.WithValues("guestbook", req.NamespacedName)

	// your logic here

	return ctrl.Result{}, nil
}

func (r *GuestbookReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&webappv1.Guestbook{}).
		Complete(r)
}

本文以體驗基本流程為主，不深入研究源碼，所以對上面的代碼僅做少量修改，用於驗證是否能生效，改動如下圖紅框所示：

執行以下命令，會編譯並啟動剛才修改的controller：

cd $GOPATH/src/helloworld
make run

此時控制台輸出以下內容，這裡要注意，controller是在kubebuilder電腦上運行的，一旦使用Ctrl+c中斷控制台，就會導致controller停止：

[root@kubebuilder helloworld]# make run
/root/gopath/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
/root/gopath/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
go run ./main.go
2021-01-23T20:58:35.107+0800	INFO	controller-runtime.metrics	metrics server is starting to listen	{"addr": ":8080"}
2021-01-23T20:58:35.108+0800	INFO	setup	starting manager
2021-01-23T20:58:35.108+0800	INFO	controller-runtime.manager	starting metrics server	{"path": "/metrics"}
2021-01-23T20:58:35.108+0800	INFO	controller-runtime.controller	Starting EventSource	{"controller": "guestbook", "source": "kind source: /, Kind="}
2021-01-23T20:58:35.208+0800	INFO	controller-runtime.controller	Starting Controller	{"controller": "guestbook"}
2021-01-23T20:58:35.209+0800	INFO	controller-runtime.controller	Starting workers	{"controller": "guestbook", "worker count": 1}

創建Guestbook資源的實例

現在kubernetes已經部署了Guestbook類型的CRD，而且對應的controller也已正在運行中，可以嘗試創建Guestbook類型的實例了(相當於有了pod的定義後，才可以創建pod)；
kubebuilder已經自動創建了一個類型的部署文件：$GOPATH/src/helloworld/config/samples/webapp_v1_guestbook.yaml ，內容如下，很簡單，接下來咱們就用這個文件來創建Guestbook實例：

apiVersion: webapp.com.bolingcavalry/v1
kind: Guestbook
metadata:
  name: guestbook-sample
spec:
  # Add fields here
  foo: bar

重新打開一個控制台，登錄kubebuilder電腦，執行以下命令即可創建Guestbook類型的實例：

cd $GOPATH/src/helloworld
kubectl apply -f config/samples/

如下所示，控制台提示資源創建成功：

[root@kubebuilder helloworld]# kubectl apply -f config/samples/
guestbook.webapp.com.bolingcavalry/guestbook-sample created

用kubectl get命令可以看到實例已經創建：

[root@kubebuilder helloworld]# kubectl get Guestbook
NAME               AGE
guestbook-sample   112s

用命令kubectl edit Guestbook guestbook-sample編輯該實例，修改的內容如下圖紅框所示：

7. 此時去controller所在控制台，可以看到新增和修改的操作都有日誌輸出，咱們新增的日誌都在裏面，代碼調用棧一目了然：

2021-01-24T09:51:50.418+0800	INFO	controllers.Guestbook	1. default/guestbook-sample
2021-01-24T09:51:50.418+0800	INFO	controllers.Guestbook	2. goroutine 188 [running]:
runtime/debug.Stack(0xc0002a1808, 0xc0002fc600, 0x1b)
	/root/go/src/runtime/debug/stack.go:24 +0x9f
helloworld/controllers.(*GuestbookReconciler).Reconcile(0xc0003c9dd0, 0xc0002d02f9, 0x7, 0xc0002d02e0, 0x10, 0x12f449647b, 0xc000456f30, 0xc000456ea8, 0xc000456ea0)
	/root/gopath/src/helloworld/controllers/guestbook_controller.go:49 +0x1a9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc00022a480, 0x1430e00, 0xc0003e7560, 0x0)
	/root/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:256 +0x166
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00022a480, 0xc000469600)
	/root/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:232 +0xb0
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker(0xc00022a480)
	/root/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:211 +0x2b
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc000292980)
	/root/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:152 +0x5f
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000292980, 0x3b9aca00, 0x0, 0x1609101, 0xc000102480)
	/root/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:153 +0x105
k8s.io/apimachinery/pkg/util/wait.Until(0xc000292980, 0x3b9aca00, 0xc000102480)
	/root/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88 +0x4d
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
	/root/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:193 +0x32d

2021-01-24T09:51:50.418+0800	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "guestbook", "request": "default/guestbook-sample"}


2021-01-24T09:52:33.632+0800	INFO	controllers.Guestbook	1. default/guestbook-sample
2021-01-24T09:52:33.633+0800	INFO	controllers.Guestbook	2. goroutine 188 [running]:
runtime/debug.Stack(0xc0002a1808, 0xc0003fa5e0, 0x1b)
	/root/go/src/runtime/debug/stack.go:24 +0x9f
helloworld/controllers.(*GuestbookReconciler).Reconcile(0xc0003c9dd0, 0xc0002d02f9, 0x7, 0xc0002d02e0, 0x10, 0x1d0410fe42, 0xc000456f30, 0xc000456ea8, 0xc000456ea0)
	/root/gopath/src/helloworld/controllers/guestbook_controller.go:49 +0x1a9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc00022a480, 0x1430e00, 0xc0003d24c0, 0x0)
	/root/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:256 +0x166
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00022a480, 0xc000469600)
	/root/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:232 +0xb0
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker(0xc00022a480)
	/root/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:211 +0x2b
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc000292980)
	/root/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:152 +0x5f
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000292980, 0x3b9aca00, 0x0, 0x1609101, 0xc000102480)
	/root/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:153 +0x105
k8s.io/apimachinery/pkg/util/wait.Until(0xc000292980, 0x3b9aca00, 0xc000102480)
	/root/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88 +0x4d
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
	/root/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:193 +0x32d

2021-01-24T09:52:33.633+0800	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "guestbook", "request": "default/guestbook-sample"}

刪除實例並停止controller

不再需要Guestbook實例的時候，執行以下命令即可刪除：

cd $GOPATH/src/helloworld
kubectl delete -f config/samples/

不再需要controller的時候，去它的控制台使用Ctrl+c中斷即可；

將controller製作成docker鏡像

至此，咱們已經體驗過了kubebuilder的基本功能，不過實際生產環境中controller一般都會運行在kubernetes環境內，像上面這種運行在kubernetes之外的方式就不合適了，咱們來試試將其做成docker鏡像然後在kubernetes環境運行；
這裡有個要求，就是您要有個kubernetes可以訪問的鏡像倉庫，例如局域網內的Harbor，或者公共的hub.docker.com，我這為了操作方便選擇了hub.docker.com，使用它的前提是擁有hub.docker.com的註冊帳號；
在kubebuilder電腦上，打開一個控制台，執行docker login命令登錄，根據提示輸入hub.docker.com的帳號和密碼，這樣就可以在當前控制台上執行docker push命令將鏡像推送到hub.docker.com上了（這個網站的網絡很差，可能要登錄好幾次才能成功）；
執行以下命令構建docker鏡像並推送到hub.docker.com，鏡像名為bolingcavalry/guestbook:002：

cd $GOPATH/src/helloworld
make docker-build docker-push IMG=bolingcavalry/guestbook:002

hub.docker.com的網絡狀況不是一般的差，kubebuilder電腦上的docker一定要設置鏡像加速，上述命令如果遭遇超時失敗，請重試幾次，此外，構建過程中還會下載諸多go模塊的依賴，也需要您耐心等待，也很容易遇到網絡問題，需要多次重試，所以，最好是使用局域網內搭建的Habor服務；
最終，命令執行成功後輸出如下：

[root@kubebuilder helloworld]# make docker-build docker-push IMG=bolingcavalry/guestbook:002
/root/gopath/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
/root/gopath/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
go test ./... -coverprofile cover.out
?   	helloworld	[no test files]
?   	helloworld/api/v1	[no test files]
ok  	helloworld/controllers	8.604s	coverage: 0.0% of statements
docker build . -t bolingcavalry/guestbook:002
Sending build context to Docker daemon  40.27MB
Step 1/14 : FROM golang:1.13 as builder
 ---> d6f3656320fe
Step 2/14 : WORKDIR /workspace
 ---> Using cache
 ---> 83d05ead1041
Step 3/14 : COPY go.mod go.mod
 ---> Using cache
 ---> ae3e15a529f4
Step 4/14 : COPY go.sum go.sum
 ---> Using cache
 ---> 082223532ccc
Step 5/14 : RUN go mod download
 ---> Using cache
 ---> bcdcfa1d65ca
Step 6/14 : COPY main.go main.go
 ---> Using cache
 ---> 81d6a629ca98
Step 7/14 : COPY api/ api/
 ---> Using cache
 ---> 75f99b174e97
Step 8/14 : COPY controllers/ controllers/
 ---> b130d9f47903
Step 9/14 : RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GO111MODULE=on go build -a -o manager main.go
 ---> Running in 768880aca19f
Removing intermediate container 768880aca19f
 ---> bb4a494d3b43
Step 10/14 : FROM gcr.io/distroless/static:nonroot
 ---> 947e6f3ed7c1
Step 11/14 : WORKDIR /
 ---> Using cache
 ---> 22cc43cef8fb
Step 12/14 : COPY --from=builder /workspace/manager .
 ---> 2137778f22c0
Step 13/14 : USER nonroot:nonroot
 ---> Running in 18295673073d
Removing intermediate container 18295673073d
 ---> f7545379ab1f
Step 14/14 : ENTRYPOINT ["/manager"]
 ---> Running in 550c47dd61dc
Removing intermediate container 550c47dd61dc
 ---> 31cb31a6b03f
Successfully built 31cb31a6b03f
Successfully tagged bolingcavalry/guestbook:002
docker push bolingcavalry/guestbook:002
The push refers to repository [docker.io/bolingcavalry/guestbook]
99035107a955: Pushed 
728501c5607d: Layer already exists 
002: digest: sha256:54f8ec88511cce5b04c5d65cc15e0f7a7b4a8afb6b235904a638bff79e3c5784 size: 739

去hub.docker.com網站看看，如下圖，新鏡像已經上傳，這樣只要任何機器只要能上網就能pull此鏡像到本地使用了：

8. 鏡像準備好之後，執行以下命令即可在kubernetes環境部署controller：

cd $GOPATH/src/helloworld
make deploy IMG=bolingcavalry/guestbook:002

控制台會提示各類資源被創建（rbac居多）：

[root@kubebuilder ~]# cd $GOPATH/src/helloworld
[root@kubebuilder helloworld]# make deploy IMG=bolingcavalry/guestbook:002
/root/gopath/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
cd config/manager && kustomize edit set image controller=bolingcavalry/guestbook:002
kustomize build config/default | kubectl apply -f -
namespace/helloworld-system created
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
customresourcedefinition.apiextensions.k8s.io/guestbooks.webapp.com.bolingcavalry configured
role.rbac.authorization.k8s.io/helloworld-leader-election-role created
clusterrole.rbac.authorization.k8s.io/helloworld-manager-role created
clusterrole.rbac.authorization.k8s.io/helloworld-proxy-role created
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRole is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRole
clusterrole.rbac.authorization.k8s.io/helloworld-metrics-reader created
rolebinding.rbac.authorization.k8s.io/helloworld-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/helloworld-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/helloworld-proxy-rolebinding created
service/helloworld-controller-manager-metrics-service created
deployment.apps/helloworld-controller-manager created

此時去看kubernetes環境的pod，發現確實已經新增了controller，如下圖紅框：

11.細心的您應該會發現上圖黃框中顯示這個pod實際上有兩個容器，用kubectl describe命令細看，分別是kube-rbac-proxy和manager，如下圖：

11. 由於有兩個容器，那麼查看日誌時就要指定其中一個了，咱們的controller對應的是manager容器，因此查看日誌的命令是：

kubectl logs -f \
helloworld-controller-manager-689d4b6f5b-h9pzg \
-n helloworld-system \
-c manager

再次創建Guestbook資源的實例，依舊是kubectl apply -f config/samples/命令，再去看manager容器的日誌，可見咱們修改的內容已經打印出來了：

卸載和清理

體驗完畢後，如果想把前面創建的資源和CRD全部清理掉，可以執行以下命令：

cd $GOPATH/src/helloworld
make uninstall

至此，通過kubebuilder創建Operator相關資源的基本流程，咱們已經體驗過一遍了，本篇以熟悉工具和流程為主，並未體驗到Operator實質性的強大功能，這些都留待後面的章節吧，咱們逐步深入學習實踐；

你不孤單，欣宸原創一路相伴

歡迎關注公眾號：程序員欣宸

微信搜索「程序員欣宸」，我是欣宸，期待與您一同暢遊Java世界…
//github.com/zq2599/blog_demos

kubebuilder實戰之二：初次體驗kubebuilder

歡迎訪問我的GitHub

系列文章鏈接

本篇概覽

創建helloworld項目

創建API(CRD和Controller)

構建和部署CRD

編譯和運行controller

創建Guestbook資源的實例

刪除實例並停止controller

將controller製作成docker鏡像

卸載和清理

你不孤單，欣宸原創一路相伴

歡迎關注公眾號：程序員欣宸

VirMach 便宜 VPS

QNews

kubebuilder實戰之二：初次體驗kubebuilder

歡迎訪問我的GitHub

系列文章鏈接

本篇概覽

創建helloworld項目

創建API(CRD和Controller)

構建和部署CRD

編譯和運行controller

創建Guestbook資源的實例

刪除實例並停止controller

將controller製作成docker鏡像

卸載和清理

你不孤單，欣宸原創一路相伴

歡迎關注公眾號：程序員欣宸

分享此文：

Related Posts

Docker | redis安裝及測試

活動預告 |「3306π」深圳站年會，猜對有獎

手機舉報違章威力巨大！有駕駛員一月被扣72分

特斯拉FSD自動駕駛秀技術：彎曲山路極限會車 穩若老司機

VirMach 便宜 VPS

QNews

熱門文章

熱門搜尋

特斯拉FSD自動駕駛秀技術：彎曲山路極限會車穩若老司機