kubernetes集群部署高可用Postgresql的Stolon方案
- 2019 年 10 月 3 日
- 筆記
目錄
前言
….前言
本文選用Stolon的方式搭建Postgresql高可用方案，主要為Harbor提供高可用資料庫，Harbor搭建可查看kubernetes搭建Harbor無坑及Harbor倉庫同步，之後會提供redis高可用及Harbor高可用方案搭建
方案比較
幾種postgresql高可用方案簡單比較：
引用https://studygolang.com/articles/19002?fr=sidebar
- 首先repmgr這種方案的演算法有明顯缺陷，非主流分散式演算法，直接pass；
- Stolon和Patroni相對於Crunchy更加Cloud Native， 後者是基於pgPool實現。
- Crunchy和Patroni相對於Stolon有更多的使用者，並且提供了Operator對於以後的管理和擴容
根據上面簡單的比較，最終選擇的stolon，作者選擇的是Patroni，感覺實際區別並不大。
一、Stolon概述：
Stolon（https://github.com/sorintlab/stolon
）
是由3個部分組成的：
- keeper：他負責管理PostgreSQL的實例匯聚到由sentinel(s)提供的clusterview。
- sentinel：it負責發現並且監控keeper，並且計算最理想的clusterview。
-
proxy：客戶端的接入點。它強制連接到右邊PostgreSQL的master並且強制關閉連接到由非選舉產生的master。
Stolon 用etcd或者consul作為主要的集群狀態存儲。二、Installation
git clone https://github.com/sorintlab/stolon.git cd XXX/stolon/examples/kubernetes如圖所示
如有興趣可查看官網搭建：https://github.com/sorintlab/stolon/blob/master/examples/kubernetes/README.md
如下為yaml中注意修改的地方
- stolon-keeper.yaml 中設置Postgresql用戶名
- name: STKEEPER_PG_SU_USERNAME value: "postgres"- stolon-keeper.yaml 中設置stolon掛載卷
volumeClaimTemplates: - metadata: name: data spec: accessModes: - "ReadWriteOnce" resources: requests: storage: "512Mi" storageClassName: nfs- secret.yaml中設置用戶密碼
apiVersion: v1 kind: Secret metadata: name: stolon type: Opaque data: password: cGFzc3dvcmQx如下是作者整理的完整的stolon的編排文件，可直接修改使用
# This is an example and generic rbac role definition for stolon. It could be # fine tuned and split per component. # The required permission per component should be: # keeper/proxy/sentinel: update their own pod annotations # sentinel/stolonctl: get, create, update configmaps # sentinel/stolonctl: list components pods # sentinel/stolonctl: get components pods annotations apiVersion: rbac.authorization.k8s.io/v1beta1 kind: Role metadata: name: stolon namespace: default rules: - apiGroups: - "" resources: - pods - configmaps - events verbs: - "*" --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: RoleBinding metadata: name: stolon namespace: default roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: stolon subjects: - kind: ServiceAccount name: default namespace: default --- apiVersion: extensions/v1beta1 kind: Deployment metadata: name: stolon-sentinel spec: replicas: 2 template: metadata: labels: component: stolon-sentinel stolon-cluster: kube-stolon annotations: prometheus.io/scrape: "true" prometheus.io/port: "8080" spec: containers: - name: stolon-sentinel image: sorintlab/stolon:master-pg10 command: - "/bin/bash" - "-ec" - | exec gosu stolon stolon-sentinel env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: STSENTINEL_CLUSTER_NAME valueFrom: fieldRef: fieldPath: metadata.labels['stolon-cluster'] - name: STSENTINEL_STORE_BACKEND value: "kubernetes" - name: STSENTINEL_KUBE_RESOURCE_KIND value: "configmap" - name: STSENTINEL_METRICS_LISTEN_ADDRESS value: "0.0.0.0:8080" ## Uncomment this to enable debug logs #- name: STSENTINEL_DEBUG # value: "true" ports: - containerPort: 8080 --- apiVersion: v1 kind: Secret metadata: name: stolon type: Opaque data: password: cGFzc3dvcmQx --- # PetSet was renamed to StatefulSet in k8s 1.5 # apiVersion: apps/v1alpha1 # kind: PetSet apiVersion: apps/v1beta1 kind: StatefulSet metadata: name: stolon-keeper spec: serviceName: "stolon-keeper" replicas: 2 template: metadata: labels: component: stolon-keeper stolon-cluster: kube-stolon annotations: pod.alpha.kubernetes.io/initialized: "true" prometheus.io/scrape: "true" prometheus.io/port: "8080" spec: terminationGracePeriodSeconds: 10 containers: - name: stolon-keeper image: sorintlab/stolon:master-pg10 command: - "/bin/bash" - "-ec" - | # Generate our keeper uid using the pod index IFS='-' read -ra ADDR <<< "$(hostname)" export STKEEPER_UID="keeper${ADDR[-1]}" export POD_IP=$(hostname -i) export STKEEPER_PG_LISTEN_ADDRESS=$POD_IP export STOLON_DATA=/stolon-data chown stolon:stolon $STOLON_DATA exec gosu stolon stolon-keeper --data-dir $STOLON_DATA env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: STKEEPER_CLUSTER_NAME valueFrom: fieldRef: fieldPath: metadata.labels['stolon-cluster'] - name: STKEEPER_STORE_BACKEND value: "kubernetes" - name: STKEEPER_KUBE_RESOURCE_KIND value: "configmap" - name: STKEEPER_PG_REPL_USERNAME value: "repluser" # Or use a password file like in the below supersuser password - name: STKEEPER_PG_REPL_PASSWORD value: "replpassword" - name: STKEEPER_PG_SU_USERNAME value: "postgres" - name: STKEEPER_PG_SU_PASSWORDFILE value: "/etc/secrets/stolon/password" - name: STKEEPER_METRICS_LISTEN_ADDRESS value: "0.0.0.0:8080" # Uncomment this to enable debug logs #- name: STKEEPER_DEBUG # value: "true" ports: - containerPort: 5432 - containerPort: 8080 volumeMounts: - mountPath: /stolon-data name: data - mountPath: /etc/secrets/stolon name: stolon volumes: - name: stolon secret: secretName: stolon # Define your own volumeClaimTemplate. This example uses dynamic PV provisioning with a storage class named "standard" (so it will works by default with minikube) # In production you should use your own defined storage-class and configure your persistent volumes (statically or dynamically using a provisioner, see related k8s doc). volumeClaimTemplates: - metadata: name: data spec: accessModes: - "ReadWriteOnce" resources: requests: storage: "512Mi" storageClassName: nfs --- apiVersion: v1 kind: Service metadata: name: stolon-proxy-service spec: ports: - port: 5432 targetPort: 5432 selector: component: stolon-proxy stolon-cluster: kube-stolon --- apiVersion: extensions/v1beta1 kind: Deployment metadata: name: stolon-proxy spec: replicas: 2 template: metadata: labels: component: stolon-proxy stolon-cluster: kube-stolon annotations: prometheus.io/scrape: "true" prometheus.io/port: "8080" spec: containers: - name: stolon-proxy image: sorintlab/stolon:master-pg10 command: - "/bin/bash" - "-ec" - | exec gosu stolon stolon-proxy env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: STPROXY_CLUSTER_NAME valueFrom: fieldRef: fieldPath: metadata.labels['stolon-cluster'] - name: STPROXY_STORE_BACKEND value: "kubernetes" - name: STPROXY_KUBE_RESOURCE_KIND value: "configmap" - name: STPROXY_LISTEN_ADDRESS value: "0.0.0.0" - name: STPROXY_METRICS_LISTEN_ADDRESS value: "0.0.0.0:8080" ## Uncomment this to enable debug logs #- name: STPROXY_DEBUG # value: "true" ports: - containerPort: 5432 - containerPort: 8080 readinessProbe: tcpSocket: port: 5432 initialDelaySeconds: 10 timeoutSeconds: 5三、部署stolon
kubectl applay -f stolon.yamlInitialize the cluster（大概意思是stolon初始化k8s集群，可以大概看下官網解釋）
All the stolon components wait for an existing clusterdata entry in the store. So the first time you have to initialize a new cluster. For more details see the cluster initialization doc. You can do this step at every moment, now or after having started the stolon components.
You can execute stolonctl in different ways:
- as a one shot command executed inside a temporary pod:
kubectl run -i -t stolonctl --image=sorintlab/stolon:master-pg10 --restart=Never --rm -- /usr/local/bin/stolonctl --cluster-name=kube-stolon --store-backend=kubernetes --kube-resource-kind=configmap init
- from a machine that can access the store backend:
stolonctl --cluster-name=kube-stolon --store-backend=kubernetes --kube-resource-kind=configmap init
- later from one of the pods running the stolon components.
執行
kubectl run -i -t stolonctl --image=sorintlab/stolon:master-pg10 --restart=Never --rm -- /usr/local/bin/stolonctl --cluster-name=kube-stolon --store-backend=kubernetes --kube-resource-kind=configmap init 如圖所示，部署成功
四、卸載Postgresql資料庫
kubectl delete -f stolon.yaml kubectl delete pvc data-stolon-keeper-0 data-stolon-keeper-1五、驗證Postgresql安裝成功（也可簡單測試下）
1、驗證數據同步
連接master並且建立test表
psql –host
–port 30543 postgres -U stolon -W
postgres=# create table test (id int primary key not null,
value text not null);
CREATE TABLE
postgres=# insert into test values (1, ‘value1’);
INSERT 0 1
postgres=# select * from test;
id | value
—- ——– 1 | value1
(1 row)
也可進入Pod執行postgresql命令
kubectl exec -ti stolon-proxy-5977cdbcfc-csnkq bash #登入sql psql --host localhost --port 5432 postgres -U postgres l #列出所有資料庫 c dbname #切換資料庫 CREATE TABLE insert into test values (1, 'value1'); INSERT 0 1 select * from test; d #列出當前資料庫的所有表 q #退出資料庫連接slave並且檢查數據。你可以寫一些資訊以便確認請求已經被slave處理了。
psql –host
–port 30544 postgres -U stolon -W
postgres=# select * from test;
id | value
—- ——– 1 | value1
(1 row)
2、測試failover
這個案例是官方程式碼庫中statefullset的一個例子。
簡單的說，就是為模擬了master掛掉，我們先刪除了master的statefulset又刪除了master的pod。
kubectl delete statefulset stolon-keeper --cascade=false kubectl delete pod stolon-keeper-0然後，在sentinel的log中我們可以看到新的master被選舉出來了。
no keeper info available db=cb96f42d keeper=keeper0
no keeper info available db=cb96f42d keeper=keeper0
master db is failed db=cb96f42d keeper=keeper0
trying to find a standby to replace failed master
electing db as the new master db=087ce88a keeper=keeper1
現在，在剛才的那兩個終端中如果我們重複上一個命令，我們可以看到如下輸出。
postgres=# select from test;
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The connection to the server was lost. Attempting reset:
Succeeded.
postgres=# select from test;
id | value
—- ——– 1 | value1
(1 row)
Kubernetes的service把不可用的pod去掉，把請求轉到可用的pod上。所以新的讀取連接被路由到了健康的pod上。
.也可用chaoskube模擬隨機的pod掛掉（准生產可以測試下）
另一個測試集群彈性（resilience）的好方法是用chaoskube。Chaoskube是一個小的服務程式，它可以周期性的在集群里隨機的kill掉一些的pod。它也可以用helm charts部署。
helm install --set labels="release=factualcrocodile, component!=factual-crocodine-etcd" --set interval=5m stable/chaoskube這條命令會運行chaoskube，它會每5分鐘刪除一個pod。它會選擇label中release=factual-crocodile的pod，但是會忽略etcd的pod。
本文按照官網搭建，主要為之後的Harbor高可用做準備，有情趣的夥伴點個贊，之後會續寫redis、Harbor高可用
參考資料：
https://my.oschina.net/u/2306127/blog/2991474
https://github.com/sorintlab/stolon/tree/master/examples/kubernetes
https://studygolang.com/articles/19002?fr=sidebar
