kubernetes集群部署高可用Postgresql的Stolon方案
- 2019 年 10 月 3 日
- 笔记
目录
前言
….前言
本文选用Stolon的方式搭建Postgresql高可用方案，主要为Harbor提供高可用数据库，Harbor搭建可查看kubernetes搭建Harbor无坑及Harbor仓库同步，之后会提供redis高可用及Harbor高可用方案搭建
方案比较
几种postgresql高可用方案简单比较：
引用https://studygolang.com/articles/19002?fr=sidebar
- 首先repmgr这种方案的算法有明显缺陷，非主流分布式算法，直接pass；
- Stolon和Patroni相对于Crunchy更加Cloud Native， 后者是基于pgPool实现。
- Crunchy和Patroni相对于Stolon有更多的使用者，并且提供了Operator对于以后的管理和扩容
根据上面简单的比较，最终选择的stolon，作者选择的是Patroni，感觉实际区别并不大。
一、Stolon概述：
Stolon（https://github.com/sorintlab/stolon
）
是由3个部分组成的：
- keeper：他负责管理PostgreSQL的实例汇聚到由sentinel(s)提供的clusterview。
- sentinel：it负责发现并且监控keeper，并且计算最理想的clusterview。
-
proxy：客户端的接入点。它强制连接到右边PostgreSQL的master并且强制关闭连接到由非选举产生的master。
Stolon 用etcd或者consul作为主要的集群状态存储。二、Installation
git clone https://github.com/sorintlab/stolon.git cd XXX/stolon/examples/kubernetes如图所示
如有兴趣可查看官网搭建：https://github.com/sorintlab/stolon/blob/master/examples/kubernetes/README.md
如下为yaml中注意修改的地方
- stolon-keeper.yaml 中设置Postgresql用户名
- name: STKEEPER_PG_SU_USERNAME value: "postgres"- stolon-keeper.yaml 中设置stolon挂载卷
volumeClaimTemplates: - metadata: name: data spec: accessModes: - "ReadWriteOnce" resources: requests: storage: "512Mi" storageClassName: nfs- secret.yaml中设置用户密码
apiVersion: v1 kind: Secret metadata: name: stolon type: Opaque data: password: cGFzc3dvcmQx如下是作者整理的完整的stolon的编排文件，可直接修改使用
# This is an example and generic rbac role definition for stolon. It could be # fine tuned and split per component. # The required permission per component should be: # keeper/proxy/sentinel: update their own pod annotations # sentinel/stolonctl: get, create, update configmaps # sentinel/stolonctl: list components pods # sentinel/stolonctl: get components pods annotations apiVersion: rbac.authorization.k8s.io/v1beta1 kind: Role metadata: name: stolon namespace: default rules: - apiGroups: - "" resources: - pods - configmaps - events verbs: - "*" --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: RoleBinding metadata: name: stolon namespace: default roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: stolon subjects: - kind: ServiceAccount name: default namespace: default --- apiVersion: extensions/v1beta1 kind: Deployment metadata: name: stolon-sentinel spec: replicas: 2 template: metadata: labels: component: stolon-sentinel stolon-cluster: kube-stolon annotations: prometheus.io/scrape: "true" prometheus.io/port: "8080" spec: containers: - name: stolon-sentinel image: sorintlab/stolon:master-pg10 command: - "/bin/bash" - "-ec" - | exec gosu stolon stolon-sentinel env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: STSENTINEL_CLUSTER_NAME valueFrom: fieldRef: fieldPath: metadata.labels['stolon-cluster'] - name: STSENTINEL_STORE_BACKEND value: "kubernetes" - name: STSENTINEL_KUBE_RESOURCE_KIND value: "configmap" - name: STSENTINEL_METRICS_LISTEN_ADDRESS value: "0.0.0.0:8080" ## Uncomment this to enable debug logs #- name: STSENTINEL_DEBUG # value: "true" ports: - containerPort: 8080 --- apiVersion: v1 kind: Secret metadata: name: stolon type: Opaque data: password: cGFzc3dvcmQx --- # PetSet was renamed to StatefulSet in k8s 1.5 # apiVersion: apps/v1alpha1 # kind: PetSet apiVersion: apps/v1beta1 kind: StatefulSet metadata: name: stolon-keeper spec: serviceName: "stolon-keeper" replicas: 2 template: metadata: labels: component: stolon-keeper stolon-cluster: kube-stolon annotations: pod.alpha.kubernetes.io/initialized: "true" prometheus.io/scrape: "true" prometheus.io/port: "8080" spec: terminationGracePeriodSeconds: 10 containers: - name: stolon-keeper image: sorintlab/stolon:master-pg10 command: - "/bin/bash" - "-ec" - | # Generate our keeper uid using the pod index IFS='-' read -ra ADDR <<< "$(hostname)" export STKEEPER_UID="keeper${ADDR[-1]}" export POD_IP=$(hostname -i) export STKEEPER_PG_LISTEN_ADDRESS=$POD_IP export STOLON_DATA=/stolon-data chown stolon:stolon $STOLON_DATA exec gosu stolon stolon-keeper --data-dir $STOLON_DATA env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: STKEEPER_CLUSTER_NAME valueFrom: fieldRef: fieldPath: metadata.labels['stolon-cluster'] - name: STKEEPER_STORE_BACKEND value: "kubernetes" - name: STKEEPER_KUBE_RESOURCE_KIND value: "configmap" - name: STKEEPER_PG_REPL_USERNAME value: "repluser" # Or use a password file like in the below supersuser password - name: STKEEPER_PG_REPL_PASSWORD value: "replpassword" - name: STKEEPER_PG_SU_USERNAME value: "postgres" - name: STKEEPER_PG_SU_PASSWORDFILE value: "/etc/secrets/stolon/password" - name: STKEEPER_METRICS_LISTEN_ADDRESS value: "0.0.0.0:8080" # Uncomment this to enable debug logs #- name: STKEEPER_DEBUG # value: "true" ports: - containerPort: 5432 - containerPort: 8080 volumeMounts: - mountPath: /stolon-data name: data - mountPath: /etc/secrets/stolon name: stolon volumes: - name: stolon secret: secretName: stolon # Define your own volumeClaimTemplate. This example uses dynamic PV provisioning with a storage class named "standard" (so it will works by default with minikube) # In production you should use your own defined storage-class and configure your persistent volumes (statically or dynamically using a provisioner, see related k8s doc). volumeClaimTemplates: - metadata: name: data spec: accessModes: - "ReadWriteOnce" resources: requests: storage: "512Mi" storageClassName: nfs --- apiVersion: v1 kind: Service metadata: name: stolon-proxy-service spec: ports: - port: 5432 targetPort: 5432 selector: component: stolon-proxy stolon-cluster: kube-stolon --- apiVersion: extensions/v1beta1 kind: Deployment metadata: name: stolon-proxy spec: replicas: 2 template: metadata: labels: component: stolon-proxy stolon-cluster: kube-stolon annotations: prometheus.io/scrape: "true" prometheus.io/port: "8080" spec: containers: - name: stolon-proxy image: sorintlab/stolon:master-pg10 command: - "/bin/bash" - "-ec" - | exec gosu stolon stolon-proxy env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: STPROXY_CLUSTER_NAME valueFrom: fieldRef: fieldPath: metadata.labels['stolon-cluster'] - name: STPROXY_STORE_BACKEND value: "kubernetes" - name: STPROXY_KUBE_RESOURCE_KIND value: "configmap" - name: STPROXY_LISTEN_ADDRESS value: "0.0.0.0" - name: STPROXY_METRICS_LISTEN_ADDRESS value: "0.0.0.0:8080" ## Uncomment this to enable debug logs #- name: STPROXY_DEBUG # value: "true" ports: - containerPort: 5432 - containerPort: 8080 readinessProbe: tcpSocket: port: 5432 initialDelaySeconds: 10 timeoutSeconds: 5三、部署stolon
kubectl applay -f stolon.yamlInitialize the cluster（大概意思是stolon初始化k8s集群，可以大概看下官网解释）
All the stolon components wait for an existing clusterdata entry in the store. So the first time you have to initialize a new cluster. For more details see the cluster initialization doc. You can do this step at every moment, now or after having started the stolon components.
You can execute stolonctl in different ways:
- as a one shot command executed inside a temporary pod:
kubectl run -i -t stolonctl --image=sorintlab/stolon:master-pg10 --restart=Never --rm -- /usr/local/bin/stolonctl --cluster-name=kube-stolon --store-backend=kubernetes --kube-resource-kind=configmap init
- from a machine that can access the store backend:
stolonctl --cluster-name=kube-stolon --store-backend=kubernetes --kube-resource-kind=configmap init
- later from one of the pods running the stolon components.
执行
kubectl run -i -t stolonctl --image=sorintlab/stolon:master-pg10 --restart=Never --rm -- /usr/local/bin/stolonctl --cluster-name=kube-stolon --store-backend=kubernetes --kube-resource-kind=configmap init 如图所示，部署成功
四、卸载Postgresql数据库
kubectl delete -f stolon.yaml kubectl delete pvc data-stolon-keeper-0 data-stolon-keeper-1五、验证Postgresql安装成功（也可简单测试下）
1、验证数据同步
连接master并且建立test表
psql –host
–port 30543 postgres -U stolon -W
postgres=# create table test (id int primary key not null,
value text not null);
CREATE TABLE
postgres=# insert into test values (1, ‘value1’);
INSERT 0 1
postgres=# select * from test;
id | value
—- ——– 1 | value1
(1 row)
也可进入Pod执行postgresql命令
kubectl exec -ti stolon-proxy-5977cdbcfc-csnkq bash #登入sql psql --host localhost --port 5432 postgres -U postgres l #列出所有数据库 c dbname #切换数据库 CREATE TABLE insert into test values (1, 'value1'); INSERT 0 1 select * from test; d #列出当前数据库的所有表 q #退出数据库连接slave并且检查数据。你可以写一些信息以便确认请求已经被slave处理了。
psql –host
–port 30544 postgres -U stolon -W
postgres=# select * from test;
id | value
—- ——– 1 | value1
(1 row)
2、测试failover
这个案例是官方代码库中statefullset的一个例子。
简单的说，就是为模拟了master挂掉，我们先删除了master的statefulset又删除了master的pod。
kubectl delete statefulset stolon-keeper --cascade=false kubectl delete pod stolon-keeper-0然后，在sentinel的log中我们可以看到新的master被选举出来了。
no keeper info available db=cb96f42d keeper=keeper0
no keeper info available db=cb96f42d keeper=keeper0
master db is failed db=cb96f42d keeper=keeper0
trying to find a standby to replace failed master
electing db as the new master db=087ce88a keeper=keeper1
现在，在刚才的那两个终端中如果我们重复上一个命令，我们可以看到如下输出。
postgres=# select from test;
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The connection to the server was lost. Attempting reset:
Succeeded.
postgres=# select from test;
id | value
—- ——– 1 | value1
(1 row)
Kubernetes的service把不可用的pod去掉，把请求转到可用的pod上。所以新的读取连接被路由到了健康的pod上。
.也可用chaoskube模拟随机的pod挂掉（准生产可以测试下）
另一个测试集群弹性（resilience）的好方法是用chaoskube。Chaoskube是一个小的服务程序，它可以周期性的在集群里随机的kill掉一些的pod。它也可以用helm charts部署。
helm install --set labels="release=factualcrocodile, component!=factual-crocodine-etcd" --set interval=5m stable/chaoskube这条命令会运行chaoskube，它会每5分钟删除一个pod。它会选择label中release=factual-crocodile的pod，但是会忽略etcd的pod。
本文按照官网搭建，主要为之后的Harbor高可用做准备，有情趣的伙伴点个赞，之后会续写redis、Harbor高可用
参考资料：
https://my.oschina.net/u/2306127/blog/2991474
https://github.com/sorintlab/stolon/tree/master/examples/kubernetes
https://studygolang.com/articles/19002?fr=sidebar
