優化：在k8s上部署的gitlab
gitlab組件圖
gitlab在k8s上佔用資源
# kubectl top pods -n default | grep git*
gitlab-gitaly-0 9m 444Mi
gitlab-gitlab-exporter-59c6bdb69c-gz9nf 5m 36Mi
gitlab-gitlab-shell-547cc48d99-kmrbp 10m 21Mi
gitlab-gitlab-shell-547cc48d99-szg74 8m 25Mi
gitlab-minio-5746f7f7c7-tdff5 0m 47Mi
gitlab-prometheus-server-7d8bcf896d-vm5kq 14m 650Mi
gitlab-redis-cd6b45457-2s4cg 9m 17Mi
gitlab-registry-6c5df4646f-mbdkr 0m 15Mi
gitlab-registry-6c5df4646f-mqkxl 0m 31Mi
gitlab-sidekiq-all-in-1-576d89544d-8swmw 28m 967Mi
gitlab-unicorn-68f8f8d8d9-6clpv 8m 1510Mi
gitlab-unicorn-68f8f8d8d9-xbt8j 10m 1522Mi
gitlab組件簡介
對比kubectl top pods -n default | grep git* 輸出
- gitlab-gitaly：Gitaly是一項由GitLab設計的服務，旨在消除在GitLab的分佈式部署（請考慮GitLab.com或高可用性部署）中對Git存儲使用NFS的需求。從11.3.0開始，此服務處理GitLab中的所有Git級別訪問。
- gitlab-gitlab-exporter：GitLab Exporter是一個內部設計的流程，允許我們將有關GitLab應用程序內部的度量導出到Prometheus。可以在項目的README中閱讀更多內容。
- gitlab-gitlab-shell：GitLab Shell是GitLab設計的程序，用於處理基於SSH的
git會話，並修改授權密鑰的列表。GitLab Shell不是Unix Shell，也不是Bash或Zsh的替代品。 - gitlab-minio：MinIO是根據Apache License v2.0發佈的對象存儲服務器。它與Amazon S3雲存儲服務兼容。它最適合存儲非結構化數據，例如照片，視頻，日誌文件，備份和容器/ VM映像。一個對象的大小範圍可以從幾個KB到最大5TB。
- gitlab-prometheus-server：Prometheus是一個時序工具，可幫助GitLab管理員公開有關用於向GitLab提供服務的各個流程的指標。
- gitlab-redis：鍵值存儲/非持久數據庫。
- gitlab-registry：gitlab自帶鏡像倉庫
- gitlab-sidekiq：Sidekiq是Ruby後台作業處理器，可從Redis隊列中提取作業並進行處理。後台作業使GitLab通過將工作移至後台來提供更快的請求/響應周期。
- gitlab-unicorn：Unicorn是一個Ruby應用程序服務器，用於運行核心的Rails應用程序，該應用程序在GitLab中提供面向用戶的功能。通常，過程輸出，你會認為這
bundle還是config.ru取決於GitLab版本。
疑問點
目的
對在k8s上部署的gitlab進行內存優化
思路：
\1. 尋找限制內存的方法((1)gitlab自身的資源限制策略【最好】，k8s做資源限制【最差】)
2.刪除非必要的應用程序(如刪除prometheus【後期會把gitlab遷移到裸機並且對這個集群系統做監控】)。
策略
嘗試關閉prometheus組件，看是否對gitlab有影響
查看與gitlab-prometheus有關的資源清單
# kubectl get svc,pod,ingress,deployment | grep prom
發現沒有prometheus的ingress，這說明沒有走ingress-nginx代理，也能說明沒有人曾訪問過！！！
導出gitlab_prometheus_deployment.yaml
# kubectl get deployment gitlab-prometheus-server -o yaml > gitlab_prometheus_deployment.yaml
# cat gitlab_prometheus_deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "1"
creationTimestamp: "2019-10-21T04:14:50Z"
generation: 1
labels:
app: prometheus
chart: prometheus-5.5.3
component: server
heritage: Tiller
release: gitlab
name: gitlab-prometheus-server
namespace: default
resourceVersion: "1438304832"
selfLink: /apis/extensions/v1beta1/namespaces/default/deployments/gitlab-prometheus-server
uid: 5360e3bf-f3b9-11e9-800a-8ad2fc7e2fa8
spec:
progressDeadlineSeconds: 2147483647
replicas: 1
revisionHistoryLimit: 2147483647
selector:
matchLabels:
app: prometheus
component: server
release: gitlab
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: prometheus
component: server
release: gitlab
spec:
containers:
- args:
- --volume-dir=/etc/config
- --webhook-url=//localhost:9090/-/reload
image: jimmidyson/configmap-reload:v0.1
imagePullPolicy: IfNotPresent
name: prometheus-server-configmap-reload
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/config
name: config-volume
readOnly: true
- args:
- --storage.tsdb.retention=15d
- --config.file=/etc/config/prometheus.yml
- --storage.tsdb.path=/data
- --web.console.libraries=/etc/prometheus/console_libraries
- --web.console.templates=/etc/prometheus/consoles
- --web.enable-lifecycle
image: prom/prometheus:v2.2.1
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /-/healthy
port: 9090
scheme: HTTP
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 30
name: prometheus-server
ports:
- containerPort: 9090
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /-/ready
port: 9090
scheme: HTTP
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 30
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/config
name: config-volume
- mountPath: /data
name: storage-volume
dnsPolicy: ClusterFirst
initContainers:
- command:
- chown
- -R
- 65534:65534
- /data
image: busybox:latest
imagePullPolicy: IfNotPresent
name: init-chown-data
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /data
name: storage-volume
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: gitlab-prometheus-server
serviceAccountName: gitlab-prometheus-server
terminationGracePeriodSeconds: 300
volumes:
- configMap:
defaultMode: 420
name: gitlab-prometheus-server
name: config-volume
- name: storage-volume
persistentVolumeClaim:
claimName: gitlab-prometheus-server
導出gitlab_prometheus_svc.yaml
# kubectl get svc gitlab-prometheus-server -o yaml > gitlab_prometheus_svc.yaml
# cat gitlab_prometheus_svc.yaml
apiVersion: v1
kind: Service
metadata:
creationTimestamp: "2019-10-21T04:14:50Z"
labels:
app: prometheus
chart: prometheus-5.5.3
component: server
heritage: Tiller
release: gitlab
name: gitlab-prometheus-server
namespace: default
resourceVersion: "144804734"
selfLink: /api/v1/namespaces/default/services/gitlab-prometheus-server
uid: 535378f3-f3b9-11e9-800a-8ad2fc7e2fa8
spec:
clusterIP: 192.168.247.105
ports:
- name: http
port: 80
protocol: TCP
targetPort: 9090
selector:
app: prometheus
component: server
release: gitlab
sessionAffinity: None
type: ClusterIP
大膽刪除deployment和svc，能有啥事？
# kubectl delete -f
手動測試，似乎沒得啥問題。
導出gitlab_exporter_deployment.yaml
# kubectl get deployment gitlab-gitlab-exporter -o yaml > gitlab_exporter_deployment.yaml
# cat gitlab_exporter_deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "2"
creationTimestamp: "2019-10-21T04:14:50Z"
generation: 2
labels:
app: gitlab-exporter
chart: gitlab-exporter-2.4.6
heritage: Tiller
release: gitlab
name: gitlab-gitlab-exporter
namespace: default
resourceVersion: "1399100783"
selfLink: /apis/extensions/v1beta1/namespaces/default/deployments/gitlab-gitlab-exporter
uid: 5358e5e4-f3b9-11e9-800a-8ad2fc7e2fa8
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: gitlab-exporter
release: gitlab
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
annotations:
checksum/config: 158ed2dba0a2fbaa0d3fda906d70cf0eb7a80650e8f01d3046aa9a6fa0bba59a
prometheus.io/path: /metrics
prometheus.io/port: "9168"
prometheus.io/scrape: "true"
creationTimestamp: null
labels:
app: gitlab-exporter
release: gitlab
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchLabels:
app: gitlab-exporter
release: gitlab
topologyKey: kubernetes.io/hostname
weight: 1
containers:
- env:
- name: CONFIG_TEMPLATE_DIRECTORY
value: /var/opt/gitlab-exporter/templates
- name: CONFIG_DIRECTORY
value: /etc/gitlab-exporter
image: registry.gitlab.com/gitlab-org/build/cng/gitlab-exporter:5.0.1
imagePullPolicy: IfNotPresent
lifecycle:
preStop:
exec:
command:
- /bin/bash
- -c
- pkill -f 'gitlab-exporter'
livenessProbe:
exec:
command:
- pgrep
- -f
- gitlab-exporter
failureThreshold: 3
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
name: gitlab-exporter
ports:
- containerPort: 9168
name: gitlab-exporter
protocol: TCP
readinessProbe:
exec:
command:
- pgrep
- -f
- gitlab-exporter
failureThreshold: 3
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
requests:
cpu: 50m
memory: 100M
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/opt/gitlab-exporter/templates/gitlab-exporter.yml.erb
name: gitlab-exporter-config
subPath: gitlab-exporter.yml.erb
- mountPath: /etc/gitlab
name: gitlab-exporter-secrets
readOnly: true
- mountPath: /etc/ssl/certs/
name: etc-ssl-certs
readOnly: true
dnsPolicy: ClusterFirst
initContainers:
- image: registry.gitlab.com/gitlab-org/build/cng/alpine-certificates:20171114-r3
imagePullPolicy: IfNotPresent
name: certificates
resources:
requests:
cpu: 50m
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/ssl/certs
name: etc-ssl-certs
- command:
- sh
- /config/configure
image: busybox:latest
imagePullPolicy: Always
name: configure
resources:
requests:
cpu: 50m
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /config
name: gitlab-exporter-config
readOnly: true
- mountPath: /init-config
name: init-gitlab-exporter-secrets
readOnly: true
- mountPath: /init-secrets
name: gitlab-exporter-secrets
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
fsGroup: 1000
runAsUser: 1000
terminationGracePeriodSeconds: 30
volumes:
- configMap:
defaultMode: 420
name: gitlab-gitlab-exporter
name: gitlab-exporter-config
- name: init-gitlab-exporter-secrets
projected:
defaultMode: 256
sources:
- secret:
items:
- key: postgres-password
path: postgres/psql-password
name: gitlab-postgresql-password
- secret:
items:
- key: secret
path: redis/password
name: gitlab-redis-secret
- emptyDir:
medium: Memory
name: gitlab-exporter-secrets
- emptyDir:
medium: Memory
name: etc-ssl-certs
資源清單好長，有點不想看 emmmmm 還是分析一波吧
看過後，刪除exporter的資源清單
# kubectl delete -f gitlab_exporter_deployment.yaml
deployment.extensions "gitlab-gitlab-exporter" deleted
導出gitlab-exporter的svc資源清單
# kubectl get svc gitlab-gitlab-exporter -o yaml > gitlab_exporter_svc.yaml
[root@mobanji gitlab]# cat gitlab_exporter_svc.yaml
apiVersion: v1
kind: Service
metadata:
creationTimestamp: "2019-10-21T04:14:50Z"
labels:
app: gitlab-exporter
chart: gitlab-exporter-2.4.6
heritage: Tiller
release: gitlab
name: gitlab-gitlab-exporter
namespace: default
resourceVersion: "219393250"
selfLink: /api/v1/namespaces/default/services/gitlab-gitlab-exporter
uid: 534c0460-f3b9-11e9-800a-8ad2fc7e2fa8
spec:
clusterIP: 192.168.180.175
ports:
- name: gitlab-exporter
port: 9168
protocol: TCP
targetPort: 9168
selector:
app: gitlab-exporter
release: gitlab
sessionAffinity: None
type: ClusterIP
分析一下，沒發現重要信息，也刪了吧
# kubectl delete -f gitlab_exporter_svc.yaml
service "gitlab-gitlab-exporter" deleted
至此，與prometheus有關的資源被從kubernetes上刪除。
測試驗證。1. 手動刷新gitlab界面看有問題沒 2. 使用gitlab發佈一個臨時項目看有問題沒。
經過驗證暫時沒的問題
限制gitlab unicorn內存使用
目前gitlab在k8s上的資源佔用情況
# kubectl top pods | grep gitlab
gitlab-gitaly-0 9m 366Mi
gitlab-gitlab-shell-547cc48d99-kmrbp 18m 19Mi
gitlab-gitlab-shell-547cc48d99-szg74 15m 17Mi
gitlab-minio-5746f7f7c7-tdff5 0m 41Mi
gitlab-redis-cd6b45457-2s4cg 9m 14Mi
gitlab-registry-6c5df4646f-mbdkr 0m 31Mi
gitlab-registry-6c5df4646f-mqkxl 0m 40Mi
gitlab-sidekiq-all-in-1-576d89544d-8swmw 28m 967Mi
gitlab-unicorn-68f8f8d8d9-6clpv 7m 1514Mi
gitlab-unicorn-68f8f8d8d9-xbt8j 8m 1526Mi
可以看出 目前佔用資源最高的是unicorn組件。Unicorn]是一個Ruby應用程序服務器，用於運行核心的Rails應用程序，該應用程序在GitLab中提供面向用戶的功能。通常，過程輸出，你會認為這bundle還是config.ru取決於GitLab版本。
通過官網可以看出，這組件屬於核心組件。貌似還不能動。還是先導出deployment，然後根據資源清單進行判斷吧
# kubectl get deployment gitlab-unicorn -o yaml > gitlab_unicorn_deployment.yaml
# cat gitlab_unicorn_deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "2"
creationTimestamp: "2019-10-21T04:14:50Z"
generation: 16
labels:
app: unicorn
chart: unicorn-2.4.6
heritage: Tiller
release: gitlab
name: gitlab-unicorn
namespace: default
resourceVersion: "1359934803"
selfLink: /apis/extensions/v1beta1/namespaces/default/deployments/gitlab-unicorn
uid: 535e0ed5-f3b9-11e9-800a-8ad2fc7e2fa8
spec:
progressDeadlineSeconds: 600
replicas: 2
revisionHistoryLimit: 10
selector:
matchLabels:
app: unicorn
release: gitlab
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
annotations:
checksum/config: 2d845854ae56029b9ee0a30ca793e5c0f74c3d708329ba605bbdc36000023517
cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
prometheus.io/path: /-/metrics
prometheus.io/port: "8080"
prometheus.io/scrape: "true"
creationTimestamp: null
labels:
app: unicorn
release: gitlab
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchLabels:
app: unicorn
release: gitlab
topologyKey: kubernetes.io/hostname
weight: 1
containers:
- env:
- name: TMPDIR
value: /tmp/gitlab
- name: GITALY_FEATURE_DEFAULT_ON
value: "1"
- name: CONFIG_TEMPLATE_DIRECTORY
value: /var/opt/gitlab/templates
- name: CONFIG_DIRECTORY
value: /srv/gitlab/config
- name: prometheus_multiproc_dir
value: /metrics
image: registry.gitlab.com/gitlab-org/build/cng/gitlab-unicorn-ee:v12.4.2
imagePullPolicy: IfNotPresent
lifecycle:
preStop:
exec:
command:
- /bin/bash
- -c
- pkill -SIGQUIT -f 'unicorn master'
livenessProbe:
exec:
command:
- /scripts/healthcheck
failureThreshold: 3
initialDelaySeconds: 20
periodSeconds: 60
successThreshold: 1
timeoutSeconds: 30
name: unicorn
ports:
- containerPort: 8080
name: unicorn
protocol: TCP
readinessProbe:
exec:
command:
- /scripts/healthcheck
failureThreshold: 3
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 2
resources:
requests:
cpu: 300m
memory: 1200M
#有requests，沒有limits。只做最低保障，不做硬限制。還不能貿然做limits限制
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /metrics
name: unicorn-metrics
- mountPath: /var/opt/gitlab/templates
name: unicorn-config
- mountPath: /etc/gitlab
name: unicorn-secrets
readOnly: true
- mountPath: /srv/gitlab/config/secrets.yml
name: unicorn-secrets
subPath: rails-secrets/secrets.yml
- mountPath: /srv/gitlab/config/initializers/smtp_settings.rb
name: unicorn-config
subPath: smtp_settings.rb
- mountPath: /srv/gitlab/INSTALLATION_TYPE
name: unicorn-config
subPath: installation_type
- mountPath: /srv/gitlab/public/uploads/tmp
name: shared-upload-directory
- mountPath: /srv/gitlab/shared
name: shared-artifact-directory
- mountPath: /tmp
name: shared-tmp
- mountPath: /etc/ssl/certs/
name: etc-ssl-certs
readOnly: true
- env:
- name: TMPDIR
value: /tmp/gitlab
- name: GITLAB_WORKHORSE_EXTRA_ARGS
- name: GITLAB_WORKHORSE_LISTEN_PORT
value: "8181"
- name: CONFIG_TEMPLATE_DIRECTORY
value: /var/opt/gitlab/templates
- name: CONFIG_DIRECTORY
value: /srv/gitlab/config
image: registry.gitlab.com/gitlab-org/build/cng/gitlab-workhorse-ee:v12.4.2
imagePullPolicy: IfNotPresent
livenessProbe:
exec:
command:
- /scripts/healthcheck
failureThreshold: 3
initialDelaySeconds: 20
periodSeconds: 60
successThreshold: 1
timeoutSeconds: 30
name: gitlab-workhorse
ports:
- containerPort: 8181
name: workhorse
protocol: TCP
readinessProbe:
exec:
command:
- /scripts/healthcheck
failureThreshold: 3
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 2
resources:
requests:
cpu: 100m
memory: 100M
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/opt/gitlab/templates
name: workhorse-config
- mountPath: /etc/gitlab
name: workhorse-secrets
readOnly: true
- mountPath: /srv/gitlab/public/uploads/tmp
name: shared-upload-directory
- mountPath: /srv/gitlab/shared
name: shared-artifact-directory
- mountPath: /tmp
name: shared-tmp
- mountPath: /etc/ssl/certs/
name: etc-ssl-certs
readOnly: true
dnsPolicy: ClusterFirst
initContainers:
- image: registry.gitlab.com/gitlab-org/build/cng/alpine-certificates:20171114-r3
imagePullPolicy: IfNotPresent
name: certificates
resources:
requests:
cpu: 50m
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/ssl/certs
name: etc-ssl-certs
- args:
- -c
- sh -x /config-unicorn/configure ; sh -x /config-workhorse/configure ; mkdir
-p -m 3770 /tmp/gitlab
command:
- sh
image: busybox:latest
imagePullPolicy: Always
name: configure
resources:
requests:
cpu: 50m
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /config-unicorn
name: unicorn-config
readOnly: true
- mountPath: /config-workhorse
name: workhorse-config
readOnly: true
- mountPath: /init-config
name: init-unicorn-secrets
readOnly: true
- mountPath: /init-secrets
name: unicorn-secrets
- mountPath: /init-secrets-workhorse
name: workhorse-secrets
- mountPath: /tmp
name: shared-tmp
- args:
- /scripts/wait-for-deps
env:
- name: GITALY_FEATURE_DEFAULT_ON
value: "1"
- name: CONFIG_TEMPLATE_DIRECTORY
value: /var/opt/gitlab/templates
- name: CONFIG_DIRECTORY
value: /srv/gitlab/config
- name: WORKHORSE_ARCHIVE_CACHE_DISABLED
value: "1"
image: registry.gitlab.com/gitlab-org/build/cng/gitlab-unicorn-ee:v12.4.2
imagePullPolicy: IfNotPresent
name: dependencies
resources:
requests:
cpu: 50m
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/opt/gitlab/templates
name: unicorn-config
- mountPath: /etc/gitlab
name: unicorn-secrets
readOnly: true
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
fsGroup: 1000
runAsUser: 1000
terminationGracePeriodSeconds: 30
volumes:
- emptyDir: {}
name: shared-tmp
- emptyDir:
medium: Memory
name: unicorn-metrics
- configMap:
defaultMode: 420
name: gitlab-unicorn
name: unicorn-config
- configMap:
defaultMode: 420
name: gitlab-workhorse-config
name: workhorse-config
- name: init-unicorn-secrets
projected:
defaultMode: 256
sources:
- secret:
items:
- key: secrets.yml
path: rails-secrets/secrets.yml
name: gitlab-rails-secret
- secret:
items:
- key: secret
path: shell/.gitlab_shell_secret
name: gitlab-gitlab-shell-secret
- secret:
items:
- key: token
path: gitaly/gitaly_token
name: gitlab-gitaly-secret
- secret:
items:
- key: secret
path: redis/password
name: gitlab-redis-secret
- secret:
items:
- key: postgres-password
path: postgres/psql-password
name: gitlab-postgresql-password
- secret:
items:
- key: registry-auth.key
path: registry/gitlab-registry.key
name: gitlab-registry-secret
- secret:
items:
- key: shared_secret
path: gitlab-workhorse/secret
name: gitlab-gitlab-workhorse-secret
- secret:
items:
- key: accesskey
path: minio/accesskey
- key: secretkey
path: minio/secretkey
name: gitlab-minio-secret
- secret:
items:
- key: password
path: smtp/smtp-password
name: smtp-password
- emptyDir:
medium: Memory
name: unicorn-secrets
- emptyDir:
medium: Memory
name: workhorse-secrets
- emptyDir: {}
name: shared-upload-directory
- emptyDir: {}
name: shared-artifact-directory
- emptyDir:
medium: Memory
name: etc-ssl-certs
得了，分析了一波還是決定不動了，優化1G也算優化吧
