雲原生之旅 – 9)雲原生時代網關的後起之秀Envoy Proxy 和基於Envoy 的 Emissary Ingress
- 2022 年 11 月 7 日
- 筆記
- SRE/DevOps, 雲原生
前言
前一篇文章講述了基於Nginx代理的Kuberenetes Ingress Nginx【雲原生時代的網關 Ingress Nginx】這次給大家介紹下基於Envoy的 Emissary Ingress。
首先什麼是Enovy?
Envoy 是由 Lyft 開源的高性能網絡代理軟件,後來捐贈給了 CNCF 基金會,已經畢業於CNCF。 相比於 Nginx、HAProxy 等經典代理軟件,Envoy 具備豐富的可觀察性和靈活的可擴展性,並且引入了基於 xDS API 的動態配置方案,Envoy 還提供了大量的開箱即用的 Filter 以滿足各種場景下流量治理的需求。
Envoy 與 Nginx 代理的區別
- Envoy 對 HTTP/2 的支持比 Nginx 更好,支持包括 upstream 和 downstream在內的雙向通信,而 Nginx 只支持 downstream 的連接。
- 高級負載均衡功能是免費的,Nginx 的高級負載均衡功能則需要商業版 Nginx Plus 支持。
- Envoy 支持熱更新,Nginx 配置更新之後需要 Reload。
- Envoy 更貼近 Service Mesh 的使用習慣,Nginx 更貼近傳統服務的使用習慣。
Envoy 有典型的兩種工作模式。一種作為中心代理,代理集群的南北向流量,這種模式下,Envoy 一般就是負載均衡設備或者是 API 網關的基礎數據面,比如 Ambassador 現在叫 Emissary,Gloo 都是新興的開源的基於 Envoy 的開源網關。另一種模式,就是作為業務進程的 Sidecar,當有業務請求訪問業務的時候,流量會被劫持到 Sidecar Envoy 當中,之後再被轉發給業務進程,典型代表 Istio 和 Linkerd.
今天我們介紹的就是代理南北向流量的網關 Emissary Ingress(原名 Ambassador)。Emissary-ingress已經是CNCF的孵化項目,並且在去年被頂級服務網狀項目Linkerd和Istio正式支持。如需集成參考文檔。
關鍵詞:基於Enovy的Emissary Ingress實踐,Emissary Ingress入門,雲原生網關Emissary Ingress,Emissary Ingress實踐
為什麼選擇 Emissary Ingress
//www.getambassador.io/docs/emissary/latest/about/alternatives/
//www.getambassador.io/docs/emissary/latest/about/faq/#why-emissary-ingress
安裝
使用Terraform Helm Provider
從 emissary-ingress 2.1開始, 它把 CRDs 從Helm Charts移除了, 現在首先需要手動 apply CRDs。
kubectl apply -f //app.getambassador.io/yaml/emissary/3.2.0/emissary-crds.yaml
所以我做了一個Helm Charts 專門裝下CRDs,否則無法全流程安裝自動化。
如果不了解Helm Chart 請參考這篇文章【Kubernetes時代的包管理工具 Helm】入門。
resource "helm_release" "emissary_crds" { name = "emissary-crds" create_namespace = true # create emissary default namespace `emissary-system` namespace = local.emissary_ns chart = "../common/helm/repos/emissary-crds-8.2.0.tgz" }
CRDs是默認裝在`emissary-system` namespace下面的,不建議修改namespace,如果要在不同的Namespace下裝多個Emissary ingress, 是可以共用這個CRDs的。
# Install Emissary-ingress from Chart Repository resource "helm_release" "emissary_ingress" { name = "emissary-ingress" repository = "//app.getambassador.io" chart = "emissary-ingress" version = local.chart_version create_namespace = true namespace = local.emissary_ns values = [ templatefile("${local.common_yaml_d}/emissary-ingress-template.yaml", local.emissary_ingress_map) ] depends_on = [ helm_release.emissary_crds ] }
最後一部分,也是自製 chart 專門負責config
# This is for install Host/Listener/Mapping/TLSContext from a local custom chart # also can upload chart to a bucket or a public github for install from a url # e.g. [Publish to a GCS bucket](//github.com/hayorov/helm-gcs) resource "helm_release" "emissary_config" { name = "emissary-config" namespace = local.emissary_ns chart = "../common/helm/repos/emissary-config-8.2.0.tgz" values = [ templatefile("${local.common_yaml_d}/emissary-listeners-template.yaml", local.emissary_listeners_map), local.emissary_config_yaml ] depends_on = [ helm_release.emissary_ingress ] }
locals 變量


locals { project_id = "global-sre-dev" cluster_name = "sre-gke" cluster_region = "us-central1" emissary_ns = "emissary" chart_version = "8.2.0" common_yaml_d = "../common/helm/yamls" ambassador_id = "ambassador" emissary_ingress_map = { ambassadorID = local.ambassador_id loadBalancerIP = "35.232.98.249" # Prepare a Static IP first instead to use Ephemeral replicaCount = 2 minReplicas = 2 maxReplicas = 3 canaryEnabled = false # set to true in Prod logLevel = "error" # valid log levels are error, warn/warning, info, debug, and trace endpointEnable = true endpointName = "my-resolver" diagnosticsEnable = false clusterRequestTimeout = 120000 # milliseconds } emissary_listeners_map = { ambassadorID = local.ambassador_id listenersEnabled = true # custom listeners } }
locals.tf
config文件


locals { emissary_config_yaml = <<-EOT hosts: - name: my-host-dev spec: ambassador_id: - ${local.ambassador_id} hostname: '*.wadexu.cloud' requestPolicy: insecure: action: Redirect tlsContext: name: my-tls-context tlsSecret: name: tls-secret namespace: secret mappings: - name: my-nginx-mapping spec: ambassador_id: - ${local.ambassador_id} hostname: dev.wadexu.cloud prefix: / service: my-nginx.nginx:80 tlscontexts: - name: my-tls-context spec: ambassador_id: - ${local.ambassador_id} hosts: - "*.wadexu.cloud" min_tls_version: v1.2 EOT }
config.tf
完整代碼請參考 my repo
kubectl create secret -n secret tls tls-secret \ --key ./xxx.key \ --cert ./xxx.pem
Install from local, (Optional) 如果要學習自動化Terraform安裝,請參考【部署Terrform基礎設施代碼的自動化利器 Atlantis】
cd terraform_helm_install/dev terraform init terraform plan terraform apply
Install result
% helm list -n emissary-system NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION emissary-crds emissary-system 1 2022-10-20 10:09:30.72553 +0800 CST deployed emissary-crds-8.2.0 3.2.0 % helm list -n emissary NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION emissary-config emissary 1 2022-10-20 10:31:24.819555 +0800 CST deployed emissary-config-8.2.0 3.2.0 emissary-ingress emissary 1 2022-10-20 10:29:33.705888 +0800 CST deployed emissary-ingress-8.2.0 3.2.0
使用 Kustomize
參考我的 quick start
如果不了解 Kustomize, 請移步我這篇文章【不能錯過的一款 Kubernetes 應用編排管理神器 Kustomize】
一個集群安裝多個Emissary Ingress
我這個例子 This example 展示了 multiple Emissary deployed in one cluster.
在一個集群里安裝多個 Emissary 一定要設置 ambassador_id 並且替換 ClusterRoleBinding name, 否則資源衝突。
- emissary-ingress-init: CRDs will be installed.
- emissary-ingress-public: An emissary-ingress with allow list = all (face to internet).
- emissary-ingress-private: Another emissary-ingress with an allow list (restrict connection) installed in same cluster.
Test in local
# apply CRDs first kustomize build emissary-ingress-init/sre-mgmt-dev > ~/init.yaml kubectl apply -f ~/init.yaml # deploy first public Emissary, this allow list = all, face to internet kustomize build emissary-ingress-public/sre-mgmt-dev > ~/emissary_deploy1.yaml kubectl apply -f ~/emissary_deploy1.yaml # deploy second private Emissary with a restrict allow list to access kustomize build emissary-ingress-private/sre-mgmt-dev > ~/emissary_deploy2.yaml kubectl apply -f ~/emissary_deploy2.yaml
通過Terraform安裝 Kustomize資源,請參考 my repo
如:
module "example_custom_manifests" { source = "kbst.xyz/catalog/custom-manifests/kustomization" version = "0.3.0" configuration_base_key = "default" configuration = { default = { resources = [ "${path.root}/../../infra/emissary-ingress-init/sre-mgmt-dev" ] common_labels = { "env" = "dev" } } } }
Test
建一個nginx service 測試下
helm install my-nginx bitnami/nginx --set service.type="ClusterIP" -n nginx --create-namespace
curl
% curl //dev.wadexu.cloud <!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title> <style> html { color-scheme: light dark; } body { width: 35em; margin: 0 auto; font-family: Tahoma, Verdana, Arial, sans-serif; } </style> </head> <body> <h1>Welcome to nginx!</h1> <p>If you see this page, the nginx web server is successfully installed and working. Further configuration is required.</p> <p>For online documentation and support please refer to <a href="//nginx.org/">nginx.org</a>.<br/> Commercial support is available at <a href="//nginx.com/">nginx.com</a>.</p> <p><em>Thank you for using nginx.</em></p> </body> </html>
FAQ
1. 這個error 代表 tls-secret 有問題,確保正確創建
error:1404B42E:SSL routines:ST_CONNECT:tlsv1 alert protocol version
2. Connection refused, 最大的可能是 Listeners 沒有配置好。
curl: (7) Failed to connect to dev.wadexu.cloud port 443 after 255 ms: Connection refused
3. CRDs 沒創建。
│ Error: unable to build kubernetes objects from release manifest: [resource mapping not found for name: "my-resolver" namespace: "emissary-system" from "": no matches for kind "KubernetesEndpointResolver" in version "getambassador.io/v2" │ ensure CRDs are installed first, resource mapping not found for name: "ambassador" namespace: "emissary-system" from "": no matches for kind "Module" in version "getambassador.io/v2" │ ensure CRDs are installed first]