k8s endpoints controller分析
- 2021 年 11 月 14 日
- 筆記
- kube-controller-manager分析, kubernetes源码解析
k8s endpoints controller分析
endpoints controller简介
endpoints controller是kube-controller-manager组件中众多控制器中的一个，是 endpoints 资源对象的控制器，其通过对service、pod 2种资源的监听，当这2种资源发生变化时会触发 endpoints controller 对相应的endpoints资源进行调谐操作，从而完成endpoints对象的新建、更新、删除等操作。
endpoints controller架构图
endpoints controller的大致组成和处理流程如下图，endpoints controller对pod、service对象注册了event handler，当有事件时，会watch到然后将对应的service对象放入到queue中，然后syncService方法为endpoints controller调谐endpoints对象的核心处理逻辑所在，从queue中取出service对象，再查询相应的pod对象列表，然后对endpoints对象做调谐处理。
service对象简介
Service 是对一组提供相同功能的 Pods 的抽象，并为它们提供一个统一的入口。借助 Service，应用可以方便的实现服务发现与负载均衡，并实现应用的零宕机升级。Service 通过标签来选取服务后端，这些匹配标签的 Pod IP 和端口列表组成 endpoints，由 kube-proxy 负责将服务 IP 负载均衡到这些 endpoints 上。
service的四种类型如下。
（1）ClusterIP
类型为ClusterIP的service，这个service有一个Cluster IP，其实就一个VIP，具体实现原理依靠kubeproxy组件，通过iptables或是ipvs实现。该类型的service 只能在集群内访问。
client访问Cluster IP，通过iptables或ipvs规则转到Real Server（endpoints），从而达到负载均衡的效果。
Headless Service
特殊的ClusterIP，通过指定 Cluster IP（spec.clusterIP）的值为 “None” 来创建 Headless Service。
使用场景一：自主选择权，client自行决定使用哪个Real Server，可以通过查询DNS来获取Real Server的信息。
使用场景二：Headless Service的对应的每一个Endpoints，即每一个Pod，都会有对应的DNS域名，这样Pod之间就可以通过域名互相访问（该用法常用于statefulset）。
（2）NodePort
在 ClusterIP 基础上为 Service在每台机器上绑定一个端口，这样就可以通过<NodeIP>:NodePort来访问该服务。在集群内，NodePort 服务仍然像之前的 ClusterIP 服务一样访问。
（3）LoadBalancer
在 NodePort 的基础上，借助 cloud provider 创建一个外部的负载均衡器，并将请求转发到 <NodeIP>:NodePort。
（4）ExternalName
将服务通过 DNS CNAME 记录方式转发到指定的域名。
apiVersion: v1
kind: Service
metadata:
name: baidu-service
namespace: test
spec:
type: ExternalName
externalName: www.baidu.com
endpoints对象简介
endpoints中指定了需要连接的服务IP和端口，可以认为endpoints定义了service的backend后端。当访问service时，实际上是会将请求负载均衡到endpoints定义的服务IP与端口上面去。
另外，endpoints对象与同名称的service对象相关联。
endpoints controller分析将分为两大块进行，分别是：
（1）endpoints controller初始化与启动分析；
（2）endpoints controller处理逻辑分析。
1.endpoints controller初始化与启动分析
基于tag v1.17.4
//github.com/kubernetes/kubernetes/releases/tag/v1.17.4
直接看到startEndpointController函数，作为endpoints controller启动分析的入口。
startEndpointController
在startEndpointController函数中启动了一个goroutine，先是调用了endpointcontroller的NewEndpointController方法初始化endpoints controller，接着调用Run方法启动endpoints controller。
// cmd/kube-controller-manager/app/core.go
func startEndpointController(ctx ControllerContext) (http.Handler, bool, error) {
go endpointcontroller.NewEndpointController(
ctx.InformerFactory.Core().V1().Pods(),
ctx.InformerFactory.Core().V1().Services(),
ctx.InformerFactory.Core().V1().Endpoints(),
ctx.ClientBuilder.ClientOrDie("endpoint-controller"),
ctx.ComponentConfig.EndpointController.EndpointUpdatesBatchPeriod.Duration,
).Run(int(ctx.ComponentConfig.EndpointController.ConcurrentEndpointSyncs), ctx.Stop)
return nil, true, nil
}
1.1 NewEndpointController
先来看到endpoints controller的初始化方法NewEndpointController。
从NewEndpointController函数代码中可以看到，endpoints controller注册了三个informer，分别是podInformer、serviceInformer与endpointsInformer，以及注册了service与pod对象的EventHandler，也即对这2个对象的event进行监听，把event放入事件队列，由endpoints controller的核心处理方法做做处理。
// pkg/controller/endpoint/endpoints_controller.go
func NewEndpointController(podInformer coreinformers.PodInformer, serviceInformer coreinformers.ServiceInformer,
endpointsInformer coreinformers.EndpointsInformer, client clientset.Interface, endpointUpdatesBatchPeriod time.Duration) *EndpointController {
broadcaster := record.NewBroadcaster()
broadcaster.StartLogging(klog.Infof)
broadcaster.StartRecordingToSink(&v1core.EventSinkImpl{Interface: client.CoreV1().Events("")})
recorder := broadcaster.NewRecorder(scheme.Scheme, v1.EventSource{Component: "endpoint-controller"})
if client != nil && client.CoreV1().RESTClient().GetRateLimiter() != nil {
ratelimiter.RegisterMetricAndTrackRateLimiterUsage("endpoint_controller", client.CoreV1().RESTClient().GetRateLimiter())
}
e := &EndpointController{
client: client,
queue: workqueue.NewNamedRateLimitingQueue(workqueue.DefaultControllerRateLimiter(), "endpoint"),
workerLoopPeriod: time.Second,
}
serviceInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{
AddFunc: e.onServiceUpdate,
UpdateFunc: func(old, cur interface{}) {
e.onServiceUpdate(cur)
},
DeleteFunc: e.onServiceDelete,
})
e.serviceLister = serviceInformer.Lister()
e.servicesSynced = serviceInformer.Informer().HasSynced
podInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{
AddFunc: e.addPod,
UpdateFunc: e.updatePod,
DeleteFunc: e.deletePod,
})
e.podLister = podInformer.Lister()
e.podsSynced = podInformer.Informer().HasSynced
e.endpointsLister = endpointsInformer.Lister()
e.endpointsSynced = endpointsInformer.Informer().HasSynced
e.triggerTimeTracker = endpointutil.NewTriggerTimeTracker()
e.eventBroadcaster = broadcaster
e.eventRecorder = recorder
e.endpointUpdatesBatchPeriod = endpointUpdatesBatchPeriod
e.serviceSelectorCache = endpointutil.NewServiceSelectorCache()
return e
}
1.2 Run
主要看到for循环处，根据workers的值（来源于kcm启动参数concurrent-endpoint-syncs配置），启动相应数量的goroutine，跑e.worker方法。
// pkg/controller/endpoint/endpoints_controller.go
func (e *EndpointController) Run(workers int, stopCh <-chan struct{}) {
defer utilruntime.HandleCrash()
defer e.queue.ShutDown()
klog.Infof("Starting endpoint controller")
defer klog.Infof("Shutting down endpoint controller")
if !cache.WaitForNamedCacheSync("endpoint", stopCh, e.podsSynced, e.servicesSynced, e.endpointsSynced) {
return
}
for i := 0; i < workers; i++ {
go wait.Until(e.worker, e.workerLoopPeriod, stopCh)
}
go func() {
defer utilruntime.HandleCrash()
e.checkLeftoverEndpoints()
}()
<-stopCh
}
1.2.1 worker
直接看到processNextWorkItem方法，从队列queue中取出一个key，然后调用e.syncService方法对该key做处理，e.syncService方法也即endpoints controller的核心处理方法，后面会做详细分析。
// pkg/controller/endpoint/endpoints_controller.go
func (e *EndpointController) worker() {
for e.processNextWorkItem() {
}
}
func (e *EndpointController) processNextWorkItem() bool {
eKey, quit := e.queue.Get()
if quit {
return false
}
defer e.queue.Done(eKey)
err := e.syncService(eKey.(string))
e.handleErr(err, eKey)
return true
}
2.endpoints controller核心处理分析
基于tag v1.17.4
//github.com/kubernetes/kubernetes/releases/tag/v1.17.4
直接看到syncService方法，作为endpoints controller核心处理分析的入口。
2.1 核心处理逻辑-syncService
主要逻辑：
（1）获取service对象，当查询不到该service对象时，删除同名endpoints对象；
（2）当service对象的.Spec.Selector为空时，不存在对应的endpoints对象，直接返回；
（3）根据service对象的.Spec.Selector，查询与service对象匹配的pod列表；
（4）查询service的annotations中是否配置了TolerateUnreadyEndpoints，代表允许为unready的pod也创建endpoints，该配置将会影响下面对endpoints对象的subsets信息的计算；
（5）遍历service对象匹配的pod列表，找出合适的pod，计算endpoints的subsets信息；
遍历pod列表时如何计算出subsets？
（5.1）过滤掉pod ip为空的pod；
（5.2）当TolerateUnreadyEndpoints配置为false且pod的deletetimestamp不为空时，过滤掉该pod；
（5.3）当service没有ports配置，且ClusterIP为None时，为headless service，调用addEndpointSubset函数计算subsets，计算出来的subsets中的ports信息为空；
（5.4）当service有ports配置，遍历ports配置，循环调用addEndpointSubset函数计算subsets（addEndpointSubset函数在后面会展开分析）。
（6）获取endpoints对象；
（7）判断现存endpoints对象与调谐中重新计算出来的的endpoints对象的subsets与labels是否一致，一致则无需更新，直接返回；
（8）当endpoints对象不存在时新建endpoints对象，当endpoints对象存在时更新endpoints对象。
func (e *EndpointController) syncService(key string) error {
startTime := time.Now()
defer func() {
klog.V(4).Infof("Finished syncing service %q endpoints. (%v)", key, time.Since(startTime))
}()
namespace, name, err := cache.SplitMetaNamespaceKey(key)
if err != nil {
return err
}
service, err := e.serviceLister.Services(namespace).Get(name)
if err != nil {
if !errors.IsNotFound(err) {
return err
}
// Delete the corresponding endpoint, as the service has been deleted.
// TODO: Please note that this will delete an endpoint when a
// service is deleted. However, if we're down at the time when
// the service is deleted, we will miss that deletion, so this
// doesn't completely solve the problem. See #6877.
err = e.client.CoreV1().Endpoints(namespace).Delete(name, nil)
if err != nil && !errors.IsNotFound(err) {
return err
}
e.triggerTimeTracker.DeleteService(namespace, name)
return nil
}
if service.Spec.Selector == nil {
// services without a selector receive no endpoints from this controller;
// these services will receive the endpoints that are created out-of-band via the REST API.
return nil
}
klog.V(5).Infof("About to update endpoints for service %q", key)
pods, err := e.podLister.Pods(service.Namespace).List(labels.Set(service.Spec.Selector).AsSelectorPreValidated())
if err != nil {
// Since we're getting stuff from a local cache, it is
// basically impossible to get this error.
return err
}
// If the user specified the older (deprecated) annotation, we have to respect it.
tolerateUnreadyEndpoints := service.Spec.PublishNotReadyAddresses
if v, ok := service.Annotations[TolerateUnreadyEndpointsAnnotation]; ok {
b, err := strconv.ParseBool(v)
if err == nil {
tolerateUnreadyEndpoints = b
} else {
utilruntime.HandleError(fmt.Errorf("Failed to parse annotation %v: %v", TolerateUnreadyEndpointsAnnotation, err))
}
}
// We call ComputeEndpointLastChangeTriggerTime here to make sure that the
// state of the trigger time tracker gets updated even if the sync turns out
// to be no-op and we don't update the endpoints object.
endpointsLastChangeTriggerTime := e.triggerTimeTracker.
ComputeEndpointLastChangeTriggerTime(namespace, service, pods)
subsets := []v1.EndpointSubset{}
var totalReadyEps int
var totalNotReadyEps int
for _, pod := range pods {
if len(pod.Status.PodIP) == 0 {
klog.V(5).Infof("Failed to find an IP for pod %s/%s", pod.Namespace, pod.Name)
continue
}
if !tolerateUnreadyEndpoints && pod.DeletionTimestamp != nil {
klog.V(5).Infof("Pod is being deleted %s/%s", pod.Namespace, pod.Name)
continue
}
ep, err := podToEndpointAddressForService(service, pod)
if err != nil {
// this will happen, if the cluster runs with some nodes configured as dual stack and some as not
// such as the case of an upgrade..
klog.V(2).Infof("failed to find endpoint for service:%v with ClusterIP:%v on pod:%v with error:%v", service.Name, service.Spec.ClusterIP, pod.Name, err)
continue
}
epa := *ep
if endpointutil.ShouldSetHostname(pod, service) {
epa.Hostname = pod.Spec.Hostname
}
// Allow headless service not to have ports.
if len(service.Spec.Ports) == 0 {
if service.Spec.ClusterIP == api.ClusterIPNone {
subsets, totalReadyEps, totalNotReadyEps = addEndpointSubset(subsets, pod, epa, nil, tolerateUnreadyEndpoints)
// No need to repack subsets for headless service without ports.
}
} else {
for i := range service.Spec.Ports {
servicePort := &service.Spec.Ports[i]
portName := servicePort.Name
portProto := servicePort.Protocol
portNum, err := podutil.FindPort(pod, servicePort)
if err != nil {
klog.V(4).Infof("Failed to find port for service %s/%s: %v", service.Namespace, service.Name, err)
continue
}
var readyEps, notReadyEps int
epp := &v1.EndpointPort{Name: portName, Port: int32(portNum), Protocol: portProto}
subsets, readyEps, notReadyEps = addEndpointSubset(subsets, pod, epa, epp, tolerateUnreadyEndpoints)
totalReadyEps = totalReadyEps + readyEps
totalNotReadyEps = totalNotReadyEps + notReadyEps
}
}
}
subsets = endpoints.RepackSubsets(subsets)
// See if there's actually an update here.
currentEndpoints, err := e.endpointsLister.Endpoints(service.Namespace).Get(service.Name)
if err != nil {
if errors.IsNotFound(err) {
currentEndpoints = &v1.Endpoints{
ObjectMeta: metav1.ObjectMeta{
Name: service.Name,
Labels: service.Labels,
},
}
} else {
return err
}
}
createEndpoints := len(currentEndpoints.ResourceVersion) == 0
if !createEndpoints &&
apiequality.Semantic.DeepEqual(currentEndpoints.Subsets, subsets) &&
apiequality.Semantic.DeepEqual(currentEndpoints.Labels, service.Labels) {
klog.V(5).Infof("endpoints are equal for %s/%s, skipping update", service.Namespace, service.Name)
return nil
}
newEndpoints := currentEndpoints.DeepCopy()
newEndpoints.Subsets = subsets
newEndpoints.Labels = service.Labels
if newEndpoints.Annotations == nil {
newEndpoints.Annotations = make(map[string]string)
}
if !endpointsLastChangeTriggerTime.IsZero() {
newEndpoints.Annotations[v1.EndpointsLastChangeTriggerTime] =
endpointsLastChangeTriggerTime.Format(time.RFC3339Nano)
} else { // No new trigger time, clear the annotation.
delete(newEndpoints.Annotations, v1.EndpointsLastChangeTriggerTime)
}
if newEndpoints.Labels == nil {
newEndpoints.Labels = make(map[string]string)
}
if !helper.IsServiceIPSet(service) {
newEndpoints.Labels = utillabels.CloneAndAddLabel(newEndpoints.Labels, v1.IsHeadlessService, "")
} else {
newEndpoints.Labels = utillabels.CloneAndRemoveLabel(newEndpoints.Labels, v1.IsHeadlessService)
}
klog.V(4).Infof("Update endpoints for %v/%v, ready: %d not ready: %d", service.Namespace, service.Name, totalReadyEps, totalNotReadyEps)
if createEndpoints {
// No previous endpoints, create them
_, err = e.client.CoreV1().Endpoints(service.Namespace).Create(newEndpoints)
} else {
// Pre-existing
_, err = e.client.CoreV1().Endpoints(service.Namespace).Update(newEndpoints)
}
if err != nil {
if createEndpoints && errors.IsForbidden(err) {
// A request is forbidden primarily for two reasons:
// 1. namespace is terminating, endpoint creation is not allowed by default.
// 2. policy is misconfigured, in which case no service would function anywhere.
// Given the frequency of 1, we log at a lower level.
klog.V(5).Infof("Forbidden from creating endpoints: %v", err)
// If the namespace is terminating, creates will continue to fail. Simply drop the item.
if errors.HasStatusCause(err, v1.NamespaceTerminatingCause) {
return nil
}
}
if createEndpoints {
e.eventRecorder.Eventf(newEndpoints, v1.EventTypeWarning, "FailedToCreateEndpoint", "Failed to create endpoint for service %v/%v: %v", service.Namespace, service.Name, err)
} else {
e.eventRecorder.Eventf(newEndpoints, v1.EventTypeWarning, "FailedToUpdateEndpoint", "Failed to update endpoint %v/%v: %v", service.Namespace, service.Name, err)
}
return err
}
return nil
}
2.1.1 addEndpointSubset
下面来展开分析下计算service对象subsets信息的函数addEndpointSubset，计算出的subsets包括了Address（ReadyAddresses）与NotReadyAddresses。
主要逻辑：
（1）当配置了tolerateUnreadyEndpoints且为true时，或pod处于ready状态时，将计算进subsets中的Addresses；
（2）当配置了tolerateUnreadyEndpoints且为false或没有配置时，或pod不处于ready状态时，调用shouldPodBeInEndpoints函数，返回true时将计算进subsets中的NotReadyAddresses。
（2.1）当pod.Spec.RestartPolicy为Never，Pod Status.Phase不为Failed/Successed时，将计算进subsets中的NotReadyAddresses；
（2.2）当pod.Spec.RestartPolicy为OnFailure, Pod Status.Phase不为Successed时，Pod对应的EndpointAddress也会被加入到NotReadyAddresses中；
（2.3）其他情况下，将计算进subsets中的NotReadyAddresses。
// pkg/controller/endpoint/endpoints_controller.go
func addEndpointSubset(subsets []v1.EndpointSubset, pod *v1.Pod, epa v1.EndpointAddress,
epp *v1.EndpointPort, tolerateUnreadyEndpoints bool) ([]v1.EndpointSubset, int, int) {
var readyEps int
var notReadyEps int
ports := []v1.EndpointPort{}
if epp != nil {
ports = append(ports, *epp)
}
if tolerateUnreadyEndpoints || podutil.IsPodReady(pod) {
subsets = append(subsets, v1.EndpointSubset{
Addresses: []v1.EndpointAddress{epa},
Ports: ports,
})
readyEps++
} else if shouldPodBeInEndpoints(pod) {
klog.V(5).Infof("Pod is out of service: %s/%s", pod.Namespace, pod.Name)
subsets = append(subsets, v1.EndpointSubset{
NotReadyAddresses: []v1.EndpointAddress{epa},
Ports: ports,
})
notReadyEps++
}
return subsets, readyEps, notReadyEps
}
func shouldPodBeInEndpoints(pod *v1.Pod) bool {
switch pod.Spec.RestartPolicy {
case v1.RestartPolicyNever:
return pod.Status.Phase != v1.PodFailed && pod.Status.Phase != v1.PodSucceeded
case v1.RestartPolicyOnFailure:
return pod.Status.Phase != v1.PodSucceeded
default:
return true
}
}
IsPodReady
当在pod的.status.conditions中，type为Ready的status属性值为True时，IsPodReady返回true。
// pkg/api/v1/pod/util.go
// IsPodReady returns true if a pod is ready; false otherwise.
func IsPodReady(pod *v1.Pod) bool {
return IsPodReadyConditionTrue(pod.Status)
}
// GetPodReadyCondition extracts the pod ready condition from the given status and returns that.
// Returns nil if the condition is not present.
func GetPodReadyCondition(status v1.PodStatus) *v1.PodCondition {
_, condition := GetPodCondition(&status, v1.PodReady)
return condition
}
总结
endpoints controller架构图
endpoints controller的大致组成和处理流程如下图，endpoints controller对pod、service对象注册了event handler，当有事件时，会watch到然后将对应的service对象放入到queue中，然后syncService方法为endpoints controller调谐endpoints对象的核心处理逻辑所在，从queue中取出service对象，再查询相应的pod对象列表，然后对endpoints对象做调谐处理。
endpoints controller核心处理逻辑
endpoints controller的核心处理逻辑是获取service对象，当service不存在时删除同名endpoints对象，当存在时，根据service对象所关联的pod列表，计算出endpoints对象的最新subsets信息，然后新建或更新endpoints对象。
