kubernetes垃圾回收器GarbageCollector Controller源碼分析(二)
- 2019 年 10 月 12 日
- 筆記
kubernetes版本:1.13.2
接上一節:kubernetes垃圾回收器GarbageCollector Controller源碼分析(一)
主要步驟
GarbageCollector Controller源碼主要分為以下幾部分:
monitors
作為生產者將變化的資源放入graphChanges
隊列;同時restMapper
定期檢測集群內資源類型,刷新monitors
runProcessGraphChanges
從graphChanges
隊列中取出變化的item
,根據情況放入attemptToDelete
隊列;runProcessGraphChanges
從graphChanges
隊列中取出變化的item
,根據情況放入attemptToOrphan
隊列;runAttemptToDeleteWorker
從attemptToDelete
隊列取出,嘗試刪除垃圾資源;runAttemptToOrphanWorker
從attemptToDelete
隊列取出,處理該孤立的資源;
程式碼較複雜,便於講的更清楚,調整了下講解順序。上一節分析了第1部分,本節分析第2、3部分。
runProcessGraphChanges處理主流程
來到源碼k8s.iokubernetespkgcontrollergarbagecollectorgraph_builder.go中,runProcessGraphChanges中一直死循環處理變化的資源對象:
func (gb *GraphBuilder) runProcessGraphChanges() { for gb.processGraphChanges() { } }
一個協程一直循環從graphChanges隊列中獲取變化的資源對象,更新圖形,填充dirty_queue。(graphChanges隊列里數據來源於各個資源的monitors監聽資源變化回調addFunc、updateFunc、deleteFunc)
// Dequeueing an event from graphChanges, updating graph, populating dirty_queue. //從graphChanges中獲取事件,更新圖形,填充dirty_queue。(graphChanges隊列里數據來源於各個資源的monitors監聽資源變化回調addFunc、updateFunc、deleteFunc) func (gb *GraphBuilder) processGraphChanges() bool { item, quit := gb.graphChanges.Get() if quit { return false } defer gb.graphChanges.Done(item) event, ok := item.(*event) if !ok { utilruntime.HandleError(fmt.Errorf("expect a *event, got %v", item)) return true } obj := event.obj //獲取該變化資源obj的accessor accessor, err := meta.Accessor(obj) if err != nil { utilruntime.HandleError(fmt.Errorf("cannot access obj: %v", err)) return true } klog.V(5).Infof("GraphBuilder process object: %s/%s, namespace %s, name %s, uid %s, event type %v", event.gvk.GroupVersion().String(), event.gvk.Kind, accessor.GetNamespace(), accessor.GetName(), string(accessor.GetUID()), event.eventType) // Check if the node already exists // 檢查節點是否已存在 //根據該變化資源obj的UID //uidToNode維護著資源對象依賴關係圖表結構 existingNode, found := gb.uidToNode.Read(accessor.GetUID()) if found { // this marks the node as having been observed via an informer event // 1. this depends on graphChanges only containing add/update events from the actual informer // 2. this allows things tracking virtual nodes' existence to stop polling and rely on informer events //這標誌著節點已經通過informer事件 // 1.進行了觀察。這取決於僅包含來自實際informer的添加/更新事件的graphChange // 2.這允許跟蹤虛擬節點的存在以停止輪詢和依賴informer事件 existingNode.markObserved() } switch { //gc第一次運行時,uidToNode尚且沒有初始化資源對象依賴關係圖表結構,所以found為false,會新增節點 case (event.eventType == addEvent || event.eventType == updateEvent) && !found: newNode := &node{ identity: objectReference{ OwnerReference: metav1.OwnerReference{ APIVersion: event.gvk.GroupVersion().String(), Kind: event.gvk.Kind, UID: accessor.GetUID(), Name: accessor.GetName(), }, Namespace: accessor.GetNamespace(), }, dependents: make(map[*node]struct{}), owners: accessor.GetOwnerReferences(), deletingDependents: beingDeleted(accessor) && hasDeleteDependentsFinalizer(accessor), beingDeleted: beingDeleted(accessor), } gb.insertNode(newNode) // the underlying delta_fifo may combine a creation and a deletion into // one event, so we need to further process the event. //底層delta_fifo可以將創建和刪除組合成一個事件,因此我們需要進一步處理事件。 gb.processTransitions(event.oldObj, accessor, newNode) //uidToNode已經初始化資源對象依賴關係圖表結構,所以found為true case (event.eventType == addEvent || event.eventType == updateEvent) && found: // handle changes in ownerReferences //處理ownerReferences中的更改 added, removed, changed := referencesDiffs(existingNode.owners, accessor.GetOwnerReferences()) if len(added) != 0 || len(removed) != 0 || len(changed) != 0 { // check if the changed dependency graph unblock owners that are // waiting for the deletion of their dependents. //檢查更改的依賴關係圖是否取消阻止等待刪除其依賴項的所有者。 gb.addUnblockedOwnersToDeleteQueue(removed, changed) // update the node itself //更新node的owner existingNode.owners = accessor.GetOwnerReferences() // Add the node to its new owners' dependent lists. //給新owner添加依賴資源列表 gb.addDependentToOwners(existingNode, added) // remove the node from the dependent list of node that are no longer in // the node's owners list. //從不再屬於該資源owner列表中刪除該節點。 gb.removeDependentFromOwners(existingNode, removed) } // 該對象正在被刪除中 if beingDeleted(accessor) { existingNode.markBeingDeleted() } gb.processTransitions(event.oldObj, accessor, existingNode) //處理資源對象被刪除的場景,涉及垃圾。比如,owner被刪除,其依賴資源(從資源)也需要被刪除掉,除非設置了Orphan case event.eventType == deleteEvent: if !found { klog.V(5).Infof("%v doesn't exist in the graph, this shouldn't happen", accessor.GetUID()) return true } // 從圖標中移除item資源,同時遍歷owners,移除owner下的item資源 gb.removeNode(existingNode) existingNode.dependentsLock.RLock() defer existingNode.dependentsLock.RUnlock() //如果該資源的從資源數大於0,則將該資源被刪除資訊加入absentOwnerCache快取 if len(existingNode.dependents) > 0 { gb.absentOwnerCache.Add(accessor.GetUID()) } //遍歷該資源的從資源加到刪除隊列里 for dep := range existingNode.dependents { gb.attemptToDelete.Add(dep) } for _, owner := range existingNode.owners { ownerNode, found := gb.uidToNode.Read(owner.UID) //owner沒發現 或者 owner的從資源不是正在被刪除(只有該資源對象的終結器為foregroundDeletion Finalizer時deletingDependents被設為true,因為後台刪除owner直接被刪除,不會被其從資源block,故這裡都不需要去嘗試刪除owner了) if !found || !ownerNode.isDeletingDependents() { continue } // 這是讓attempToDeleteItem檢查是否刪除了owner的依賴項,如果是,則刪除所有者。 gb.attemptToDelete.Add(ownerNode) } } return true }
該方法功能主要將對象、owner、從資源加入到attemptToDelete或attemptToOrphan。
1、 出隊
從graphChanges隊列取出資源對象,從GraphBuilder.uidToNode中讀取該資源節點(uidToNode維護著資源對象依賴關係圖表結構),found為true時表示圖表存在該資源節點;
2、switch的第一個case
如果該資源是新增或者更新觸發,且該資源對象不存在於圖表中,gb.uidToNode.Write(n)會將其寫入圖標;
gb.insertNode(newNode)中的gb.addDependentToOwners(n, n.owners)方法則會遍歷該資源的owner,如果其owner不存在於圖標中,則新增owner的虛擬節點到圖標中,並將該資源和owner產生關聯。如果owner不存在時,則嘗試將owner加入到attemptToDelete隊列中去;
// addDependentToOwners將n添加到所有者的從屬列表中。如果所有者不存在於gb.uidToNode中,則將創建"虛擬"節點以表示 // 所有者。 "虛擬"節點將入隊到attemptToDelete,因此 // attemptToDeleteItem()將根據API伺服器驗證所有者是否存在。 func (gb *GraphBuilder) addDependentToOwners(n *node, owners []metav1.OwnerReference) { //遍歷owner for _, owner := range owners { //獲取owner node如果不存在於圖中,則加虛擬owner節點 ownerNode, ok := gb.uidToNode.Read(owner.UID) if !ok { // Create a "virtual" node in the graph for the owner if it doesn't // exist in the graph yet. //如果圖形中尚未存在,則在圖表中為所有者創建「虛擬」節點。 ownerNode = &node{ identity: objectReference{ OwnerReference: owner, Namespace: n.identity.Namespace, }, dependents: make(map[*node]struct{}), virtual: true, } klog.V(5).Infof("add virtual node.identity: %snn", ownerNode.identity) gb.uidToNode.Write(ownerNode) } //給owner加該資源作為依賴 ownerNode.addDependent(n) //owner不存在於圖中時,才往刪除隊列添加 if !ok { // Enqueue the virtual node into attemptToDelete. // The garbage processor will enqueue a virtual delete // event to delete it from the graph if API server confirms this // owner doesn't exist. //將虛擬節點排入attemptToDelete。 // 如果API伺服器確認owner不存在,垃圾處理器將排隊虛擬刪除事件以將其從圖中刪除。 gb.attemptToDelete.Add(ownerNode) } } }
gb.processTransitions方法:
新item正在被刪,舊item沒開始被刪除,且終結器為Orphan Finalizer加入到attemptToOrphan隊列;
新item正在被刪,舊item沒開始被刪除,且終結器為foregroundDeletion Finalizer,則加入到attemptToDelete隊列。
func (gb *GraphBuilder) processTransitions(oldObj interface{}, newAccessor metav1.Object, n *node) { //新的正在被刪,舊的沒開始被刪除,且終結器為Orphan Finalizer if startsWaitingForDependentsOrphaned(oldObj, newAccessor) { klog.V(5).Infof("add %s to the attemptToOrphan", n.identity) //加入到Orphan隊列 gb.attemptToOrphan.Add(n) return } //新的正在被刪,舊的沒開始被刪除,且終結器為foregroundDeletion Finalizer if startsWaitingForDependentsDeleted(oldObj, newAccessor) { klog.V(2).Infof("add %s to the attemptToDelete, because it's waiting for its dependents to be deleted", n.identity) // if the n is added as a "virtual" node, its deletingDependents field is not properly set, so always set it here. n.markDeletingDependents() for dep := range n.dependents { gb.attemptToDelete.Add(dep) } gb.attemptToDelete.Add(n) } }
3、switch的第二個case
如果該資源是新增或者更新觸發,且該資源對象存在於圖表中。對比owneReferences是否有變更,referencesDiffs方法里會根據uid對比,added表示新owner里有,舊owner里沒有的, removed表示舊owner里有,新owner里沒有的, changed表示相同uid的owner不deepEqual的。
func referencesDiffs(old []metav1.OwnerReference, new []metav1.OwnerReference) (added []metav1.OwnerReference, removed []metav1.OwnerReference, changed []ownerRefPair) { //key為uid, value為OwnerReference oldUIDToRef := make(map[string]metav1.OwnerReference) for _, value := range old { oldUIDToRef[string(value.UID)] = value } oldUIDSet := sets.StringKeySet(oldUIDToRef) //key為uid, value為OwnerReference newUIDToRef := make(map[string]metav1.OwnerReference) for _, value := range new { newUIDToRef[string(value.UID)] = value } newUIDSet := sets.StringKeySet(newUIDToRef) //新的里有,舊的里沒有的為新增(根據uid判斷) addedUID := newUIDSet.Difference(oldUIDSet) //舊的里有,新的里沒有的為刪除(根據uid判斷) removedUID := oldUIDSet.Difference(newUIDSet) //取交集, 舊的和新的里都有的owner(根據uid判斷) intersection := oldUIDSet.Intersection(newUIDSet) for uid := range addedUID { added = append(added, newUIDToRef[uid]) } for uid := range removedUID { removed = append(removed, oldUIDToRef[uid]) } //根據uid判斷,兩個uid相等的OwnerReference是否deepEqual,不等則加到changed for uid := range intersection { if !reflect.DeepEqual(oldUIDToRef[uid], newUIDToRef[uid]) { changed = append(changed, ownerRefPair{oldRef: oldUIDToRef[uid], newRef: newUIDToRef[uid]}) } } return added, removed, changed }
整體來說,owner發生變化,addUnblockedOwnersToDeleteQueue方法會判斷:如果阻塞ownerReference指向某個對象被刪除,或者設置為BlockOwnerDeletion=false
,則將該對象添加到attemptToDelete隊列;
// if an blocking ownerReference points to an object gets removed, or gets set to // "BlockOwnerDeletion=false", add the object to the attemptToDelete queue. //如果阻塞ownerReference指向某個對象被刪除,或者設置為 // "BlockOwnerDeletion = false",則將該對象添加到attemptToDelete隊列。 func (gb *GraphBuilder) addUnblockedOwnersToDeleteQueue(removed []metav1.OwnerReference, changed []ownerRefPair) { for _, ref := range removed { //被移除的OwnersReferences,BlockOwnerDeletion為true if ref.BlockOwnerDeletion != nil && *ref.BlockOwnerDeletion { //依賴圖表中發現,則加入刪除隊列 node, found := gb.uidToNode.Read(ref.UID) if !found { klog.V(5).Infof("cannot find %s in uidToNode", ref.UID) continue } //加入嘗試刪除隊列刪除這個owner gb.attemptToDelete.Add(node) } } // Owners存在且發生變化,舊的BlockOwnerDeletion為true, 新的BlockOwnerDeletion為空或者BlockOwnerDeletion為false則刪除owner(父節點) for _, c := range changed { wasBlocked := c.oldRef.BlockOwnerDeletion != nil && *c.oldRef.BlockOwnerDeletion isUnblocked := c.newRef.BlockOwnerDeletion == nil || (c.newRef.BlockOwnerDeletion != nil && !*c.newRef.BlockOwnerDeletion) if wasBlocked && isUnblocked { node, found := gb.uidToNode.Read(c.newRef.UID) if !found { klog.V(5).Infof("cannot find %s in uidToNode", c.newRef.UID) continue } gb.attemptToDelete.Add(node) } } }
更新node的owner;
在依賴圖表中給新owner添加該node;
在依賴圖表中,被刪除的owner列表下刪除該節點。
gb.processTransitions方法:
新item正在被刪,舊item沒開始被刪除,且終結器為Orphan Finalizer加入到attemptToOrphan隊列;
新item正在被刪,舊item沒開始被刪除,且終結器為foregroundDeletion Finalizer,則加入到attemptToDelete隊列。
4、switch的第三個case
如果該資源是刪除時觸發,從圖表中移除item資源,同時遍歷owners,移除owner下的item資源;
如果該資源的從資源數大於0,則將該資源被刪除資訊(uid)加入absentOwnerCache快取,這樣處理該資源的從資源時,就知道owner不存在了。
遍歷該資源的從資源加到刪除隊列里;
如果從圖表中發現 owner或者 owner的從資源正在被刪除,則嘗試將owner加入到attemptToDelete隊列中,去嘗試刪除owner。
整理流程
- 當controllermanager重啟時,會全量listwatch一遍所有對象,gc collector維護的uidToNode圖表裡各個資源對象node是不存在的,此時會走第一個switch case,構建完整關係圖表,如果owner不存在則先構建虛擬owner節點,同時加入attemptToDelete隊列,嘗試去刪除這個owner,其實即使加入到attemptToDelete隊列,也不一定會被刪除,還會進行一系列判斷,這個下一節再分析;將正在刪除的資源,同時Finalizer為Orphan的加入到attemptToOrphan隊列;為foreground的資源以及其從資源加入到attemptToDelete隊列,並將deletingDependents設置為true;
- 添加或者更新事件時,且圖表中存在item資源對象時,會走第二個switch case,對item的owner變化進行判斷,並維護更新圖表;同理將正在刪除的資源,同時Finalizer為Orphan的加入到attemptToOrphan隊列;Finalizer為foreground的資源以及其從資源加入到attemptToDelete隊列,並將deletingDependents設置為true;
- 如果是刪除事件,則會更新圖表,並處理和其相關的從資源和其owner加入到attemptToDelete隊列。
參考:
k8s官方文檔garbage-collection英文版:
https://kubernetes.io/docs/concepts/workloads/controllers/garbage-collection/
依賴圖標生成庫gonum Api文檔:
https://godoc.org/gonum.org/v1/gonum/graph
graphviz下載:
https://graphviz.gitlab.io/_pages/Download/Download_windows.html
本公眾號免費提供csdn下載服務,海量IT學習資源,如果你準備入IT坑,勵志成為優秀的程式猿,那麼這些資源很適合你,包括但不限於java、go、python、springcloud、elk、嵌入式 、大數據、面試資料、前端 等資源。同時我們組建了一個技術交流群,裡面有很多大佬,會不定時分享技術文章,如果你想來一起學習提高,可以公眾號後台回復【2】,免費邀請加技術交流群互相學習提高,會不定期分享編程IT相關資源。
掃碼關注,精彩內容第一時間推給你