thanos內核
- 2019 年 12 月 29 日
- 筆記
設計
組件
- Sidecar: StoreAPI：查詢 Prometheus 數據；Shipper 上傳數據到（對象）存儲
- Store Gateway: StoreAPI：查詢存儲在（對象）存儲上的指標數據.
- Compactor: 對對象存儲中的數據進行：壓縮，降取樣，期限管理 (retention).
- Receiver: 從 Prometheus 的 remote-write WAL 獲取數據，暴露或者上傳到（對象）存儲 （圖中未畫出，是目前還在 beta 狀態的實現，用Prometheus的 remote-write 介面實現）.
- Ruler/Rule: 和 Prometheus中的 recording/alerting rules 相同，只不過目標數據是 Thanos中的數據，可以暴露查詢或上傳.
- Querier/Query: 實現 Prometheus v1 API，他是一個 API 聚合層，查詢底層數據源，並匯總去重.
上面的組件又可以按照作用分成三類：
- Metric Sources
- Stores
- Queriers
Metric Sources
產生數據，目前只有兩種 Prometheus sidecar 和 rule nodes.
┌────────────┬─────────┐ ┌────────────┬─────────┐ ┌─────────┐ │ Prometheus │ Sidecar │ ... │ Prometheus │ Sidecar │ │ Rule │ └────────────┴────┬────┘ └────────────┴────┬────┘ └┬────────┘ │ │ │ Blocks Blocks Blocks │ │ │ v v v ┌──────────────────────────────────────────────────┐ │ Object Storage │ └──────────────────────────────────────────────────┘
Stores
這裡 store 指 store gateway，轉換用戶的 metrics 請求為 對象存儲 API，It implements various strategies to minimize the number of requests to the object storage such as filtering relevant blocks by their metadata (e.g. time range and labels) and caching frequent index lookups.
Store 和 Data Sources 一樣, 實現了相同的 gRPC Store API，所以對 client 來說，他們是一樣的，無需特殊處理。同時各種 Store API 的實現者會提供他們存儲數據的
meta information, 這使得客戶端可以最小化他們的查詢目標。
┌──────────────────────┐ ┌────────────┬─────────┐ ┌────────────┐ │ Google Cloud Storage │ │ Prometheus │ Sidecar │ │ Rule │ └─────────────────┬────┘ └────────────┴────┬────┘ └─┬──────────┘ │ │ │ Block File Ranges │ │ │ Store API │ v │ │ ┌──────────────┐ │ │ │ Store │ │ Store API └────────┬─────┘ │ │ │ │ │ Store API │ │ │ │ │ v v v ┌──────────────────────────────────┐ │ Client │ └──────────────────────────────────┘
Query Layer
無狀態，自動發現 store，查詢數據。Based on the metadata of store and source nodes, they attempt to minimize the request fanout to fetch data for a particular query.
┌──────────────────┐ ┌────────────┬─────────┐ ┌────────────┐ │ Store Node │ │ Prometheus │ Sidecar │ │ Rule │ └─────────────┬────┘ └────────────┴────┬────┘ └─┬──────────┘ │ │ │ │ │ │ │ │ │ v v v ┌─────────────────────────────────────────────────────┐ │ Query layer │ └─────────────────────────────────────────────────────┘ ^ ^ ^ │ │ │ ┌────────┴────────┐ ┌──────┴─────┐ ┌────┴───┐ │ Alert Component │ │ Dashboards │ ... │ Web UI │ └─────────────────┘ └────────────┘ └────────┘
Compactor
Compactor 是一個可選的單獨組件，不參與 Thanos 集群
實現
各種組件以 thanos 子命令的方式提供
// /thanos-io/thanos/cmd/thanos/main.go registerSidecar(cmds, app) registerStore(cmds, app) registerQuery(cmds, app) registerRule(cmds, app) registerCompact(cmds, app) registerBucket(cmds, app, "bucket") registerDownsample(cmds, app) registerReceive(cmds, app) registerChecks(cmds, app, "check") // 使用的第三方庫 // github.com/golang/snappy - snappy 壓縮演算法s // github.com/hashicorp/serf - Gossip-based Membership // github.com/prometheus/prometheus - 指標,version,發現,promql等等 // google.golang.org/grpc - gprc // github.com/go-kit/kit - log, sd, transport, ratelimit 等等 // github.com/oklog/run - goroutine // gopkg.in/alecthomas/kingpin.v2 - cmd工具 // go.uber.org/automaxprocs/maxprocs - maxproc設置小工具 // github.com/opentracing/opentracing-go - tracing // gopkg.in/check.v1；onsi/ginkgo；onsi/gomega；smartystreets/goconvey - testing // github.com/fsnotify/fsnotify // github.com/fortytw2/leaktest - leak test // github.com/fatih/structtag - parsing and manipulating struct tag fields // github.com/hashicorp/golang-lru // github.com/mwitkow/go-conntrack - Go middleware for net.Conn tracking (Prometheus/trace)
Bucket
對（對象）存儲中的 metrics 數據進行 ls/inspect/verify/repair
➜ ./thanos bucket --objstore.config-file=`pwd`/cos.yaml inspect level=info ts=2019-12-29T08:43:27.744236Z caller=main.go:149 msg="Tracing will be disabled" level=info ts=2019-12-29T08:43:27.744752Z caller=factory.go:43 msg="loading bucket configuration" | ULID | FROM | UNTIL | RANGE | UNTIL-DOWN | #SERIES | #SAMPLES | #CHUNKS | COMP-LEVEL | COMP-FAILED | LABELS | RESOLUTION | SOURCE | |----------------------------|---------------------|---------------------|--------|------------|---------|-----------|---------|------------|-------------|---------------------------------------------------------------------------------|------------|---------| | 01DWQ7AVJAST6B734CCBWJYXV0 | 22-12-2019 20:00:00 | 22-12-2019 22:00:00 | 2h0m0s | 38h0m0s | 36,534 | 1,858,180 | 36,534 | 1 | false | prometheus=monitoring/prometheus-2,prometheus_replica=prometheus-prometheus-2-0 | 0s | sidecar | | 01DWQ8TA11YH5AZ592JC83HGXP | 22-12-2019 22:00:00 | 23-12-2019 00:00:00 | 2h0m0s | 38h0m0s | 38,277 | 8,785,813 | 74,002 | 1 | false | prometheus=monitoring/prometheus-2,prometheus_replica=prometheus-prometheus-2-0 | 0s | sidecar |
Check
tools for validation of Prometheus rules.
Compact
壓縮，降取樣，retention 工具，非並發安全，需要單獨部署；需要本地盤存儲中間數據
- creating 5m downsampling for blocks larger than 40 hours (2d, 2w)
- creating 1h downsampling for blocks larger than 10 days (2w).
- downsampling 不是為了節約空間，二是為了提高查詢效率 (不會減少空間，反而對一個 raw block會增加兩個 block)
- compact 內部使用
tsdb.NewLeveledCompactor進行壓縮 - downsample 是降低取樣
thanos-io/thanos/pkg/compact/downsample/downsample.go
# compact / downsample 後 ➜ ./thanos bucket --objstore.config-file=`pwd`/cos.yaml inspect level=info ts=2019-12-29T09:02:54.011068Z caller=main.go:149 msg="Tracing will be disabled" level=info ts=2019-12-29T09:02:54.011681Z caller=factory.go:43 msg="loading bucket configuration" | ULID | FROM | UNTIL | RANGE | UNTIL-DOWN | #SERIES | #SAMPLES | #CHUNKS | COMP-LEVEL | COMP-FAILED | LABELS | RESOLUTION | SOURCE | |----------------------------|---------------------|---------------------|--------|------------|---------|------------|---------|------------|-------------|---------------------------------------------------------------------------------|------------|-----------| | 01DX8E6DKHG0TEJQG85CPXEEEB | 22-12-2019 20:00:00 | 23-12-2019 00:00:00 | 4h0m0s | 36h0m0s | 38,562 | 10,643,993 | 110,536 | 2 | false | prometheus=monitoring/prometheus-2,prometheus_replica=prometheus-prometheus-2-0 | 0s | compactor | | 01DWQ97RAVF6ADC47073366595 | 22-12-2019 22:00:00 | 23-12-2019 00:00:00 | 2h0m0s | 38h0m0s | 36,616 | 8,229,587 | 36,664 | 1 | false | prometheus=monitoring/prometheus-1,prometheus_replica=prometheus-prometheus-1-0 | 0s | sidecar | | 01DWQFP1915WVACNP2W187QWJ6 | 23-12-2019 00:00:00 | 23-12-2019 02:00:00 | 2h0m0s | 38h0m0s | 36,625 | 8,793,602 | 73,282 | 1 | false | prometheus=monitoring/prometheus-2,prometheus_replica=prometheus-prometheus-2-0 | 0s | sidecar | | 01DX8E0MV53DH41VWNW4TJXXKN | 23-12-2019 00:00:00 | 23-12-2019 08:00:00 | 8h0m0s | 32h0m0s | 36,625 | 35,169,407 | 293,112 | 2 | false | prometheus=monitoring/prometheus-1,prometheus_replica=prometheus-prometheus-1-0 | 0s | compactor |
Query
- query 是 thanos 中核心組件之一，從
StoreAPIs收集數據，實現Prometheus HTTP v1 API返回給 client. - 他的最主要的作用是
提供全局視圖(Global View), 和某些 exporter 的作用很類似，只不過他是在查詢層，而不是在存儲層做聚合 - 另一個作用是
Run-time deduplication of HA group, prometheus可以起多個，作為一個高可用架構，由 query 做數據的聚合和去重，（比如一個 prometheus 有宕機，query可以fill gap） - 數據源支援:
- Prometheus (Sidecar)
- Object Storage (Store Gateway)
- Global alerting/recording rules evaluations (Ruler)
- Metrics received from Prometheus remote write streams (Thanos Receiver)
- Another Querier (you can stack Queriers on top of each other)
- Non-Prometheus systems. e.g OpenTSDB
- 筆者todo: 這裡我們可以實現雲上監控的 adapter, 作為 store api 的一種實現
- 除了compatible with Prometheus 2.x. API 之外增加了支援
- partial response behaviour 當其中一個 StoreAPI 出錯或者超時的時候返回 warnings
- several additional parameters listed below
- custom response fields.
- 實現：
- 有 grpc 的 (proxy) store api, 會去發現的 store查詢 （storeMatches 時間+labelSetsMatch）
- Http 查詢 api, 創建查詢 engine，最後會形成 grpc 查詢請求，查詢本地的 grpc api
- dns provider：dns 發現 store
Store
在 （對象）存儲上實現 store api, 本地會定期同步 metadata, index. 實現在 bucket.go裡面，同時實現了 store的 grpc service hanos-io/thanos/pkg/store/storepb/rpc.proto.
// BucketStore implements the store API backed by a bucket. It loads all index // files to local disk. type BucketStore struct { // ...略 indexCache storecache.IndexCache // Sets of blocks that have the same labels. They are indexed by a hash over their label set. blocks map[ulid.ULID]*bucketBlock blockSets map[uint64]*bucketBlockSet // samplesLimiter limits the number of samples per each Series() call. samplesLimiter *Limiter partitioner partitioner } // Series implements the storepb.StoreServer interface. // 省略了非核心程式碼, 本地有 cache(posting, series), 先查本地，查遠程，存本地cache func (s *BucketStore) Series(req *storepb.SeriesRequest, srv storepb.Store_SeriesServer) (err error) { // Concurrently get data from all blocks. for _, bs := range s.blockSets { blockMatchers, ok := bs.labelMatchers(matchers...) blocks := bs.getFor(req.MinTime, req.MaxTime, req.MaxResolutionWindow) for _, b := range blocks { // We must keep the readers open until all their data has been sent. indexr := b.indexReader(ctx) chunkr := b.chunkReader(ctx) g.Go(func() error { part, pstats, err := blockSeries(ctx, b.meta.ULID, b.meta.Thanos.Labels, indexr, chunkr, blockMatchers, req, s.samplesLimiter) res = append(res, part) return nil }) } } // Merge the sub-results from each selected block. { // Merge series set into an union of all block sets. This exposes all blocks are single seriesSet. // Chunks of returned series might be out of order w.r.t to their time range. // This must be accounted for later by clients. set := storepb.MergeSeriesSets(res...) for set.Next() { var series storepb.Series series.Labels, series.Chunks = set.At() stats.mergedSeriesCount++ stats.mergedChunksCount += len(series.Chunks) if err := srv.Send(storepb.NewSeriesResponse(&series)); err != nil { return status.Error(codes.Unknown, errors.Wrap(err, "send series response").Error()) } } } return nil }Time based partitioning
- store 默認會根據對象存儲中的時間範圍提供 metrcis; 但是也可以用
--min-time,--max-time設置提供metrics的時間範圍。 - 新數據不一定能馬上被查詢到，Time partitioning 每3分鐘同步一次
- 建議 Thanos Store gateways 中的時間範圍和Thanos Sidecar 由一定重合，以預防失敗
Rule
You can think of Rule as a simplified Prometheus that does not require a sidecar and does not scrape and do PromQL evaluation
Sidecar
作用
- 實現 Store API 直接查詢 promethues 數據
- 可選的每2小時上傳 tsdb數據，這樣promethues的 retention 可以設置得很短
- 這裡的 Store API 實現剛好和 Query 相反，這裡是把 grpc請求轉換成 promethues 的http 請求
- Shipper 實現在
thanos-io/thanos/pkg/shipper/shipper.go, 把每二小時形成的 block/meta.. 進行上傳
Receiver
實現 promethues 的remote write 介面
參考
- 官方文檔
- 一個很詳細的介紹
- 上篇文章的翻譯，翻譯得不太好
- 一篇中文介紹
- 另一篇中文介紹
- Comparing Thanos to VictoriaMetrics cluster, VictoriaMetrics也是一個方案，不過使用磁碟存儲
- 另一個方案，uber開源
- thanos在bucket很大的時候可能會有性能問題
