.NET Core + K8S + Loki 玩轉日誌聚合

Grafana loki

1. Intro

最近在了解日誌聚合系統,正好前幾天看到一篇文章《用了日誌系統新貴Loki,ELK突然不香了!》,所以就決定動手體驗一下。本文就帶大家快速了解下Loki,並簡單介紹.NET Core如何集成Loki。

2. What’s Loki Stack

Grafana Loki like Prometheus, but for logs。其是一個水平可擴展,高可用性,多租戶的日誌聚合系統,基於Apatch 2.0開源。其有三部分組成:

  1. Loki 是主服務器,負責存儲日誌和處理查詢。對標ELK中的ElasticSearch。
  2. Promtail 是代理,負責收集日誌並將其發送給loki。對標ELK中的Logstash。
  3. Grafana提供用戶界面。對標ELK中的Kibana。

3. Why Use Loki

日誌聚合系統的目的是為了方便我們進行日誌跟蹤和故障排查,尤其在雲原生的環境之下。目前主流的日誌聚合系統,當數ELK、EFK和Loki。Loki相較於ELK Stack有以下優勢:

  • Elasticsearch中的數據作為非結構化JSON對象存儲在磁盤上,Loki以二進制的形式存儲。

  • Elasticsearch採用全文索引,倒排索引的切分和共享的成本較高。Loki僅索引元數據,比如標籤。

  • 和Prometheus無縫集成。

4. How Use Loki

首先我們先來基於Heml安裝Loki到本地K8S集群。

1. 添加Loki Chart 倉庫:

PS C:\Users\Shengjie> helm repo add loki //grafana.github.io/loki/charts
"loki" has been added to your repositories
PS C:\Users\Shengjie> helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "stable" chart repository
...Successfully got an update from the "loki" chart repository
Update Complete. ⎈ Happy Helming!⎈

2. 安裝Loki Stack

PS C:\Users\Shengjie> helm search hub loki-stack
URL                                             CHART VERSION   APP VERSION     DESCRIPTION
//hub.helm.sh/charts/loki/loki-stack      0.38.3          v1.5.0          Loki: like Prometheus, but for logs.
PS C:\Users\Shengjie> helm show values loki/loki-stack
loki:
  enabled: true

promtail:
  enabled: true

fluent-bit:
  enabled: false

grafana:
  enabled: false
  sidecar:
    datasources:
      enabled: true
  image:
    tag: 6.7.0

prometheus:
  enabled: false

filebeat:
  enabled: false
  filebeatConfig:
    filebeat.yml: |
      # logging.level: debug
      filebeat.inputs:
      - type: container
        paths:
          - /var/log/containers/*.log
        processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"
      output.logstash:
        hosts: ["logstash-loki:5044"]

logstash:
  enabled: false
  image:
    repository: grafana/logstash-output-loki
    tag: 1.0.1
  filters:
    main: |-
      filter {
        if [kubernetes] {
          mutate {
            add_field => {
              "container_name" => "%{[kubernetes][container][name]}"
              "namespace" => "%{[kubernetes][namespace]}"
              "pod" => "%{[kubernetes][pod][name]}"
            }
            replace => { "host" => "%{[kubernetes][node][name]}"}
          }
        }
        mutate {
          remove_field => ["tags"]
        }
      }
  outputs:
    main: |-
      output {
        loki {
          url => "//loki:3100/loki/api/v1/push"
          #username => "test"
          #password => "test"
        }
        # stdout { codec => rubydebug }
      }

從上面的Values中,可以看出,可以自定義啟用fluent-bit、grafana、filebeat、prometheus、logstash組件。這裡我們僅啟用grafana日誌界面。

PS C:\Users\Shengjie> helm install loki-stack loki/loki-stack --set grafana.enabled=true
coalesce.go:165: warning: skipped value for filters: Not a table.
coalesce.go:165: warning: skipped value for filters: Not a table.
NAME: loki-stack
LAST DEPLOYED: Sun Jul 26 11:58:11 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
The Loki stack has been deployed to your cluster. Loki can now be added as a datasource in Grafana.

See //docs.grafana.org/features/datasources/loki/ for more detail.

3. 登錄Grafana
首先確認Loki是否成功部署:

PS C:\Users\Shengjie> helm list
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
loki-stack      default         1               2020-07-26 11:58:11.022896 +0800 CST    deployed        loki-stack-0.38.3       v1.5.0
PS C:\Users\Shengjie> kubectl get pod -w
NAME                                READY   STATUS        RESTARTS   AGE
loki-stack-0                        1/1     Running       0          2m33s
loki-stack-grafana-c447cfbd-z6tbg   1/1     Running       0          2m33s
loki-stack-promtail-j47hl           1/1     Running       0          2m33s
PS C:\Users\Shengjie> kubectl get svc -w
NAME                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
kubernetes            ClusterIP   10.96.0.1       <none>        443/TCP    56d
loki-stack            ClusterIP   10.110.83.209   <none>        3100/TCP   2m52s
loki-stack-grafana    ClusterIP   10.111.24.26    <none>        80/TCP     2m52s
loki-stack-headless   ClusterIP   None            <none>        3100/TCP   2m52s

從上面可知,已經成功啟動,其中loki暴露的容器端口為3100,grafana暴露的端口為80。
因此我們下一步需要進行端口轉發,才能訪問grafana。

PS C:\Users\Shengjie> kubectl port-forward svc/loki-stack-grafana 3000:80
Forwarding from 127.0.0.1:3000 -> 3000
Forwarding from [::1]:3000 -> 3000

然後本地瀏覽器打開//localhost:3000/就可以訪問了。
其默認用戶是admin,默認密碼在哪裡呢,別慌,我們去看下loki-stack chart 的readme:

PS C:\Users\Shengjie> helm show readme loki/loki-stack
# Loki-Stack Helm Chart
## Prerequisites

Make sure you have Helm [installed](//helm.sh/docs/using_helm/#installing-helm) and
[deployed](//helm.sh/docs/using_helm/#installing-tiller) to your cluster. Then add
Loki's chart repository to Helm:
$ helm repo add loki //grafana.github.io/loki/charts

You can update the chart repository by running:
$ helm repo update

## Deploy Loki and Promtail to your cluster
### Deploy with default config
$ helm upgrade --install loki loki/loki-stack
### Deploy in a custom namespace
$ helm upgrade --install loki --namespace=loki-stack loki/loki-stack

### Deploy with custom config

$ helm upgrade --install loki loki/loki-stack --set "key1=val1,key2=val2,..."
## Deploy Loki and Fluent Bit to your cluster
$ helm upgrade --install loki loki/loki-stack \
    --set fluent-bit.enabled=true,promtail.enabled=false
## Deploy Grafana to your cluster
The chart loki-stack contains a pre-configured Grafana, simply use `--set grafana.enabled=true`

To get the admin password for the Grafana pod, run the following command:
$ kubectl get secret --namespace <YOUR-NAMESPACE> loki-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
To access the Grafana UI, run the following command:
$ kubectl port-forward --namespace <YOUR-NAMESPACE> service/loki-grafana 3000:80
Navigate to //localhost:3000 and login with `admin` and the password output above.
Then follow the [instructions for adding the loki datasource](/docs/getting-started/grafana.md), using the URL `//loki:3100/`

上面已經說了很清楚了,可以從secret中獲取。

如果在windows powersheel中執行,需要分兩步:
1. 先獲取base64加密的密碼
PS C:\Users\Shengjie> $pwd= kubectl get secret --namespace default loki-stack-grafana -o jsonpath="{.data.admin-password}"
2. decode base64
PS C:\Users\Shengjie> [Text.Encoding]::Utf8.GetString([Convert]::FromBase64String($pwd))
CjnbkkQmwQynZ96gCsynSf0elYQLOp4dyuDnp9jJ

------------------------
Linux命令行執行:
[email protected]:/mnt/c/Users/Shengjie$ kubectl get secret --namespace default loki-stack-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
CjnbkkQmwQynZ96gCsynSf0elYQLOp4dyuDnp9jJ

然後使用admin/CjnbkkQmwQynZ96gCsynSf0elYQLOp4dyuDnp9jJ即可成功登錄//localhost:3000/

Grafnan Explore

5. Use Loki With .NET Core

下一步,我們就來創建一個ASP.NET Core Web 應用,將日誌記錄到Loki,並通過Grafana進行聚合分析。

PS C:\Users\Shengjie> dotnet new web -n Loki.K8s.Demo
The template "ASP.NET Core Empty" was created successfully.

Processing post-creation actions...
Running 'dotnet restore' on Loki.K8s.Demo\Loki.K8s.Demo.csproj...
  Determining projects to restore...
  Restored C:\Users\Shengjie\Loki.K8s.Demo\Loki.K8s.Demo.csproj (in 150 ms).

Restore succeeded.
PS C:\Users\Shengjie> cd .\Loki.K8s.Demo\
# 添加Serilog.AspNetCore和Serilog.Sinks.Loki Nuget包。
PS C:\Users\Shengjie\Loki.K8s.Demo> dotnet add package Serilog.AspNetCore
PS C:\Users\Shengjie\Loki.K8s.Demo> dotnet add package Serilog.Sinks.Loki

從上可知,日誌組件選用的是Serilog,因為其支持持久化日誌到Loki。
修改Program.cs如下:

public static IHostBuilder CreateHostBuilder(string[] args) =>
    Host.CreateDefaultBuilder(args)
        .ConfigureWebHostDefaults(webBuilder =>
        {
            webBuilder.UseStartup<Startup>();
        }).UseSerilog((ctx, cfg) =>
        {
            //cfg.MinimumLevel.Override("Microsoft", LogEventLevel.Warning);//Microsoft框架本身的日誌,僅輸出Warning以上級別
            cfg.Enrich.FromLogContext()
            .Enrich.WithProperty("App", ctx.HostingEnvironment.ApplicationName)
            .Enrich.WithProperty("ENV", ctx.HostingEnvironment.EnvironmentName)
            .WriteTo.LokiHttp(new NoAuthCredentials("//localhost:3100"))//配置Loki Url和認證方式
            .WriteTo.Console();
        });

修改Startup.csConfigure方法如下:

public void Configure(IApplicationBuilder app, IWebHostEnvironment env,ILogger<Startup> logger)
{
    if (env.IsDevelopment())
    {
        app.UseDeveloperExceptionPage();
    }
    app.UseRouting();
    app.UseEndpoints(endpoints =>
    {
        endpoints.MapGet("/", async context =>
        {
            logger.LogInformation("start handle request!");
            await context.Response.WriteAsync("Hello World!");
            logger.LogInformation("end handle request!");
        });
    });
}

因為不想把應用打包成鏡像運行到K8S中,所以我們需要把K8S的Loki服務做一次端口轉發暴露到本機,轉發後,就可以使用//localhost:3100作為Loki的Url進行日誌寫入啦。

PS C:\Users\Shengjie\Loki.K8s.Demo> kubectl port-forward svc/loki-stack 3100:3100
Forwarding from 127.0.0.1:3100 -> 3100
Forwarding from [::1]:3100 -> 3100

運行項目後,重新打開Grafana,添加過濾條件,就可以查看應用日誌了。

參考資料:

  1. 日誌聚合工具loki
  2. CSDN-linkt1234-Loki
  3. Loki官方文檔