在 Grafana Loki 中用 Logging operator 来访问 Kubernetes pod 日志

原创

philentso

修改于 2022-12-28 17:35:04

1K0

修改于 2022-12-28 17:35:04

文章被收录于专栏：philentso

在 Grafana Loki 中用 Logging operator 来访问 kubernetes pod 日志

通过前一节 Logging operator 基本了解，本节实战部署日志系统来收集容器应用日志！

本示例介绍如何使用 Logging operator 在 Kubernetes 中收集应用程序和容器日志，以及如何将它们发送到 Grafana Loki.

下图概述了系统的工作原理， Logging operator 从应用程序收集日志，选择要转发到输出的日志，并将选定的日志消息发送到输出：

部署 Loki 和 Grafana

Add the chart repositories of Loki and Grafana using the following commands:

$ helm repo add grafana https://grafana.github.io/helm-charts
"grafana" has been added to your repositories

$ helm repo add loki https://grafana.github.io/loki/charts
"loki" has been added to your repositories

$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Unable to get an update from the "testchartmuseum" chart repository (http://192.168.86.129:32155):
        Get "http://192.168.86.129:32155/index.yaml": dial tcp 192.168.86.129:32155: connect: connection refused
...
...Successfully got an update from the "flagger" chart repository
...Successfully got an update from the "argo" chart repository
Update Complete. ⎈Happy Helming!⎈

安装 Loki into the logging namespace:

$ helm upgrade --install --create-namespace --namespace logging loki loki/loki

Release "loki" does not exist. Installing it now.
WARNING: This chart is deprecated
NAME: loki
LAST DEPLOYED: Wed Dec 28 03:34:53 2022
NAMESPACE: logging
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Verify the application is working by running these commands:
  kubectl --namespace logging port-forward service/loki 3100
  curl http://127.0.0.1:3100/api/prom/label


// Grafana Loki Documentation:
https://github.com/grafana/loki/tree/master/production/helm

安装 Grafana into the logging namespace:

$ helm upgrade --install --create-namespace --namespace logging grafana grafana/grafana \
--set "datasources.datasources\\.yaml.apiVersion=1" \
 --set "datasources.datasources\\.yaml.datasources[0].name=Loki" \
 --set "datasources.datasources\\.yaml.datasources[0].type=loki" \
 --set "datasources.datasources\\.yaml.datasources[0].url=http://loki:3100" \
 --set "datasources.datasources\\.yaml.datasources[0].access=proxy"

Release "grafana" does not exist. Installing it now.
NAME: grafana
LAST DEPLOYED: Wed Dec 28 03:47:41 2022
NAMESPACE: logging
STATUS: deployed
REVISION: 1
NOTES:
1. Get your 'admin' user password by running:

   kubectl get secret --namespace logging grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

2. The Grafana server can be accessed via port 80 on the following DNS name from within your cluster:

   grafana.logging.svc.cluster.local

   Get the Grafana URL to visit by running these commands in the same shell:
     export POD_NAME=$(kubectl get pods --namespace logging -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=grafana" -o jsonpath="{.items[0].metadata.name}")
     kubectl --namespace logging port-forward $POD_NAME 3000

3. Login with the password from step 1 and the username: admin
#################################################################################
######   WARNING: Persistence is disabled!!! You will lose your data when   #####
######            the Grafana pod is terminated.                            #####
#################################################################################

验证 loki 与 grafana 的 Install：

$ kubectl get pods --namespace logging
NAME                       READY   STATUS    RESTARTS   AGE
grafana-698cbcc54b-v6b8t   1/1     Running   0          77s
loki-0                     1/1     Running   0          14m

以上说明都安装 OK!...

安装过程比较详细，安装参数比较简单，不再一一说明！

部署 Logging operator 和 demo application

Install the Logging operator and a demo application to provide sample log messages.

用 Helm 部署 Logging operator

To install the Logging operator using Helm, complete these steps.

Note:

For the Helm-based installation you need Helm v3.2.1 or later.

. Add the chart repository of the Logging operator using the following commands:

$ helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com
"banzaicloud-stable" has been added to your repositories

$ helm repo update
...
...Successfully got an update from the "grafana" chart repository
...Successfully got an update from the "banzaicloud-stable" chart repository
...Successfully got an update from the "bitnami" chart repository
Update Complete. ⎈Happy Helming!⎈

. 将 Logging operator 安装到 logging 命名空间中:

$ helm upgrade --install --wait --create-namespace --namespace logging logging-operator banzaicloud-stable/logging-operator

Release "logging-operator" has been upgraded. Happy Helming!
NAME: logging-operator
LAST DEPLOYED: Wed Dec 28 05:55:20 2022
NAMESPACE: logging
STATUS: deployed
REVISION: 2
TEST SUITE: None

. 安装演示应用程序及其日志记录定义.

$ helm upgrade --install --wait --create-namespace --namespace logging logging-demo banzaicloud-stable/logging-demo \
  --set "loki.enabled=True"

Release "logging-demo" does not exist. Installing it now.
NAME: logging-demo
LAST DEPLOYED: Wed Dec 28 05:59:29 2022
NAMESPACE: logging
STATUS: deployed
REVISION: 1

. Validate your deployment.

$ kubectl get pods --namespace logging
NAME                                         READY   STATUS      RESTARTS   AGE
grafana-698cbcc54b-v6b8t                     1/1     Running     0          139m
logging-demo-fluentbit-l7pmd                 1/1     Running     0          3m46s
logging-demo-fluentd-0                       2/2     Running     0          3m46s
logging-demo-fluentd-configcheck-3861c1ba    0/1     Completed   0          7m51s
logging-demo-log-generator-585749975-k6w7s   1/1     Running     0          7m51s
logging-operator-69b48dbf9d-mtbtt            1/1     Running     0          18m
loki-0

. 创建 logging 资源

$ kubectl -n logging apply -f - <<"EOF"
apiVersion: logging.banzaicloud.io/v1beta1
kind: Logging
metadata:
  name: default-logging-simple
spec:
  fluentd: {}
  fluentbit: {}
  controlNamespace: logging
EOF

logging.logging.banzaicloud.io/default-logging-simple created

$ kubectl get Logging -n logging
NAME                     AGE
default-logging-simple   29s
logging-demo             10m

注意：只能在 controlNamespace 中使用 ClusterOutput 和 ClusterFlow 资源.

. 创建 Loki output 定义

$ kubectl -n logging apply -f - <<"EOF"
apiVersion: logging.banzaicloud.io/v1beta1
kind: Output
metadata:
 name: loki-output
spec:
 loki:
   url: http://loki:3100
   configure_kubernetes_labels: true
   buffer:
     timekey: 1m
     timekey_wait: 30s
     timekey_use_utc: true
EOF

output.logging.banzaicloud.io/loki-output created

$ kubectl get Output  -n logging
NAME                       ACTIVE   PROBLEMS
logging-demo-output-loki   true
loki-output

注意：在生产环境中，请使用较长的 timekey 间隔以避免生成过多对象

. 创建 flow 资源

$ kubectl -n logging apply -f - <<"EOF"
apiVersion: logging.banzaicloud.io/v1beta1
kind: Flow
metadata:
  name: loki-flow
spec:
  filters:
    - tag_normaliser: {}
    - parser:
        remove_key_name_field: true
        reserve_data: true
        parse:
          type: nginx
  match:
    - select:
        labels:
          app.kubernetes.io/name: log-generator
  localOutputRefs:
    - loki-output
EOF

flow.logging.banzaicloud.io/loki-flow created

. 安装 demo 应用程序

$ kubectl -n logging apply -f - <<"EOF"
apiVersion: apps/v1
kind: Deployment
metadata:
 name: log-generator
spec:
 selector:
   matchLabels:
     app.kubernetes.io/name: log-generator
 replicas: 1
 template:
   metadata:
     labels:
       app.kubernetes.io/name: log-generator
   spec:
     containers:
     - name: nginx
       image: banzaicloud/log-generator:0.3.2
EOF

deployment.apps/log-generator created

. Validate your deployment

$ kubectl get po  -n logging
NAME                                                  READY   STATUS      RESTARTS   AGE
default-logging-simple-fluentbit-cglgc                1/1     Running     0          7m52s
default-logging-simple-fluentd-0                      2/2     Running     0          7m52s
default-logging-simple-fluentd-configcheck-8677d9e7   0/1     Completed   0          119s
grafana-698cbcc54b-v6b8t                              1/1     Running     0          149m
log-generator-cf689c55d-8xrdp                         1/1     Running     0          46s
logging-demo-fluentbit-l7pmd                          1/1     Running     0          13m
logging-demo-fluentd-0                                2/2     Running     0          13m
logging-demo-fluentd-configcheck-d8327387             0/1     Completed   0          119s
logging-demo-log-generator-585749975-k6w7s            1/1     Running     0          17m
logging-operator-69b48dbf9d-mtbtt                     1/1     Running     0          28m
loki-0

Validate the deployment

Grafana Dashboard

使用下面的命令来检索 Grafana admin 用户的密码:

$ kubectl get secret --namespace logging grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
a6Uo92.........2tnh

启用到 Grafana 服务的端口转发.

// 也可设置 svc 为 NodePort 访问
$ kubectl -n logging port-forward svc/grafana 3000:80

Open the Grafana Dashboard: http://localhost:3000
使用步骤1中检索到的 admin 用户名和密码来登录.
选择菜单 > Explore, 选择 Data source > Loki, 然后选择 Log labels > namespace > logging，应该出现一个日志列表.

类似于以下界面：