学习 Kubernetes(二十一):EFK 应用日志监控

日志监控也是应用可观察性的一种,通过调研基于 EFK 的 Kubernetes 日志监控方案有以下:

  • DaemonSet 部署 Fluent Bit 采集输出到 stdout 和 stderr 的 Docker 日志,缺点:不支持多行日志;
  • 使用边车(sidecar)模式,采集输出到文件的应用日志,缺点:浪费资源;
  • DaemonSet 部署 Fluentd 采集输出到文件挂载到 Node 本地的应用日志,也是最终选择的方案。

EFK

应用将日志目录挂载到 Node 本地:

containers:  
  - name: app
    volumeMounts:
      - name: log
        mountPath: <LOG_DIR>
    ...
volumes:  
  - name: log
    hostPath:
      path: /log/<APP_NAME>
      type: DirectoryOrCreate

Fluentd Docker Image

构建包含 ElasticSearch 插件的镜像:

docker build -t dyingbleed/fluentd:v1.8 . && docker push dyingbleed/fluentd:v1.8  

Dockerfile

FROM fluent/fluentd:v1.8-debian  
USER root  
RUN ["fluent-gem", "install", "fluent-plugin-elasticsearch"]  
USER fluent  

Fluentd DaemonSet

fluentd-ds.yaml 🌰

apiVersion: apps/v1  
kind: DaemonSet  
metadata:  
  name: fluentd-ds
  namespace: logging
spec:  
  selector:
    matchLabels:
      app: fluentd
  template:
    metadata:
      namespace: logging
      labels:
        app: fluentd
    spec:
      containers:
        - name: fluentd
          image: dyingbleed/fluentd:v1.8
          volumeMounts:
            - name: log
              mountPath: /log
            - name: config
              mountPath: /fluentd/etc
      volumes:
        - name: log
          hostPath:
            path: /log
            type: Directory
        - name: config
          configMap:
            name: fluent-cm
            items:
              - key: fluent.conf
                path: fluent.conf

创建 ConfigMap 脚本🌰:

kubectl create configmap fluentd-cm --from-file fluentd.conf  

Fluentd 配置文件 fluentd.conf 🌰:

<source>  
  @type tail
  path /log/<APP_NAME>/*.log
  pos_file /log/<APP_NAME>/log.pos
  tag <APP_NAME>.log
  <parse>
    @type multiline
    format_firstline /\d{4}-\d{1,2}-\d{1,2}/
    format1 /^(?<time>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}) \[(?<thread>.*)\] (?<level>[^\s]+)(?<message>.*)/
  </parse>
</source>

<match *.log>  
  @type elasticsearch
    host <ES_HOST>
    port <ES_PORT>
    logstash_format true
    logstash_prefix fluentd
</match>  

参考