監視 Docker Container 資源 (cAdvisor + node-exporter + prometheus + grafana)

cAdvisor 是一個開源的容器資源使用情況和效能分析工具,主要由 Google 開發。它專門用於收集、聚合、處理和匯出有關執行中容器的資訊。
Node-Exporter 收集主機層級的各種指標,包括:CPU 使用率、記憶體使用情況、磁碟 I/O 統計、網路流量、檔案系統使用率、系統負載

  • Node Exporter:關注整個主機/節點層級的指標
  • cAdvisor:專注於容器層級的指標

再搭配上 Prometheus 收集指標,用 Grafana 呈現資訊

圖片

出自 https://blog.techbridge.cc/2019/08/26/how-to-use-prometheus-grafana-in-flask-app/

開發環境

  • Windows 11 Home
  • WSL2
  • Untuntu 24.04.01

設定

docker-compose.yml 配置如下:

version: "3"

networks:
  prometheus:
    driver: bridge

services:
  node-exporter:
    image: prom/node-exporter:v1.9.0
    container_name: node-exporter
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
    command:
      - '--path.procfs=/host/proc'
      - '--path.sysfs=/host/sys'
      - --collector.filesystem.ignored-mount-points
      - "^/(sys|proc|dev|host|etc|rootfs/var/lib/docker/containers|rootfs/var/lib/docker/overlay2|rootfs/run/docker/netns|rootfs/var/lib/docker/aufs)($$|/)"
    ports:
      - "9100:9100"
    networks:
      - prometheus
  
  cadvisor:
    image: gcr.io/cadvisor/cadvisor:v0.49.1
    container_name: cadvisor
    #network_mode: "host"
    ports:
      - "8080:8080"
    volumes:
      - /:/rootfs
      - /var/run:/var/run:rw
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk
    privileged: true
    devices:
      - "/dev/kmsg"
    networks:
      - prometheus
  
  prometheus:
    image: prom/prometheus:v3.1.0
    container_name: prometheus
    #network_mode: "host"
    ports:
      - "9090:9090"
    volumes:
      - ./config/prometheus:/etc/prometheus
    networks:
      - prometheus
  
  grafana:
    image: grafana/grafana:11.4.0
    container_name: grafana
    #network_mode: "host"
    volumes:
      - ./data/grafana:/var/lib/grafana
      - ./config/grafana/provisioning:/etc/grafana/provisioning  # 添加 provisioning 映射
    environment:
      - GF_PATHS_PROVISIONING=/etc/grafana/provisioning
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
    ports:
      - "3000:3000"
    depends_on:
      - loki
      - prometheus
    networks:
      - prometheus 

 

./config/grafana/provisioningdashboards

裡面有兩個 Dashboard

Docker Containers

https://github.com/yaochangyu/sample.dotblog/blob/master/Prometheus/Docker%20Monitor/config/grafana/provisioning/dashboards/Docker%20Containers.json

Docker Host

https://github.com/yaochangyu/sample.dotblog/blob/master/Prometheus/Docker%20Monitor/config/grafana/provisioning/dashboards/Docker%20Host.json

 

./config/grafana/provisioning/datasources/datasources.yml

datasources:
  - name: prometheus
    type: prometheus
    url: http://prometheus:9090  # 修改為容器名稱
    access: proxy
    isDefault: true

 

./config/prometheus.yml

每 15 秒收集一次資訊

global:
  scrape_interval: 15s #Scrape interval to every 15 seconds.
  evaluation_interval: 15s #Evaluate rules every 15 seconds.
  
scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["prometheus:9090"]
  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node-exporter:9100']
  - job_name: "cadvisor"
    static_configs:
      - targets: ["cadvisor:8080"]

 

檔案結構,如下圖

 

準備好相關配置之後,就可以部署到測試環境,我將使用 WSL2,可以參考上篇

https://dotblogs.com.tw/yc421206/2025/02/23/installing_docker_on_windows_wsl

 

把檔案放到 WSL 最簡單的方式就是在檔案總管訪問 \\wsl$,以我的電腦為例 Ubuntu-24.04 的路徑為 \\wsl.localhost\Ubuntu-24.04,把 docker-compose.yml 複製到 \\wsl.localhost\Ubuntu-24.04 並增加 data 資料夾

 

修改 data 權限

sudo chown -R 472:472 ./data
sudo chmod -R 755 ./data

 

接下來把 lab 叫起來,docker compose up

 

訪問 cAdvisor,http://localhost:8080/,可以看到他已經收集到了 container

 

訪問 Node Exporter,http://localhost:9100/

 

訪問 prometheus,http://localhost:9100/

 

訪問 grafana,http://localhost:3000/,查看 dashboard

心得

這是一個蠻簡單的 Lab,可以從網路上面找到很多範例,但是卡了我好久,不論我怎麼調整配置,cAdvisor 都不會有 container 的數據,最後,我從原本的 Docker Desktop 換成了 WSL + Docker,數據就蒐集到了。

範例位置

https://github.com/yaochangyu/sample.dotblog/tree/master/Prometheus/Docker%20Monitor

 

若有謬誤,煩請告知,新手發帖請多包涵


Microsoft MVP Award 2010~2017 C# 第四季
Microsoft MVP Award 2018~2022 .NET

Image result for microsoft+mvp+logo