{D4} - Prometheus的Relabeling机制

Fri, 22 Apr 2022 09:21:54 +0800

写这篇文章的目的是解释Prometheus relabeling的价值，以及它在整个prometheus数据流不同阶段的重要性。

在面向大型互联网公司、金融行业等大规模业务监控的场景，原生的 Prometheus 单实例模式无法直接满足需求，需要一种面向生产环境的集群化高可用方案来进行支撑。

: http://sconts.com/legacy/post/prometheus-relabel/

{D3} - Prometheus数据格式及指标类型

Sat, 16 Apr 2022 09:20:54 +0800

Prometheus metrics 概念已被广泛采用，不仅被Prometheus用户采用，还被包括InfluxDB、OpenTSDB、Graphite 和Sysdig Monitor在内的其他监控系统广泛采用。如今，许多CNCF项目使用Prometheus指标格式公开了开箱即用的指标。您还可以在API服务器、etcd、CoreDNS 等核心Kubernetes组件中找到它们。您可以在使用Prometheus的Kubernetes监控指南中了解更多信息。

: http://sconts.com/legacy/post/prometheus-metric/

{D2} - Prometheus配置详解之global,alerting,rule_files,scrape_configs,remote_read,remote_write

Fri, 15 Apr 2022 09:19:54 +0800

今天这里就不做过多解释了，直接上配置，可以先对prometheus的配置参数有个了解。

  1global:
  2  # 抓取指标的间隔，默认1m
  3  scrape_interval: 10s
  4  # 抓取指标的超时时间，默认10s
  5  scrape_timeout: 15s
  6  # 指定Prometheus评估规则的频率[记录规则(record)和告警规则(alert)],默认1m.
  7  # 可以理解为执行规则的时间间隔
  8  evaluation_interval: 15s
  9  # PromQL查询日志的相关记录文件，有点类似mysql slowlog
 10  query_log_file: prometheus_query_log
 11  # 用于区分不同的prometheus
 12  external_labels:
 13    datacenter: 'hangzhou-1'
 14    region: 'huadong'
 15
 16
 17# Alertmanager configuration
 18alerting:
 19  alertmanagers:
 20    - static_configs:
 21      - targets:
 22        - 192.168.56.11:9093
 23  alert_relabel_configs:
 24    - source_labels: [dc]
 25      regex: (.+)\d+
 26      target_label: dc
 27    - source_labels: "host"
 28      target_label: "instance"
 29      regex: "(.+)"
 30
 31
 32# Load rules once and periodically evaluate them
 33# according to the global 'evaluation_interval'.
 34rule_files:
 35  - "rules/node_alerts.yml"
 36  - "rules/node_rules.yml"
 37
 38
 39# 发现和抓取target配置相关
 40scrape_configs:
 41  # The job name is added as a label `job=<job_name>`
 42  # to any timeseries scraped from this config.
 43  - job_name: "prometheus"
 44    # metrics_path defaults to '/metrics'
 45    # scheme defaults to 'http'.
 46    static_configs:
 47      - targets: ["192.168.56.11:9090"]
 48        labels:
 49          instance: "prometheus"
 50
 51  - job_name: "pushgateway"
 52    honor_labels: true
 53    static_configs:
 54      - targets: ['192.168.56.11:9091']
 55        labels:
 56          instance: "push-gateway"
 57
 58  - job_name: "node-exporter"
 59    metrics_path: /metrics
 60    scrape_interval: 15s
 61    static_configs:
 62      - targets: ['192.168.56.11:9100']
 63
 64  - job_name: "victoriametrics-exporter"
 65    metrics_path: /metrics
 66    static_configs:
 67      - targets: ['192.168.56.11:8428']
 68        labels:
 69          instance: "victoriametrics"
 70
 71  - job_name: "app_http_list"
 72    http_sd_configs:
 73      - url: http://192.168.56.11:8010/jobs.json
 74        refresh_interval: 5s
 75#        basic_auth:
 76#          username: prometheus
 77#          password: changeme
 78#        authorization:
 79#          type: "Token"
 80#          credentials: "0123456789abcdef0123456789abcdef01234567"
 81    relabel_configs:
 82      - source_labels:     # 对replace有效
 83        - __meta_datacenter
 84        #regex: "(http|https)(.*)"
 85        #separator: ""
 86        target_label: "datacenter"
 87        replacement: "${1}"
 88        action: replace
 89#      - source_labels: [__address__]
 90#        target_label: __param_target
 91#        target_label: __address__
 92#        replacement: 127.0.0.1:9115
 93
 94#  - job_name: 'scrape-then-use-relabel-to-rename-selected-metrics'
 95#    static_configs:
 96#      - targets: ['10.10.10.1:19126']
 97#    metric_relabel_configs:
 98#      - source_labels: [ __name__ ]
 99#        target_label: __name__
100#        regex: '(^(?:go|http|process)_.*$)'
101#        action: replace
102#        replacement: telegraf_${1}
103
104#  - job_name: 'scrape-then-use-relabel-to-remove-a-label'
105#    static_configs:
106#      - targets: ['10.10.10.2:19126']
107#    metric_relabel_configs:
108#      - source_labels: [ host ]
109#        target_label: host
110#        action: replace
111#        replacement: ''
112
113#  - job_name: 'scrape-then-use-relabel-add-an-arbitrary-label'
114#    static_configs:
115#      - targets: ['10.10.10.3:19126']
116#    metric_relabel_configs:
117#     - source_labels: [ instance ]
118#       target_label: instance
119#       action: replace
120#       replacement: log-downloader
121
122
123remote_write:
124  # 指定写入数据的url
125  - url: http://192.168.56.11:8428/api/v1/write
126    remote_timeout: 30s
127    # 远程写配置的名称，如果指定，则在远程写配置中必须是唯一的。
128    # 该名称将用于度量标准和日志记录中，代替生成的值，以帮助用户区分远程写入配置。
129    name: drop_expensive
130    # 远程写重新打标签配置
131    write_relabel_configs:
132    - source_labels: [__name__]
133      regex: expensive.*
134      action: drop
135    queue_config:
136      batch_send_deadline: 5s
137      max_shards: 30
138      min_shards: 4
139      max_samples_per_send: 500
140      capacity: 20000
141
142
143# !!!提示!!!
144# 后续我们使用victoria-metrics作为远端存储
145# `remote_read`将直接读取vm的查询接口数据
146remote_read:
147  # 指定读取数据的url
148  - url: http://remote1/read
149    # 表示近期数据也要从远程存储读取，
150    # 因为Prometheus近期数据无论如何都是要读本地存储的。
151    # 设置为true时，Prometheus会把本地和远程的数据进行Merge。
152    # 默认是false，即从本地缓存查询近期数据.
153    read_recent: true
154    name: default
155  # 指定读取数据的第二个url
156  - url: http://remote3/read
157    # 从本地缓存查询近期数据
158    read_recent: false
159    name: read_special
160    # 可选的匹配器列表，必须存在于选择器中以查询远程读取端点。
161    required_matchers:
162      job: special
163    tls_config:
164      cert_file: valid_cert_file
165      key_file: valid_key_file

: http://sconts.com/legacy/post/prometheus-configs/

{D1} - Prometheus初识和服务部署

Thu, 14 Apr 2022 00:00:00 +0000

+++ 第一部分：Prometheus简介及一些必要的名词解释 +++

From metrics to insight

Power your metrics and alerting with the leading open-source monitoring solution.

从指标到洞察力，使用领先的开源监控解决方案为您的数据指标和警报提供助力。

: http://sconts.com/legacy/post/getting-started-with-prometheus/

Monitor on 六二三's Blog

{D4} - Prometheus的Relabeling机制

{D3} - Prometheus数据格式及指标类型

{D2} - Prometheus配置详解之global,alerting,rule_files,scrape_configs,remote_read,remote_write

{D1} - Prometheus初识和服务部署

+++ 第一部分：Prometheus简介及一些必要的名词解释 +++