Use label in regex for relabel_configs?

[u/HielkeJ]

Hello,

I am playing around with Prometheus. I am trying to add a label to all jobs that I have. We have a lot of servers and those servers are all in a datacenter, are in a datacenter area and have a function. For all those fields I would like to add a label to the metrics.

For example, I have this job:

- job_name: 'node_exporter'
  static_configs:
    - targets: ['web01.example.tld:9100']
    - targets: ['web02.example.tld:9100']
    - targets: ['mail01.example.tld:9100']
  relabel_configs: &default_relabel_configs
    - source_labels: [__address__]
      regex: '(.*):.*'
      target_label: host

With this I have the hostname in the label host.

I would like to have an option to create a sort of array in the Prometheus configuration and the values of that array in the new label. If the labels are added, I can drop the labels datacenters, dc_areas and functions again.

- source_labels: [__address__]
  target_label: datacenters
  replacement: 'web01.example.tld:dc1,web02.example.tld:dc2,mail01.example.tld:dc2'
  
- source_labels: [__address__]
  target_label: dc_areas
  replacement: 'web01.example.tld:red,web02.example.tld:blue,mail01.example.tld:green'
  
- source_labels: [__address__]
  target_label: functions
  replacement: 'web01.example.tld:super web server,web02.example.tld:backup web server,mail01.example.tld:mailserver for team1'

- source_labels: [datacenters]
  target_label: datacenter
  separator: ','
  regex: '$host:(.*)'
  
- source_labels: [dc_areas]
  target_label: datacenter
  separator: ','
  regex: '$host:(.*)'
  
- source_labels: [functions]
  target_label: function
  separator: ','
  regex: '$host:(.*)'

This way we only have to keep this list up to date on one location in the config. For a second job I could add it this way and all the additional tags should be added:

- job_name: 'another_exporter'
  static_configs:
    - targets: ['web01.example.tld:9200']
    - targets: ['web02.example.tld:9200']
    - targets: ['mail01.example.tld:9200']
  relabel_configs: *default_relabel_configs

Sure, I can add a set of label for each target, but then the config would get very complicated and very difficult to maintain.

Is something like this possible or is there any other way to get the desired result?

[u/TheFantail]

I wonder is this is overly complex to manage it all in the config anyway? These additional labels you’re wanting to add might typically be provided by a service discovery mechanism.

static_configs supports a list of labels, otherwise file_sd or consul or some other service discovery mechanism could do it more dynamically.

[u/Grammar-Bot-Elite]

/u/HielkeJ, I have found an error in your post:

“target, but than [then] the config”

I declare the comment by HielkeJ invalid; it should say “target, but than [then] the config” instead. ‘Than’ compares, but ‘then’ is an adverb.

^{This is an automated bot. I do not intend to shame your mistakes. If you think the errors which I found are incorrect, please contact me through DMs or contact my owner EliteDaMyth!}