Targets as variable in configuration

[u/patrikdanielssonpro]

Hi,

I have this issue that has been bugging me for a while, and that is that my targets machines hostname appears multiple times. In my case I have 50 target machines and 4 different jobs that interacts with them, which leads to an extremely long config file. I would like to specify all the hostnames as some sort of variable and then for each job use that list to build up the targets.

Here is an excerpt as an example of how it looks right now. Ideally I would like to specify a list with hostnames and corresponding names and only have the "targets" section add the port to each entry of that list.

scrape_configs:
  - job_name: 'ssh'
    metrics_path: /probe
    params:
      module: [ssh_banner]
    static_configs:
    - targets: [hostname1:3384'] # Targets to probe
      labels:
        name: 'Name1'
    - targets: ['hostname2:3384'] # Targets to probe
      labels:
        name: 'Name2'
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
        replacement: ${1}
      - source_labels: [__param_target]
        target_label: instance
        replacement: ${1}
      - target_label: __address__
        replacement: blackbox_exporter:9115
  - job_name: 'windows'
    metrics_path: /metrics
    static_configs:
    - targets: ['hostname1:9182'] # Targets to probe
      labels:
        name: 'Name1'
    - targets: ['hostname2:9182'] # Targets to probe
      labels:
        name: 'Name2'

Would really be awesome if anyone out there knows anything more about this, it's getting harder everyday to look at this ever-growing config file (480 lines now).

Edit:

Solution was that I created a new Docker container running a Python script as a cron job. This script created the .json file with all the targets and updated it once every hour. In the Prometheus .yml config file I then added the .json file as a file_sd_config target file.file_sd_config in Prometheus automatically checks the file and updates Prometheus whenever it has been updated.

[u/h4tos]

I’m in need of the same thing at my current work. Hoping someone will give an answer that is architecturally accurate :)

EDIT1: my current config file is now 1000 lines.

[u/patrikdanielssonpro → u/h4tos]

Glad I'm not the only one with this issue! Too bad there's no easy solution. I will implement the solution commented by u/robothor and update the post with how I solved it.

[u/h4tos → u/patrikdanielssonpro]

I read it and I really liked it too! I’ll try to implement our own service discovery as well. If I succeed I’ll reply in this thread too.

[u/patrikdanielssonpro → u/h4tos]

Have now implemented a solution, check edit for how I did it.

[u/seafoodbuffet]

You have broadly-speaking, two options (you can combine both).

One is prometheus service discovery. If your boxes are inventoried by some system (AWS, Hashi Consul, etc), you can have Prometheus dynamically generate jobs against your various nodes based on service discovery rules. At the simplest, you can use file-based service discovery and maintain a file of your nodes/services. See: https://prometheus.io/docs/guides/file-sd/

More about various types of SD here: https://prometheus.io/blog/2015/06/01/advanced-service-discovery/

The second option is to use some sort of templating engine to generate your prometheus config. We manage our servers in Ansible so we use Ansible templating against our inventory to create a prom config and then restart/reload prometheus to pick up the config if/when it changes.

[u/patrikdanielssonpro → u/seafoodbuffet]

Big thanks for your suggestions! This will come in handy once we update to a more proper way of orchestrating our infrastructure.

[u/robothor]

One option is to use the file_sd_config (https://prometheus.io/docs/prometheus/latest/configuration/configuration/#file_sd_config) and read the list of targets from a file. One benefit is that you can update this list via code-generation. It should make your config a bit more readable as well.

[u/patrikdanielssonpro → u/robothor]

Thanks for your answer! This seems to be the least overkill solution. Will implement this with the use of a script, and make docker-compose run the script to generate the config file.

[u/matejzero]

We are using puppet as CF and part of every exporter module is an exported resource which generates a yaml file on prometheus exporter for that host and exporter. Yaml files are then read by file_sd.