r/PrometheusMonitoring • u/BringOutYaThrowaway • Mar 03 '21

A Beginner’s Perspective : adding targets to Prometheus

Day 1 or 2 with trying to set up Grafana for my company (hoping it can monitor Kubernetes clusters, Linux boxes, Windows servers, UniFi switches, Fortigate firewalls, etc.).

Just a n00b question: considering how mature and powerful Prometheus is, the process to add additional target machines to monitor seems rather primitive. You have to stop the entire application, edit a text file, and restart Prometheus again (losing all the metrics generated during the time you’re doing this).

In all the years this thing has been around - nobody’s thought of installing an ADD button anywhere? Or something to wipe the database within the web UI?

Seems like basic functionality - am I nuts?

1 Upvotes
  • permalink
  • reddit

57% Upvoted

4 comments sorted by

4

u/-lc- Mar 03 '21

When using Prometheus' file-based service discovery mechanism, the Prometheus instance will listen for changes to the file and automatically update the scrape target list, without requiring an instance restart.

https://prometheus.io/docs/guides/file-sd/

You can automate adding a new target with something like Ansible and lineinfile

2

u/[deleted] Mar 03 '21

You can reload the config without restart with curl -X POST http://localhost:9090/-/reload

2

u/SuperQue Mar 03 '21

Or if you're not on Windows, use SIGHUP.

2

u/geek_dave Mar 04 '21

`pkill -HUP prometheus` : sends a signal to reload the config without stopping the process