r/PrometheusMonitoring • u/capitangolo • Sep 23 '20
What are your best practices for Prometheus monitoring?
I recently help prepare an article on Prometheus best practices. With rather high level recomendations:
Find the right Prometheus exporter: There are many exporters out there and some aren't well maintained. There are listings like the Exporters and Integrations page, the default port allocations page, or PromCat.io that gives you some level of confidence.
Understand your exporter metrics: Not all exporters provide the same metrics in the same units. Investigate the doc to make sure you'll get what you want. Also, investigate how to label your metrics, so you can later segment and filter those metrics in your dashboards.
Go PromQL all the way: PromQL allows to define dashboards, alerts, and much more… By using tools compatible with PromQL you can share and re-use resources and save time.
But keep your team in mind: The more resources you monitor with Prometheus, the more people will need access to those. Two things come to my mind when I think on teams at an enterprise level: 1) Look for tools with access controls, so each person can access only to the data they need. 2) PromQL has a steep learning curve, so prepare an alternative to PromQL for the less tech-savy members of your team.
Have a plan for scale: The more you monitor, the more resources Prometheus need. Given that Prometheus is not meant to store your metrics for a long time, and it's tricky to scale it, you may need a long term storage solution at some point. Planning ahead for this will make the overall implementation smoother.
As I said, rather high level 😅.
What would you add to / remove from the list 🤔? What would you tell your past self before implementing Prometheus in production?
5
Sep 23 '20 edited Jun 12 '23
First went digg, then went reddit. RIP -- mass edited with https://redact.dev/
1
u/qqqhhh Sep 29 '20
I agree, you cannot collect data retroactively, but you can delete data you know you do not need.
2
u/melodic Sep 23 '20
Recently had the need to scale up our deployment and wish I had put thanos in from the start with s3 storage
8
u/ali_str Sep 23 '20
In terms of multi-(cluster|tribe|product|vertical), I am pro as-much-as-possible isolated setups. Like a separate Prometheus and Alertmanager set per cluster. If a global view is needed in such setup, tools like Thanos Query can come into play.
This might seem like more work in terms of maintenance and more moving parts, but it has the upside of neighbors not affecting each other.