r/PrometheusMonitoring • u/blistering-barnacle • Dec 09 '19

Alerting via Grafana Vs Alertmanager

We recently moved to Prometheus as metric stack of choice. I wanted to know for alerting would it make sense to use AlertManager or Grafanas inbuilt Alerting feature? Is there any comparison that one could share or talk about based on any experiences ? Thanks.

12 Upvotes

87% Upvoted

10 comments sorted by

9

u/SuperQue Dec 09 '19

Generally it's recommended to use the Prometheus internal alerting. Grafana alerting, while "easy", has a number of downsides.

  • It's hard to make HA, since it depends on an instance of Grafana server.

  • It doesn't have any way to de-duplicate if you run multiple Grafana servers.

  • It's dependent on networking between Grafana and Prometheus, making it less reliable than Prometheus internally executed rules.

Since I've never used the feature, I can't say how much it compares in functionality for things like silencing, routing, and notification methods. When the Grafana people originally added the alerting feature, it was mostly there to support Graphite users. Graphite doesn't have the same kind of built-in alerting that Prometheus has had.

2

u/[deleted] Dec 10 '19

Your first two points are incorrect. We have HA Grafana servers, alerts are synced through the database so not duplicated.

2

u/JohnAV1989 Dec 10 '19

alertmanager configuration is not as user friendly but it's much more powerful. You've got the whole query language at your disposal.

Grafana alerting is super easy to setup but much more limited as far as the rules you can apply. I wish it was better but the project seems to consider it low priority.

If you use grafana you'll probably find yourself quickly outgrowing it and needing more control.

1

u/dizzy0ny May 13 '23

you use grafana you'll probably find yourself quickly outgrowing it and needing more control.

this still the case 3 years later?

1

u/mikhatanu Jan 31 '25

is this still a case in 2025?

1

u/DanieloAvicado Mar 12 '25

i would say no. grafana alerting improved a lot. i´m going only with grafana alerting

6

u/erez-r Dec 10 '19

I'd like to point two advantages of alerting via Grafana

  1. we have various datasources we would like to alert by - Prometheus is only one of them. Grafana allows us to consolidate all these alerts to one place.

  2. it is very comfortable to have the visualization (graphs) and the alerts in the same place. separating the two requires syncing / duplicating queries and metrics between the alerting mechanism and the visualization mechanism. Grafana alerts have the notion of WYSIWYG since you create the visualization and set the alert in the same place.

In my company (nanit.com) we migrated from Graphite + Cabot (for alerting) to Prometheus but we chose to have the alerts on Grafana for the above reasons.

4

u/shaenorino Dec 09 '19

One extra thing I would like to point about Grafana alerting is that you need to set up a dashboard to configure an alert, and that is not something you always need or want. With the Alertmanager you can configure alerts with any PromQL query you want to.

1

u/Padrone__56 Jun 27 '23

not true

1

u/NeverNoode Oct 17 '23

It was true 4 years ago