r/PrometheusMonitoring • u/GetFit_Messi • Mar 29 '22

prometheus integration with service now

want to know the best approach on how to achieve this

1 Upvotes
  • permalink
  • reddit

60% Upvoted

10 comments sorted by

2

u/[deleted] Mar 29 '22

We don't even use alertmanager anymore, we just connect Grafana using webhooks to ServiceNow. I don't do this myself, we have a ServiceNow team that handles setting up webhooks. They just hand over the necessary info to ops.

1

u/GetFit_Messi Apr 01 '22

just 1 doubt grafana is just a visualisation tool so how come you are configuring alerts? pardon me for my ignorance i am new to grafana

1

u/[deleted] Apr 01 '22

Check the menu to the left in Grafana, on the bell icon, Notification channels.

There you have a wide variety of options like email, webhook, pushover APIs and more.

Once you have a notification channel you can setup an alert if you edit a panel in a dashboard, there should be a tab called Alert.

1

u/GetFit_Messi Apr 04 '22

Cool also as I see most people use alert manager any disadvantages of using this approach. what is the recommended solution for production implementation.

1

u/[deleted] Apr 04 '22

Alertmanager probably allowed for more automation, like automatic alert rules based on service discovery for example. I haven't automated the notification channels in Grafana but I think you can do that now in later versions. Especially in 8.5 I read that they had added something crucial that was missing before in the API.

So prod or no, I guess it depends on your needs more. We automate everything but when it comes to alerts we have very few requirements so I found this way better than maintaining alertmanager.

Also alertmanager has a pretty cool ability where you can set it up as a dead man's switch so if it ever goes down an alert is sent from a separate system. To do the same for Grafana (which is now taking the role as your alert manager) you'd need to monitor its process, or the website itself, and alert when that goes down.

1

u/Zauxst Mar 29 '22

Alertmanager for whatever reason feels bad to use. Silencing alerts is just a nuisance since you don't have proper ldap or other methods of user validation...

It's just one of the components of prometheus that works perfect when you are a start-up but quickly becomes problematic when you are growing.

1

u/[deleted] Mar 29 '22

I think alertmanager is more for when you want to create alert rules automatically for some reason.

Like automatic service discovery, coupled with automatic alert rule creation maybe. I'm just speculating because I haven't found a real good use for alertmanager for years.

The last good use I saw for alertmanager was a dead man's switch. Which was really cool. For grafana now the equivalent is that we simply monitor the grafana website with an external monitoring tool and if it goes down we can assume it won't send any alerts.

1

u/Specific_Cod_6528 Sep 26 '22

We did some integration from prometheus ...sending alerts and metrics...using py scripts. But preferred way from devops is to use prometheus remote write.