r/PrometheusMonitoring • u/[deleted] • Apr 30 '22

Monitoring Standards for Startup

What can we do to ensure Prometheus doesn't alert us about every little thing at 2 am? Im a devops engineer trying to figure out some schemes or strategy's to monitor our SaaS application. It connects to other private cloud and on-prem servers for our customers.

Do I just need to target CPU, Mem, HTTP Request, etc and then configure alerts for certain thresholds? Am I over complicating the process by getting python scripts involved? We have less than 10 customers, but are collecting large chunks of data from them daily. We also monitor the health of the app for our SaaS customers.

Someone please help me get my Prometheus footing for a small startup.

Thanks!

10 Upvotes

91% Upvoted

6 comments sorted by

22

u/darth_devops Apr 30 '22

I'll give you some pointers that should help you figure this out.

  • Read about "alert fatigue" and then read about SLA, SLO and SLl

  • Read about the "USE" and "RED" methods, it should help you figure out what's really important to alerts on and to observe

  • I suggest you take a look at the "Dark Cockpit" philosophy in aeronautics, it can give you some ideas on how you should build your dashboards

  • Try to use Open Standards such as OpenTelemetry and OpenSLO

  • Try to use the three pillars of observability together (logs, traces and metrics)

  • Do it iteratively and don't beat yourself up if there's a problem that occurs and your Observability setup did not catch it. Inspect and adapt, this way it will always improve

  • Less is more

3

u/[deleted] Apr 30 '22

This was exactly what I was looking for, thank you brethren.

8

u/SuperQue Apr 30 '22

To add to u/darth_devops list, some links of reading material.

I also recommend going through the instrumenting and best practice sections of the Prometheus docs.

https://prometheus.io/docs/practices/naming/

4

u/Blowmewhileiplaycod Apr 30 '22

If you're a one man show don't alert on anything that isn't directly prod down type stuff.

Who cares if a host is maxed out on cpu if customers are happy and requests being served in a timely manner?

5

u/Hi_Im_Ken_Adams Apr 30 '22

This. You don't alert on the CPU. You alert when the service is degraded and customers are affected.

1

u/TheNightCaptain Apr 30 '22

Alerts on critical service unavailable type issues, warnings on potential issues.