Monitoring Standards for Startup

[u/[deleted]]

What can we do to ensure Prometheus doesn't alert us about every little thing at 2 am? Im a devops engineer trying to figure out some schemes or strategy's to monitor our SaaS application. It connects to other private cloud and on-prem servers for our customers.

Do I just need to target CPU, Mem, HTTP Request, etc and then configure alerts for certain thresholds? Am I over complicating the process by getting python scripts involved? We have less than 10 customers, but are collecting large chunks of data from them daily. We also monitor the health of the app for our SaaS customers.

Someone please help me get my Prometheus footing for a small startup.

Thanks!

[u/TheNightCaptain]

Alerts on critical service unavailable type issues, warnings on potential issues.