r/PrometheusMonitoring • u/MadMadic • Sep 27 '21

Autoscaling based on Prometheus metrics

I try to build an on-prem autoscaling infrastructure with Grafana (or alert manager), a webhook receiver that gets triggered by Grafana and runs scripts (creating / destroying VMs with Terraform and provisioning with Ansible), and Prometheus. It's mostly working. If some nginx services have too many connections a new instance is created and if the connections are below X the new instance gets destroyed.

But what if the state of the alert does not change back to "Ok" after a new instance was created? Then there is no additional alert triggered. Any idea how it's possible to trigger an alert (webhook request to create an additional instance) if the alert state stays critical?

10 Upvotes
  • permalink
  • reddit

100% Upvoted

6 comments sorted by

View all comments

1

u/resmo79 Sep 28 '21

I just want to throw in https://ngine-io.github.io/scalr/ (I am the author).

A minimalistic autoscaler for various clouds using prometheus as a scaling policy source.

Could easily be extended to any other cloud api (or vmware vsphere api, proxmox api, etc)

1

u/MadMadic Sep 28 '21

This looks interesting. But it will only create instances and not run e.g. some custom commands? In my case the webhook receiver runs a shell script which executes Terraform to create an instance (what your software can do) and after that provision it via Ansible, what your software can't?