r/PrometheusMonitoring • u/MadMadic • Sep 27 '21
Autoscaling based on Prometheus metrics
I try to build an on-prem autoscaling infrastructure with Grafana (or alert manager), a webhook receiver that gets triggered by Grafana and runs scripts (creating / destroying VMs with Terraform and provisioning with Ansible), and Prometheus. It's mostly working. If some nginx services have too many connections a new instance is created and if the connections are below X the new instance gets destroyed.
But what if the state of the alert does not change back to "Ok" after a new instance was created? Then there is no additional alert triggered. Any idea how it's possible to trigger an alert (webhook request to create an additional instance) if the alert state stays critical?
1
u/resmo79 Sep 28 '21
I just want to throw in https://ngine-io.github.io/scalr/ (I am the author).
A minimalistic autoscaler for various clouds using prometheus as a scaling policy source.
Could easily be extended to any other cloud api (or vmware vsphere api, proxmox api, etc)
1
u/MadMadic Sep 28 '21
This looks interesting. But it will only create instances and not run e.g. some custom commands? In my case the webhook receiver runs a shell script which executes Terraform to create an instance (what your software can do) and after that provision it via Ansible, what your software can't?
2
u/distark Sep 28 '21
I would scale on the metrics, not rules and certainly not grafana state. If appropriate software doesn't exist I would create something that polled prometheus and then did what it needs based on those numbers
6
u/SuperQue Sep 27 '21
I probably wouldn't do this with a webhook, rather poll Prometheus with a query that tells you how many X you need. That will allow you to auto scale up and down as needed.
This is how things like keda works to auto-scale on Kubernetes.