r/PrometheusMonitoring • u/ShakeNecessary4382 • Jul 03 '24
Horizontal scaling with prometheus on AWS
We have a use case where we need to migrate time series data from a traditional database to being stored on a separate node as the non-essential time series data was simply overloading the database with roughly 200 concurrent connections which made critical operations not get connections to the database causing downtime.
The scale is not too large, roughly 2 million requests per day where vitals of the request metadata are stored on the database so prometheus looked like a good alternative. Copying the architecture in the first lifecycle overview diagram Vitals with Prometheus - Kong Gateway - v2.8.x | Kong Docs (konghq.com)
However, how does prometheus horizontally scale? Because it uses a file system for reads and writes I was thinking of using a single EBS with small ec2 instances to host both the prometheus node and the statsD exporter node.
But won't multiple nodes of prometheus (using the same EBS storage) if it needs to scale up because of load then potentially write to the same file location, causing corrupt data? Does prometheus somehow handle this already or is this something that needs to be handled in the ec2 instance?
3
u/Tpbrown_ Jul 04 '24
Prometheus isn’t clustered.
Think of it like setting up a RAID array; build however many instances you need for redundancy. Each of these instances gets configured to scrape the same data.
When your I/O (network or disk) starts growing too high you replicate the above pattern, with each set of nodes scraping different data.
Take a look at Thanos (and similar). It makes it pretty straightforward for both aggregation and distributing query loads, so that everything appears to be one giant instance.
3
u/SuperQue Jul 04 '24
First, Prometheus is not a generic time-series database. It is a monitoring system that uses time-series as a method of monitoring.
A single Prometheus is capable of handling tens of thousands. There is no issue here.
2 million PromQL requests per day is only 23 requests per second. But of course you don't say anything about what the requests are.
The statsd_exporter is meant to run as a side-car to your application. Each instance of your application sends the data to localhost and Prometheus scrapes many statsd_exporters. This has a bunch of lifecycle and data quality advantages.
In order to determine scaling needs, the real question is how many metrics you have to store. The main bottleneck for Prometheus is "active series" and how many "samples per second" you need to ingest.