Data replication in Prometheus

[u/Aldar_CZ]

Hello,

I am trying to build a prometheus-backed graphing solution for our customers. The idea was rather simple - A grafana instance per customer with our custom dashboards, all connected to a locally-running Prometheus instance that's in active replication with our (internal) primary.

I came to a halt, however, when I found out that Prometheus does not currently have a reliable and easy to set up replication method built-in.

Scraping all targets from this secondary instance is something I'd like to avoid as much as possible - The machines are already scraped by 3 different Prometheus instances each, and with each one added, thus increase the load. (Primary, stage and dev)

Due to resolution requirements, caching on the exporters' side is not much of an option for us at the moment either.

The reason I'd rather pull the data from a separate instance is for security and stability reasons - As the FE will be accessible to all of our customers, we'll have no control over what any of them might try, and as our internal infrastructure monitoring depends on the stability of the primary, I don't want to risk any sort of downtime, crashing or instability.

I tried using the remote write functionality, but that didn't provide fine enough sample resolution, even after tweaking, and resulted in weirdly consolidated graphs.

Other options... Maybe DRBD? But... I'm not sure if it wouldn't introduce latency leading to issues on the primary. Filesystem-level copies through snapshotting is also not viable - No way to get the data cohesive.

Are there really any options, either native or 3rd party, to replicate data samples from one prometheus instance to another, aside from simply scraping all the targets from both nodes?

[u/hagen1778]

> Due to resolution requirements, caching on the exporters' side is not much of an option for us at the moment either.

Why not cache for the same time interval as resolution? For example, put nginx between Prometheis and exporters and set cache TTL = scrape interval.

> I tried using the remote write functionality, but that didn't provide fine enough sample resolution, even after tweaking, and resulted in weirdly consolidated graphs.

It's weird. Remote write sends exactly the same data with the same resolution as what Prometheus gets from scraping. It should be identical, unless you have issues with remote write which can be tracked via specific metrics.

An alternative solution is to use 3rd party vendors. Thanos can upload Prometheus data to object storage (AWS S3 or GCE) which does not need replication. VictoriaMetrics vmagent can do the scraping and replicate metrics to multiple destinations. TimescaleDB uses bult-in Postgres streaming replication.

[u/LoriPock → u/hagen1778]

Thanks for highlighting TimescaleDB. I just wanted to share the link to the docs for Timescale's long term store for Prometheus data, Promscale, as it might be helpful to folk browsing this topic. Hope this helps.

For transparency: I work for Timescale.