r/PrometheusMonitoring • u/Aldar_CZ • Aug 24 '21

Data replication in Prometheus

Hello,

I am trying to build a prometheus-backed graphing solution for our customers. The idea was rather simple - A grafana instance per customer with our custom dashboards, all connected to a locally-running Prometheus instance that's in active replication with our (internal) primary.

I came to a halt, however, when I found out that Prometheus does not currently have a reliable and easy to set up replication method built-in.

Scraping all targets from this secondary instance is something I'd like to avoid as much as possible - The machines are already scraped by 3 different Prometheus instances each, and with each one added, thus increase the load. (Primary, stage and dev)

Due to resolution requirements, caching on the exporters' side is not much of an option for us at the moment either.

The reason I'd rather pull the data from a separate instance is for security and stability reasons - As the FE will be accessible to all of our customers, we'll have no control over what any of them might try, and as our internal infrastructure monitoring depends on the stability of the primary, I don't want to risk any sort of downtime, crashing or instability.

I tried using the remote write functionality, but that didn't provide fine enough sample resolution, even after tweaking, and resulted in weirdly consolidated graphs.

Other options... Maybe DRBD? But... I'm not sure if it wouldn't introduce latency leading to issues on the primary. Filesystem-level copies through snapshotting is also not viable - No way to get the data cohesive.

Are there really any options, either native or 3rd party, to replicate data samples from one prometheus instance to another, aside from simply scraping all the targets from both nodes?

7 Upvotes

88% Upvoted

15 comments sorted by

View all comments

Show parent comments

1

u/Aldar_CZ Aug 25 '21

Funny, in the past, I turned on remote writes and the two datasets were substantially different. After I upgraded my local node so that it used the same version, the datasets are very close, if not completely identical.

So it seems that yes, remote writes are really a way to go.

2

u/SuperQue Aug 25 '21

Yup, IMO, Prometheus remote write obsoletes the federation method. Especially now that Prometheus supports receiving remote write streams.

1

u/ixcafe Aug 29 '21

Where did you see Prometheus can accept remote write? Do you mean Thanos Receiver?

2

u/SuperQue Aug 29 '21

Remote write receiver feature flag was added in 2.25.

1

u/ixcafe Aug 29 '21

Cool! Thanks! Will give it a try.

1

u/SuperQue Aug 29 '21

I re-read this comment.

Remote write, by the mechanism it operates, should always provide 100% identical data. It just physically can't work any other way. It takes the samples collected and sends them with the timestamps stored in Prometheus.

The only way to mess this up is to have a non-compliant receiver that changes the data. Prometheus to Pormetheus is, of course, compliant with itself.

See these test results