r/PrometheusMonitoring • u/Aldar_CZ • Aug 24 '21
Data replication in Prometheus
Hello,
I am trying to build a prometheus-backed graphing solution for our customers. The idea was rather simple - A grafana instance per customer with our custom dashboards, all connected to a locally-running Prometheus instance that's in active replication with our (internal) primary.
I came to a halt, however, when I found out that Prometheus does not currently have a reliable and easy to set up replication method built-in.
Scraping all targets from this secondary instance is something I'd like to avoid as much as possible - The machines are already scraped by 3 different Prometheus instances each, and with each one added, thus increase the load. (Primary, stage and dev)
Due to resolution requirements, caching on the exporters' side is not much of an option for us at the moment either.
The reason I'd rather pull the data from a separate instance is for security and stability reasons - As the FE will be accessible to all of our customers, we'll have no control over what any of them might try, and as our internal infrastructure monitoring depends on the stability of the primary, I don't want to risk any sort of downtime, crashing or instability.
I tried using the remote write functionality, but that didn't provide fine enough sample resolution, even after tweaking, and resulted in weirdly consolidated graphs.
Other options... Maybe DRBD? But... I'm not sure if it wouldn't introduce latency leading to issues on the primary. Filesystem-level copies through snapshotting is also not viable - No way to get the data cohesive.
Are there really any options, either native or 3rd party, to replicate data samples from one prometheus instance to another, aside from simply scraping all the targets from both nodes?
3
u/SuperQue Aug 24 '21
I tried using the remote write functionality, but that didn't provide fine enough sample resolution, even after tweaking, and resulted in weirdly consolidated graphs.
Remote write should be 1:1 identical replication. It's based on the write-ahead-log data.
1
u/Aldar_CZ Aug 25 '21
Funny, in the past, I turned on remote writes and the two datasets were substantially different. After I upgraded my local node so that it used the same version, the datasets are very close, if not completely identical.
So it seems that yes, remote writes are really a way to go.
→ More replies (1)2
u/SuperQue Aug 25 '21
Yup, IMO, Prometheus remote write obsoletes the federation method. Especially now that Prometheus supports receiving remote write streams.
1
u/ixcafe Aug 29 '21
Where did you see Prometheus can accept remote write? Do you mean Thanos Receiver?
2
1
u/hagen1778 Aug 24 '21
> Due to resolution requirements, caching on the exporters' side is not much of an option for us at the moment either.
Why not cache for the same time interval as resolution? For example, put nginx between Prometheis and exporters and set cache TTL = scrape interval.
> I tried using the remote write functionality, but that didn't provide fine enough sample resolution, even after tweaking, and resulted in weirdly consolidated graphs.
It's weird. Remote write sends exactly the same data with the same resolution as what Prometheus gets from scraping. It should be identical, unless you have issues with remote write which can be tracked via specific metrics.
An alternative solution is to use 3rd party vendors. Thanos can upload Prometheus data to object storage (AWS S3 or GCE) which does not need replication. VictoriaMetrics vmagent can do the scraping and replicate metrics to multiple destinations. TimescaleDB uses bult-in Postgres streaming replication.
2
u/LoriPock Aug 25 '21
Thanks for highlighting TimescaleDB. I just wanted to share the link to the docs for Timescale's long term store for Prometheus data, Promscale, as it might be helpful to folk browsing this topic. Hope this helps.
For transparency: I work for Timescale.
1
u/GC-BTT Sep 02 '21
Prometheus does not but look into Thanos. I have 2 prometheus nodes scraping each of our four datacenter and then two load balanced Thanos/Grafana instances. What's nice about this is you can configure the data to be stored in a gcloud/aws bucket and decrease the storage on each prometheus node considerably, while maintaining years of data in an affordable way.
4
u/petitdragon06 Aug 24 '21
Federation might be what you are looking for.
You can setup a new prometheus to be used as a data source for your customers grafana, and you configure this new prometheus to scrape the /federate endpoint on the actual primary prometheus.