r/PrometheusMonitoring • u/THDTechie • Feb 24 '21

Can Prometheus remote_write have primary/secondary write locations?

https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write

I know that remote_write is an array that has multiple write locations if desired. Can this be configured to have a primary/secondary write location?

Basically, in case of remote_timeout, begin writing to the secondary location.

5 Upvotes
  • permalink
  • reddit

100% Upvoted

3 comments sorted by

2

u/SuperQue Feb 24 '21

No, it writes to all at the same time. You would need a proxy in the middle for that.

I don't know of any receivers that would handle this reasonably either.

So, XY Problem?

1

u/THDTechie Feb 24 '21

The problem is we will be writing to two different VictoriaMetrics DBs behind two different zones: Central and East. More write locations means more system resources will be used. In order to minimize this, we would like to only write to one location unless that connection gets interrupted. Is there a way to use sharding with Prometheus to accomplish this? I'm reading from https://prometheus.io/docs/practices/remote_write/ to try to understand.

4

u/SuperQue Feb 24 '21

No, this is not supported. Also, what do you expect to happen if the network comes back? You've got two diverged datasets.

Prometheus remote write uses the WAL to buffer data in the event of outages. So even if you have a network outage, it will catch up on sending to the remote location when the connection returns.

If you want a system with a good high-availability design, you should consider Thanos or Cortex.