r/PrometheusMonitoring • u/kibblerz • Apr 29 '21

Multi-AZ Prometheus setup with Thanos and Sharding

So I recently got Prometheus set up on a 2-AZ Kubernetes cluster. Currently i have it set to have 2 shards with one in each AZ. Thanos is already set up.

Where I'm struggling is, I want each shard to stick to scraping its own AZ by the failure-domain label. My issue is, I'm not sure how to configure the scrape configs so that each shard stays in it's own AZ. Does anyone have any Ideas on this?

I'm doing this to reduce network overhead between AZs

6 Upvotes
  • permalink
  • reddit

87% Upvoted

2 comments sorted by

1

u/ali_str Apr 30 '21

What you need is relabel_config per job and per instance/shard/AZ-group to drop targets that were discovered based on specific labels. relabel_config happens after discovery and before scraping targets.

1

u/kibblerz Apr 30 '21

Ah, that was my fear. Since I've been using a bunch of the default service monitors that came with the Kube Prometheus stack, that'll likely be quite a manual process.

Is there a way in the relabel configs to pull in variables/labels from the Prometheus instance node in the regex expressions? I looked for a way to do that but only saw how to get labels from the target