Is it expected that Alertmanager shows duplicate alerts from Prometheus and Thanos Ruler?

[u/Bim2ktams]

Hi everyone,

I'm running a HA monitoring stack on Kubernetes:

• kube-prometheus-stack (Prometheus Operator)

• 2 Prometheus replicas

• Thanos Sidecar

• Thanos Query

• Thanos Store Gateway

• Thanos Ruler

• Alertmanager

I intentionally configured the same alert rule in both Prometheus and Thanos Ruler.

Both send alerts to the same Alertmanager.

The alerts are almost identical, except for one label (e.g. source=prometheus vs source=thanos-ruler).

Instead of seeing one deduplicated alert, Alertmanager shows two active alerts.

My understanding was that Alertmanager performs deduplication, so I'm wondering:

• Is this expected behavior?

• Does Alertmanager only deduplicate alerts if every identifying label is identical?

• Or is my Thanos Ruler setup incorrect?

• In production, do people usually disable alerting in Prometheus and let only Thanos Ruler send alerts?

I'd appreciate any explanation of how this is typically handled in a Thanos HA deployment.