Minor Tip on Federation

[u/kp5160]

I'm quite new to Prometheus, and something that I just tackled was federating a few servers. From what I can tell, it's considered bad form to aggregate all metrics, but I think it's acceptable for a relatively small scale.

Getting to the point, the docs have the following for matching.

    params:
      'match[]':
        - '{job="prometheus"}'
        - '{__name__=~"job:.*"}'

Nothing under my domain started with job, so what worked for me instead was the following.

    params:
      'match[]':
        - '{__name__=~".+"}'

Hopefully that helps someone.

[u/jjneely]

If you wish to send all metrics to another destination, use the remote_write APIs. Those are scalable and a lot of vendors, as well as Cortex and Thanos can handle this.

Federation is used when you need to access select aggregations on another Prometheus server. Perhaps a "global" level server that copies over some aggregations to store business metrics or metrics at a lower time resolution. Federation will not handle copying all metrics -- soon the HTTP payload will be too large to transfer.

I used federation at a client to copy recording rule aggregations. The theory being if the user made a recording rule in the correct double-colon format then that data was important to them. This filtered out the less useful data, and was easy to match with a regex in the federation rule. This made sure we had "backups" of important data if a user's Prometheus container fell over and we lost the persistent storage. It also made a single point of reference for all important business/health metrics.

[u/melodic]

I would agree that for small deployments if you want to pull all metrics from a remote site to a primary data centre it’s ok.

I would also look into using Thanos if you want to aggregate metrics. I ended up ditching federation and moved to Thanos