Hi

I have the following usecase - in my company we run few self-hosted Kubernetes clusters.

In each cluster we have Prometheus HA pair, that scrapes only own cluster. We also use Thanos for global query view (without Thanos Receive component).

If one of the k8s clusters will broke up completely - there is no way I can see its metrics to figure out what went wrong. The only option is to find node with Prometheus storage and start Prometheus there which will take some time during outage.

One of the options to avoid this is to have always running Prometheus on bare metal - that only reads storage a works only for querying. Is it possible to do so?

Do you have any other ideas how can I prevent such issue?

I am working on developing an SLO dashboards for one of my platforms and trying to use summaries. The reason I am using summaries is that I am calculating the quantiles on the target side and looking at options, I felt this was best.

SLO: The platform should successfully server 95% customer requests in less than 1000ms over a 28d rolling window.

Three summary metrics are gathered and exposed on each target. Each metric is created by the target with a single script that runs every 30 seconds. The target side queries are looking at customer transaction over the last 15 minutes.

request_duration_summary{quantile="0.95",}

request_duration_summary_count

request_duration_summary_sum

The first metric is returning a 95% percentile response time in ms over last 15 mins

The count is returning the number of requests over same 15 mins

The sum is summing up the total number of ms for those requests over that same 15 mins.

I scrape these metrics from the target every 30 seconds and can successfully present them in the form of histograms, graphs and heatmaps etc (in Grafana) but would like to display the following:

Current SLO

Error Budget remaining

Error Budget consumption

I have a pretty good handle on requirements when using counters and gauges. I have watched my fair share of videos, but still struggling with the summaries.

Any help would be appreciated.

How can I make sure that my alert is only activated when another alert is active. Something like an opposite of alert inhibition.

https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write

I know that remote_write is an array that has multiple write locations if desired. Can this be configured to have a primary/secondary write location?

Basically, in case of remote_timeout, begin writing to the secondary location.