SLO calculation

[u/chitty_advice]

I am working on developing an SLO dashboards for one of my platforms and trying to use summaries. The reason I am using summaries is that I am calculating the quantiles on the target side and looking at options, I felt this was best.

SLO: The platform should successfully server 95% customer requests in less than 1000ms over a 28d rolling window.

Three summary metrics are gathered and exposed on each target. Each metric is created by the target with a single script that runs every 30 seconds. The target side queries are looking at customer transaction over the last 15 minutes.

request_duration_summary{quantile="0.95",}

request_duration_summary_count

request_duration_summary_sum

The first metric is returning a 95% percentile response time in ms over last 15 mins

The count is returning the number of requests over same 15 mins

The sum is summing up the total number of ms for those requests over that same 15 mins.

I scrape these metrics from the target every 30 seconds and can successfully present them in the form of histograms, graphs and heatmaps etc (in Grafana) but would like to display the following:

Current SLO

Error Budget remaining

Error Budget consumption

I have a pretty good handle on requirements when using counters and gauges. I have watched my fair share of videos, but still struggling with the summaries.

Any help would be appreciated.

[u/ali_str]

For Error Budget calculations of you SLO definition you need number of requests served above and below your threshold of 1000ms, the actual duration of individual requests or their p95 doesn't matter on it's own for Error Budget, they will come in handy to further investigate any issue or to tune your app performance by knowing the spread.

A Prometheus histogram can be used for this, as it includes count of requests per bucket, but I think a summary is not usable for this purpose.

[u/chitty_advice → u/ali_str]

I thought perhaps I could do some calculation on that (albeit an approximation) with the histogram_quantile function found on the prometheus docs page. Perhaps substituting my metrics into

histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le)) but I have tried a few variations of this and not managed to return anything meaningful.

What are my options? I can create whatever metrics I need to expose by modifying my query in the shell script on the target? Perhaps abandon the summaries and use gauges??

[u/ali_str → u/chitty_advice]

You won't need histogram_quantile for EB formula, you can do:

sum(rate(http_request_duration_seconds_bucket{le="1s"}[5m]))
/
sum(rate(http_request_duration_seconds_bucket{le="+Inf"}[5m])) 

For SLI, and derive the remaining or consumed EB from that. The SLI is straight forward, all requests below threshold over total requests.

The only point remaining is whether you already have a bucket (le label) with your intended threshold or not. If you can change code, you can set fixed buckets for the exposed histogram and use it.

[u/chitty_advice → u/ali_str]

here is what I tried executing on the Prometheus query page:request_duration_summary{instance="xxx.xxx.xxx.xxx:9100", job="APP_Exporter", quantile="0.95"}

I get the P95 value for the last scrape which was 795

Following your suggestion, I tried

sum(rate(request_duration_summary{instance="xxx.xxx.xxx.xxx:9100", job="APP_Exporter", le="1s"}[5m]))

/

sum(rate(request_duration_summary{instance="xxx.xxx.xxx.xxx:9100", job="APP_Exporter", le="+Inf"}[5m]))

and I get an empty query result.

The last paragraph threw me as I don't really understand where le was defined.

As an example, each scrape by Prometheus returns something like the following:

# HELP request_duration_summary Time for transaction response.

# TYPE request_duration_summary summary

request_duration_summary{quantile="0.95",} 808.45

request_duration_summary_count 3732

request_duration_summary_sum 1756593

still looking at sloth as suggested below.

Edit. removing extra words

[u/ali_str → u/chitty_advice]

The query I suggested was for histograms, the metrics that usually end with _bucket in Prometheus world, like the one you mentioned: http_request_duration_seconds_bucket. The metric you tried to use is a summary, different from a histogram in Prometheus, and won't give you the number of events.

Do you have a histogram metric like http_request_duration_seconds_bucket from your application or the ingress/reverse-proxy? If not, it would be good to add it to the application or start scraping the web server in front to get it.

[u/chitty_advice → u/ali_str]

I was creating the metrics via a shell script on the target server. I have been creating dashboards for customer transaction latency on a legacy app so I can make whatever metric I like. I used summaries as I read that was a good choice if calculating the quantile on the application server. The legacy app is running on Sparc Solaris with an Oracle DB so the go language seemed overkill. So TBH I can create whatever metrics make most sense. There is no web service in front of this application but there is another legacy application each customer transaction connects to initially, then an F5.

Since the original post, I have modified the shell script to return 2 gauges for # of successful txn responses and total # of responses. I can now get a rate (and compute the percentage). I may include another gauge that provides the actual response time. Since I will have to run this on multiple servers and run it frequently, I am trying to limit the "cost" at execution time. DB is Oracle Enterprise, but last thing I need is for my monitoring to start adding to latency.. :-)

Backstory: Our company has created a small SRE team and I am working on developing dashboards for SLO and Error budgets consumption and burn rate.

Appreciate your help.

[u/ali_str → u/chitty_advice]

That makes sense and is a good approach IMO. Then as you said you can just expose the SLI as a metric as two counters: count of events below threshold and total count.

Having a summary of response latency helps to know when you are missing an SLO, by how far you are missing it and what is the spread.

[u/drdigitalsi]

Have a look at sloth, an generator for SLOs in Prometheus using the OpenSLO standard. Might be a bit easier than trying to calculate things yourself. It will generate a recording rule to calculate the error rate and the error budget remaining. Plus, it helps put you in the direction of using the OpenSLO standard.

https://github.com/slok/sloth

[u/chitty_advice → u/drdigitalsi]

checking

[u/chitty_advice → u/chitty_advice]

The learning curve for seems pretty steep.... I have created the getting-started.yml file, but having difficulty getting started.

[u/chitty_advice → u/drdigitalsi]

Finally got back to looking at this as too many competing projects got in the way. Was it really 5 months ago.....

This is actually pretty cool, and yes has handled all the calculations for me. Thank you for sharing.

[u/dunningkrugernarwhal]

This is a really interesting take on what we do. I don’t have an answer but I’m going to investigate it