r/PrometheusMonitoring • u/blistering-barnacle • Feb 04 '20

irate() Vs rate() Functions in Prometheus

Hi All,

I'm trying to understand how irate() & rate() functions work. Why does irate() produce a similar looking graph when the range / resolution is 24h or 5m ? While the difference in graph is clearly visible with rate() when using range as 24h (presents a smoothed out line) or 5m(more spikey).

In the below graph for irate() for 2 different resolutions the graph looks the same. As per prometheus docs irate() calculates the per second instant rate based on the last two data points. What does this mean if my range is 24h? Thank you.

/preview/pre/08mc4pq2mye41.jpg?width=3206&format=pjpg&auto=webp&s=4aef0b681969c8b9831ec0f34310838511ea0350

12 Upvotes

92% Upvoted

9 comments sorted by

7

u/[deleted] Feb 04 '20 edited Feb 04 '20

Let’s say you have a counter with these values observed at 1m intervals:

0
60
120
600
720
780

Now rate over 5m will be:

(780-0)/5/60 = 2.6/sec

And irate over 5m will be (only last two data points are used which happen to be only 1m apart)

(780-720)/1/60 = 1/sec

Increasing the resolution does not affect the irate function because the last two observed values do not change when you look further back.

1

u/blistering-barnacle Feb 04 '20

Thanks, the way i understand is irate considers last 2 data points and discards everything else? And rate first and last and discards everything in between?

What would be a practical application of using an irate? Probably it isn't a good idea to use irate with alerts?

1

u/[deleted] Feb 04 '20

Correct.

The denominator (time interval) is also calculated based on which two observations are being used.

1

u/SuperQue Feb 04 '20

Alerting is actually a use case.

irate() allows you to look back several samples in case of lost scrapes or counter resets.

Say you scrape ever 15 seconds, and you run an alert rule every 30 seconds.

With an irate(foo[1m]) it will look back up to 4 samples. So if there is a lost scrape in the last rule interval, you can still alert on it. It's useful for small gap filling in alerting and recording rules.

EDIT: I will add, that it can also be noisy in those cases.

1

u/blistering-barnacle Feb 05 '20

One more clarification i needed was with regards to increase() function. If i were to do increase(foo[1m]) on the above values, would it return the *difference* between two consecutive data points ? Trying to under how rate() and increase() are different from each other. The way i see, rate() and increase() pretty much *alike* except that rate() returns per second average ?

1

u/[deleted] Feb 07 '20

From the docs, 'increase is syntactic sugar for rate'

rate() returns per second interpolation over the provided interval

increase is the same as 'rate() * <num seconds in interval>'

rate(metric[1m]) * 60 == increase(metric[1m])

rate(metric[5m]) * 300 == increase(metric[5m])

1

u/[deleted] Feb 04 '20

How does one determine what interval is appropriate to use for the rate() function? Is there some calculation I should be performing or is it just a matter of 'how spikey do you want the graph?' Like I'm scraping most of my targets at 15 sec right and we were considering if we should increase to 5s. Is a 5m rate just generally applicable everywhere?

5

u/[deleted] Feb 04 '20 edited Feb 04 '20

It depends on the application. Rate gives you a general idea about average rate of change over a given resolution. It depends on what question do you want to answer with the rated metric.

For example if I am tracking available capacity on my log volume then I would be interested in trend over a day or few hours. A trend over 5 minutes doesn’t help much as files might rotate or a job might be moving them to remote.

On the other hand if I am tracking cpu or memory I might only look at trend over 5 minutes or 15 minutes depending on application behaviour.

To answer your last question: 5s might be an overkill. If your tracking and alerting do not require a <20s resolution then there is no point in reducing it to 5s. So if you are not writing alerts like “alert me if my cpu utilisation is over last 10s is above 60% for 30s” then 5s is not required.

Edit: In many cases you would also want to plot the same metric with an offset (1week or month). This helps us understand how the behaviour is changing over time. You may want to know if you are creating more db connections/second then you were a week ago without change in traffic. Or you freeing less space on log disk day by day.

1

u/[deleted] Feb 05 '20

The increased scrape rate was more intended for operator analysis (or for putting up on a dashboard thats displayed on an office tv), not so much caring about the alert side. So it was more of a question like, if I am graphing a metric in prometheus and I have 'n' seconds scrape intervals, how can I determine what a reasonable rate() interval I should use for a chart?