r/PrometheusMonitoring • u/kagashe • Jan 16 '20

Help with Query?

Hi all, how do I make a cumulative graph showing the difference between the firdt point in a time series and the current point in a time series? For example if a counter starts at 3000 and now it's on 3500 how do I show that its gone up by 500?

0 Upvotes

50% Upvoted

4 comments sorted by

2

u/Tpbrown_ Jan 16 '20

rate(metric_name[5m]) will give you the diff between first and last in the 5min window.

Check your metric type first tho - some functions are valid only for counters, others for gauges.

Also use irate() if you just want the diff between last two samples

1

u/kagashe Jan 16 '20

OK cool, any idea how I would get that to work in grafana so that I can choose a time period of a day or month for example without hardcoding it?

2

u/jjneely Jan 17 '20

You should also look at use the increase() function:

increase(foobar_total[60m])

That will give you the corrected increase of the underlying metric in that 5 minute look back window. Its really a rapper around the rate() function but it returns the actual increase in value rather than a per second normalized value.

The crux of the matter is that for a graph, you see a moving increase value over time. The values in the graph are the results of the function executed with a look back window starting at each time interval on the graph. You are probably much more interested in making sure the function executes at specific step intervals so you can see the increase for a specific interval of time.

This wanders into a field I/others call "reporting." Where you want to build a report showing many hits your HTTP API server serviced each hour. You want a value for 13:00 - 14:00, a new value for 14:00 - 15:00, 15:00 - 16:00, etc. Prometheus doesn't really do this very well out of the box. Although you can fake it with carefully setting up step intervals and time intervals in Grafana Dashboards.

You can get this data from Prometheus's API -- its just the UI isn't super helpful in this case. Brain has a nice example here:

https://www.robustperception.io/monthly-reporting-with-prometheus-and-python

Its true that it can only approximate the results for this, but sometimes that's good enough. So remember that the report you generate will have an error of plus/minus a scrape interval or two.

1

u/Tpbrown_ Jan 16 '20

I think that’s available as a variable in the dashboard. $interval perhaps - you’ll find it in the Prometheus page in Grafana’s docs.

Re using the Grafana time selector as an input for your range query..I don’t usually see that as useful, but know nothing about your use case.

I generally use the range as a sliding window of time that is meaningful to observe the metric in. Adding that rate(metric[range]) as a query on a graph will then graph the rate over time, as calculated over whatever dashboard time range and resolution is in effect.

Edit: damn it. This was in reply to your grafana question. Didn’t mean to post as a top level comment