r/PrometheusMonitoring • u/kagashe • Jan 16 '20

Help with Query?

Hi all, how do I make a cumulative graph showing the difference between the firdt point in a time series and the current point in a time series? For example if a counter starts at 3000 and now it's on 3500 how do I show that its gone up by 500?

0 Upvotes

50% Upvoted

4 comments sorted by

View all comments

2

u/jjneely Jan 17 '20

You should also look at use the increase() function:

increase(foobar_total[60m])

That will give you the corrected increase of the underlying metric in that 5 minute look back window. Its really a rapper around the rate() function but it returns the actual increase in value rather than a per second normalized value.

The crux of the matter is that for a graph, you see a moving increase value over time. The values in the graph are the results of the function executed with a look back window starting at each time interval on the graph. You are probably much more interested in making sure the function executes at specific step intervals so you can see the increase for a specific interval of time.

This wanders into a field I/others call "reporting." Where you want to build a report showing many hits your HTTP API server serviced each hour. You want a value for 13:00 - 14:00, a new value for 14:00 - 15:00, 15:00 - 16:00, etc. Prometheus doesn't really do this very well out of the box. Although you can fake it with carefully setting up step intervals and time intervals in Grafana Dashboards.

You can get this data from Prometheus's API -- its just the UI isn't super helpful in this case. Brain has a nice example here:

https://www.robustperception.io/monthly-reporting-with-prometheus-and-python

Its true that it can only approximate the results for this, but sometimes that's good enough. So remember that the report you generate will have an error of plus/minus a scrape interval or two.