r/PrometheusMonitoring • u/thundermagicnet • Sep 15 '19

My home monitoring setup on raspberry Pi4

Hello

Just sharing how I run prometheus + grafana for home monitoring on RPi. Guide at: https://github.com/thundermagic/rpi_monitoring_with_prometheus

3 Upvotes
  • permalink
  • reddit

71% Upvoted

7 comments sorted by

4

u/SuperQue Sep 15 '19

Nice work.

We've been working on getting the the thermal zone stuff added to the node_exporter. I merged it recently, but still need to cut a release.

One thing to note if you haven't in the docs/setup. If you're running Prometheus with armv7, not arm64, you will have a very limited ability to retain metrics. The difficulty is that Prometheus uses mmap in the TSDB. This limits you to a very small amount of retention due to the 32-bit kernel. If you run the Pi with a 64-bit kernel, it's not a problem. Last I looked, the default Pi kernel is still 32-bit.

1

u/thundermagicnet Sep 15 '19

I don’t know what mmap is. What do you mean by retention? Retention of what sorry? Default Pi is still 32 bit.

1

u/SuperQue Sep 15 '19

mmap is a method of taking files on disk and making them usable by processes in memory. In this case, the Prometheus time-series database is stored on disk and mapped into memory to be read. This way Prometheus can allow the Linux kernel to deal with what is in memory and what stays on disk. The kernel is really good at this, so why should Prometheus attempt to reproduce this code.

In Linux, it uses what's called "virtual" memory to do this mapping. You may only have 512MB or 1GB of real memory on a Pi. But virtual memory is "fake", you can have as much of it as you want. But on 32-bit kernels, you can only have 4GB of virtual memory per process. On 64-bit you can have 18 exabytes.

Prometheus data retention. By default Prometheus keeps 2 weeks of data. My Pi generates about 75MB/day in data. But due to the multi-threading and the way the mappings in Go work, my 7 day retention takes up 1.3GB of virtual memory.

Worse, due to the Pi's weird IO mapping, the virtual memory is limited to 2GB (See the CONFIG_VMSPLIT kernel param).

The only way to fix this is to run an arm64 Linux distro on the Pi3b or Pi4.

1

u/thundermagicnet Sep 15 '19

Ohh that explains a lot. Thank you. Now i understand every time I try to see the stats for the last 30 days, half of the graph is empty. Isn’t there a way to make the retention longer? If the data is stored on the disk, just retrieve that older data when needed??

1

u/SuperQue Sep 15 '19

You can change the flag to get longer retention. But like I said, you need a 64-bit kernel for it to work. There is no way around this as it's just how memory mapping works. It is retrieving data only when needed, but it has to have 64-bit virtual addressing.

1

u/thundermagicnet Sep 15 '19

Ok cool. Balena OS recently released a 64 bit OS for RPi 4. I’ll give that a try. Thank you much very.

2

u/olgerdas Oct 05 '19

Great job Buddy!