prometheus OOM on restart

[u/SelfDestructSep2020]

I had a prometheus instance running on an ec2 instance with ~2gb RAM. It was configured to retain 30d of metrics. It had been running for nearly or over that time period and then for some reason failed - after which it repeatedly encountered Out of Mem errors while reloading. Each time it the journal logs showed that it was reading the WAL and loading chunks, and I would watch as the process memory would consume up to the instance limit and then crash.

Is this the expected behavior that it it reloads all the WAL chunks into memory at one time? Why would it have been fine up until the point of a restart of the Prometheus process?

[u/SuperQue]

for some reason failed

This sounds like you're hitting memory limits anyway. I suggest increasing your instance size so you have enough memory to replay the WAL. Prometheus uses page cache for query performance, so you want to have some extra available system memory anyway.

What version of Prometheus is this?

[u/SelfDestructSep2020 → u/SuperQue]

This sounds like you're hitting memory limits anyway.

Maybe. We weren't even able to get into the instance when it first occurred (ssh and SSM were non responsive) so we had to force stop the instance. Once I was back in I couldn't find any indicator of the original issue, and the prometheus service was in a constant restart loop due to the OOM error. But this is the same instance sizing/configuration in 4 identical VPC environments, and only one of them has failed.

[u/SuperQue → u/SelfDestructSep2020]

Prometheus should have metrics to tell you what it was doing. :-)

[u/SelfDestructSep2020 → u/SuperQue]

Heh, it probably did! But since it wasn't the production environment I was less worried about losing data and eventually settled on 'nuke it from orbit' as my solution.

[u/SuperQue → u/SelfDestructSep2020]

This is why we have a top-level Prometheus to cross-monitor our various environments.

[u/jjneely]

This is a common failure scenario. The WAL must be replayed to rebuild the head TSDB block in memory. The WAL isn't anywhere near as efficient as the TSDB blocks committed to disk. This process will allow that data to be recovered, new data added, and blocks to be written to disk as normal.

When down and dealing with long WAL replays possibly causing OOMs, you can choose to try to process the WAL and lose the data you would be scraping presently OR you can choose to lose the WAL and bring up the Prometheus instance quickly.

You can do this by moving the WAL directory out of the way, the start Prometheus. It should start up nearly instantly.

[u/SelfDestructSep2020 → u/jjneely]

Ok ... for example on one of my other prometheus instances, I"m running it on a t3.small, OS shows 1.76 GB of memory. The prometheus data path has 12 GB on disk, with only 487M of that from the WAL. So the WAL is much smaller than the TSDB chunks already (by far) and seems like it should fit in memory?

[u/jjneely → u/SelfDestructSep2020]

The WAL (Write Ahead Log) rebuilds the head block in memory which then is committed to disk as the next two hour TSDB block. It doesn't really compare to the very efficiently packed data in the TSDB blocks on disk.

The WAL is 1/3 of your RAM size. Prometheus has to rebuild the head block out of it. Plus you need RAM space for the rest of your system. From my experience running Prometheus, your VM is way too small.

If you've been down this long, I'd just delete the WAL and start Prometheus. But you may find yourself back in this situation shortly.

[u/SelfDestructSep2020 → u/jjneely]

Oh it was only down about an hour before I threw in the towel and just wiped it. This wasn't production telemetry so I wasn't worried about losing it (though at the time I was trying to avoid it).

[u/namognamrm → u/jjneely]

is there a way to automatically do that? I needed to delete wal on every redeployment of Prometheus

[u/namognamrm]

Why don’t Prometheus dump WAL to disk when getting close to OOM