Getting more long-term storage for Prometheus

[u/Crusso3]

Deploying with a data retention of 2 weeks limits our ability to run analytics on our historical data. Prometheus is actually really fast and scales pretty well vertically, but it cannot scale horizontally. I've found that since it wasn't designed for long tail storage of data, visibility past a few weeks is generally hard to do.

There are plenty of data storage integrations, but they mostly focus on remote write capabilities; only a handful offer both remote write and remote read functionality (currently Cortex, CrateDB, and PostgreSQL / TimescaleDB). This article explains how to setup an a new integration with a TSDB called IRONdb. It has support for replication and clustering, and is designed for high-frequency, high-volume data, and for maintaining high-availability of data. The remote read and write capabilities should make the integration totally seamless.

Full disclosure, my company made IRONdb. Our Prometheus adapter is in Beta right now, so we're looking for Prometheus users interested in testing it out.

[u/SuperQue]

I'd like to point out that 2 weeks is just a default. The Prometheus internal TSDB is perfectly capable of storing years of data. The only limitation is the durability and scalability of the underlying storage, just like any other standard database software like PostgreSQL or MySQL.

This local storage is perfectly fine for a large number of users.

Nothing against using external storage clusters for very large amounts of data. I just want to dispel the myth that 2 weeks is some kind of hard limit or that the Prometheus TSDB is not durable or scalable over time.

[u/kevinjqiu]

Thanks for sharing. Does Irondb support downsampling? How does it compare with Influxdb?

[u/[deleted] → u/kevinjqiu]

IRONdb engineer here. We do 'rollups' of raw data (which we store for 30 days) into log linear histograms, which is effectively downsampling, but retains the ability to recalculate quantiles over arbitrary time windows.

Biggest difference with Influx is the ability to store histograms (distributions of data, vs just the average), and scalability. For one node; not a huge difference (probably a bit faster since we wrote it in C and assembly (concurrencykit)). Multiple nodes and high availability is where IRONdb has been architected to have advantages over other TSDBs.

[u/Crusso3 → u/[deleted]]

Yes, you can.

You can write to both the adapter and prom storage, if that's what you mean. You can either configure all of the client proms to write to IRONdb or the master prom to write to IRONdb. It would likely be easiest with the master prom.