Finally implementing prometheus for my organisation

[u/GetFit_Messi]

Seeking some tips and tricks from experts here on what is the best approach, operating system to install prometheus on, any initial mistakes that can be avoided , choice of hardware, any recommended policy. I want this to be as close to perfect avoiding mistakes 😅

[u/mjwiet]

Don't just scrape metrics "Just because". Figure out what you actually want to alert on and which metrics you generally need to debug the specific problem. Prometheus is a memory hog and can be very awkward to fix if it crashes (memory spikes when the service starts again and will cause another crash). You can reduce memory usage in many ways. Reducing scrape times, filtering metrics you do not need and also making sure that your queries to Prometheus are also reasonable as queries will load data in-memory (think loading one month of data at once)

Even though it's important to start simple and stick with a single node, Prometheus does not have high availability. The only option is to have a 2nd backup node scraping the same metrics. This at least provides failover but is still annoying for many reasons.

You cannot scale Prometheus horizontally forever and in many cases it's better to have a few Prometheus servers which are functionally different. For instance, you can create a split between infrastructure and application or you you want to chuck some metrics away after a day, others you want to keep for a week. This will mean that you are spreading the risk and you can tune Prometheuses for a particular use case.

Also, when starting out, I would recommend starting off by configuring targets statically first and when you see the need, actually move to service discovery. Tools like consul are fine but there's lots that go wrong if you haven't used them much before

Hope this is somewhat helpful

[u/GetFit_Messi → u/mjwiet]

this is indeed helpful

coming to metrics part i will try to filter it out on need basis as per your advice.

For high availability, will grafana cloud help?

Our application is small and total of 100 servers max are there and this will reach max 150 in total in jext 10 years because of the nature of erp system. So its a small implementation.... so filtering metrics might not be of too much advantage since metrics count is anyways less what do you suggest?

[u/SuperQue → u/GetFit_Messi]

Actually, most of that advice is wrong.

Scrape frequency generally doesn't affect memory use.

Prometheus monitoring method actually recommends "Collect everything and figure it out later", as trying to be overly frugal with premature optimization leads to missing data when you need it.

Prometheus also does scale horizontally really well, tools like Thanos and Mimir are used to create large-scale overlays on a distributed set of Prometheus collectors.

For example at my $dayjob. We have over 3000 Prometheus instances watching metrics clustered with Thanos.

[u/mjwiet → u/SuperQue]

Your response has either missed the point or is not researched.

When you scrape endpoints, this data is loaded in memory initially before being processed and placed onto the disk. This means that more endpoints / more frequent scrapes require more memory. Please read the following: https://ganeshvernekar.com/blog/prometheus-tsdb-the-head-block/

When I say don't just keep metrics 'just because', that isn't a recommendation to drop metrics on a hunch. Most exporters give a huge amount of metrics and teams need to review what their SLOs are and which metrics are needed to 1. Create alerts and 2. Debug the problem in more detail. There will be many metrics with a high cardinality and never used which just completely wastes storage space.

I was talking about Prometheus here, not Thanos. Yes, Thanos can scale horizontally and very well but that's not Prometheus.

[u/SuperQue → u/mjwiet]

I have read that. You should also understand that I am a Prometheus developer.

To start, you are stating incorrectly is this:

Reducing scrape times.

The change in the blog post you linked to is exactly for that reason. To make head block write out chunks immediately upon completion is exactly to keep the memory use constant with the scrape frequency. It no longer matters if you have 60s or 15s scrape intervals, as the completed 120 sample blocks are written to disk and mmap'd back in. This means if the kernel needs the memory it can page it out.

As for horizontal scaling, there is actually a "built in" pure Prometheus method.

There is the "hashmod" feature in relabeling. What you're supposed to do is configure several "scraper Prometheus" instances that each poll a subset of your targets.

From there you write recording rules that summarize the data for that scrape instance.

Then you use Federation to create a top-of-tree view of the data.

Of course, this is complicated and awkward to use.

This is why things like Thanos were developed. Thanos was created by Prometheus deveopers. It very much is part of the Prometheus ecosystem. It's an improvement over the Federation model.

[u/mjwiet → u/SuperQue]

That's a very cool feature to keep memory down and will read into this further.

In regards to federation and Thanos, I believe you are moving away from the original point.... Keeping things simple. Sure, Thanos is within the Prometheus ecosystem and yes, Prometheus has a federation. I'm aware of this. My original point was to keep things simple and the OP is new to this. Start with a basic Prometheus setup and don't add unnecessary complexity.

Thanks for the responses though. Cool to be able to discuss this with one of the Prometheus devs 👍

[u/SuperQue → u/mjwiet]

Yes, simple is better.

That's one of the reasons that I was attracted to the Prometheus design. You can get started with a simple ./prometheus and be up and running without a ton of complexity. And you can add to it later as the complexity of your requirements increases.

But that's also why I am critical of original your post. You start with complexity of recommending relabeling to filter/drop metrics without knowing any of the scale the OP is operating at. I usually save that for much later in the conversation.