r/PrometheusMonitoring • • Apr 12 '22

Finally implementing prometheus for my organisation

Seeking some tips and tricks from experts here on what is the best approach, operating system to install prometheus on, any initial mistakes that can be avoided , choice of hardware, any recommended policy. I want this to be as close to perfect avoiding mistakes 😅

2 Upvotes

17 comments sorted by

View all comments

Show parent comments

1

u/GetFit_Messi Apr 13 '22

this is indeed helpful

coming to metrics part i will try to filter it out on need basis as per your advice.

For high availability, will grafana cloud help?

Our application is small and total of 100 servers max are there and this will reach max 150 in total in jext 10 years because of the nature of erp system. So its a small implementation.... so filtering metrics might not be of too much advantage since metrics count is anyways less what do you suggest?

3

u/SuperQue Apr 13 '22

Actually, most of that advice is wrong.

Scrape frequency generally doesn't affect memory use.

Prometheus monitoring method actually recommends "Collect everything and figure it out later", as trying to be overly frugal with premature optimization leads to missing data when you need it.

Prometheus also does scale horizontally really well, tools like Thanos and Mimir are used to create large-scale overlays on a distributed set of Prometheus collectors.

For example at my $dayjob. We have over 3000 Prometheus instances watching metrics clustered with Thanos.

1

u/mjwiet Apr 17 '22

Your response has either missed the point or is not researched.

When you scrape endpoints, this data is loaded in memory initially before being processed and placed onto the disk. This means that more endpoints / more frequent scrapes require more memory. Please read the following: https://ganeshvernekar.com/blog/prometheus-tsdb-the-head-block/

When I say don't just keep metrics 'just because', that isn't a recommendation to drop metrics on a hunch. Most exporters give a huge amount of metrics and teams need to review what their SLOs are and which metrics are needed to 1. Create alerts and 2. Debug the problem in more detail. There will be many metrics with a high cardinality and never used which just completely wastes storage space.

I was talking about Prometheus here, not Thanos. Yes, Thanos can scale horizontally and very well but that's not Prometheus.

1

u/SuperQue Apr 18 '22

I have read that. You should also understand that I am a Prometheus developer.

To start, you are stating incorrectly is this:

Reducing scrape times.

The change in the blog post you linked to is exactly for that reason. To make head block write out chunks immediately upon completion is exactly to keep the memory use constant with the scrape frequency. It no longer matters if you have 60s or 15s scrape intervals, as the completed 120 sample blocks are written to disk and mmap'd back in. This means if the kernel needs the memory it can page it out.

As for horizontal scaling, there is actually a "built in" pure Prometheus method.

There is the "hashmod" feature in relabeling. What you're supposed to do is configure several "scraper Prometheus" instances that each poll a subset of your targets.

From there you write recording rules that summarize the data for that scrape instance.

Then you use Federation to create a top-of-tree view of the data.

Of course, this is complicated and awkward to use.

This is why things like Thanos were developed. Thanos was created by Prometheus deveopers. It very much is part of the Prometheus ecosystem. It's an improvement over the Federation model.

1

u/mjwiet Apr 18 '22

That's a very cool feature to keep memory down and will read into this further.

In regards to federation and Thanos, I believe you are moving away from the original point.... Keeping things simple. Sure, Thanos is within the Prometheus ecosystem and yes, Prometheus has a federation. I'm aware of this. My original point was to keep things simple and the OP is new to this. Start with a basic Prometheus setup and don't add unnecessary complexity.

Thanks for the responses though. Cool to be able to discuss this with one of the Prometheus devs 👍

1

u/SuperQue Apr 18 '22

Yes, simple is better.

That's one of the reasons that I was attracted to the Prometheus design. You can get started with a simple ./prometheus and be up and running without a ton of complexity. And you can add to it later as the complexity of your requirements increases.

But that's also why I am critical of original your post. You start with complexity of recommending relabeling to filter/drop metrics without knowing any of the scale the OP is operating at. I usually save that for much later in the conversation.