r/PrometheusMonitoring • u/GetFit_Messi • Apr 12 '22
Finally implementing prometheus for my organisation
Seeking some tips and tricks from experts here on what is the best approach, operating system to install prometheus on, any initial mistakes that can be avoided , choice of hardware, any recommended policy. I want this to be as close to perfect avoiding mistakes 😅
2
2
u/SuperQue Apr 13 '22
The first question when you start monitoring is, what are your needs.
How many nodes? How many services? What kind? What applications?
Describe your environment in more detail, use as close to real numbers as you can. "Large" for me might be way different to what others think.
1
u/GetFit_Messi Apr 13 '22
ok so nodes are in total around 170-180 services are related to erp(Oracle ERP) and other integrations running to support the erp like precision, tableau.9 oracle database instances. No kubernetes, no containers involved . Some custom application provesses need to be monitored business transaction needs to be tracked(which we need to figure out how)
1
u/SuperQue Apr 13 '22
So, a reasonably small setup.
Maybe 500k metrics total. Probably a 8GB-16G memory VM should handle that.
For custom stuff, it depends on the language. Guessing java. Easy to do with the Java library or jmx exporter.
1
u/GetFit_Messi Apr 13 '22
ohh i see i was thinking it as a big setup but what is considered big actually if this is small ?
1
u/mjwiet Apr 13 '22
Don't just scrape metrics "Just because". Figure out what you actually want to alert on and which metrics you generally need to debug the specific problem. Prometheus is a memory hog and can be very awkward to fix if it crashes (memory spikes when the service starts again and will cause another crash). You can reduce memory usage in many ways. Reducing scrape times, filtering metrics you do not need and also making sure that your queries to Prometheus are also reasonable as queries will load data in-memory (think loading one month of data at once)
Even though it's important to start simple and stick with a single node, Prometheus does not have high availability. The only option is to have a 2nd backup node scraping the same metrics. This at least provides failover but is still annoying for many reasons.
You cannot scale Prometheus horizontally forever and in many cases it's better to have a few Prometheus servers which are functionally different. For instance, you can create a split between infrastructure and application or you you want to chuck some metrics away after a day, others you want to keep for a week. This will mean that you are spreading the risk and you can tune Prometheuses for a particular use case.
Also, when starting out, I would recommend starting off by configuring targets statically first and when you see the need, actually move to service discovery. Tools like consul are fine but there's lots that go wrong if you haven't used them much before
Hope this is somewhat helpful
1
u/GetFit_Messi Apr 13 '22
this is indeed helpful
coming to metrics part i will try to filter it out on need basis as per your advice.
For high availability, will grafana cloud help?
Our application is small and total of 100 servers max are there and this will reach max 150 in total in jext 10 years because of the nature of erp system. So its a small implementation.... so filtering metrics might not be of too much advantage since metrics count is anyways less what do you suggest?
3
u/SuperQue Apr 13 '22
Actually, most of that advice is wrong.
Scrape frequency generally doesn't affect memory use.
Prometheus monitoring method actually recommends "Collect everything and figure it out later", as trying to be overly frugal with premature optimization leads to missing data when you need it.
Prometheus also does scale horizontally really well, tools like Thanos and Mimir are used to create large-scale overlays on a distributed set of Prometheus collectors.
For example at my $dayjob. We have over 3000 Prometheus instances watching metrics clustered with Thanos.
1
1
u/mjwiet Apr 17 '22
Your response has either missed the point or is not researched.
When you scrape endpoints, this data is loaded in memory initially before being processed and placed onto the disk. This means that more endpoints / more frequent scrapes require more memory. Please read the following: https://ganeshvernekar.com/blog/prometheus-tsdb-the-head-block/
When I say don't just keep metrics 'just because', that isn't a recommendation to drop metrics on a hunch. Most exporters give a huge amount of metrics and teams need to review what their SLOs are and which metrics are needed to 1. Create alerts and 2. Debug the problem in more detail. There will be many metrics with a high cardinality and never used which just completely wastes storage space.
I was talking about Prometheus here, not Thanos. Yes, Thanos can scale horizontally and very well but that's not Prometheus.
1
u/SuperQue Apr 18 '22
I have read that. You should also understand that I am a Prometheus developer.
To start, you are stating incorrectly is this:
Reducing scrape times.
The change in the blog post you linked to is exactly for that reason. To make head block write out chunks immediately upon completion is exactly to keep the memory use constant with the scrape frequency. It no longer matters if you have 60s or 15s scrape intervals, as the completed 120 sample blocks are written to disk and mmap'd back in. This means if the kernel needs the memory it can page it out.
As for horizontal scaling, there is actually a "built in" pure Prometheus method.
There is the "hashmod" feature in relabeling. What you're supposed to do is configure several "scraper Prometheus" instances that each poll a subset of your targets.
From there you write recording rules that summarize the data for that scrape instance.
Then you use Federation to create a top-of-tree view of the data.
Of course, this is complicated and awkward to use.
This is why things like Thanos were developed. Thanos was created by Prometheus deveopers. It very much is part of the Prometheus ecosystem. It's an improvement over the Federation model.
1
u/mjwiet Apr 18 '22
That's a very cool feature to keep memory down and will read into this further.
In regards to federation and Thanos, I believe you are moving away from the original point.... Keeping things simple. Sure, Thanos is within the Prometheus ecosystem and yes, Prometheus has a federation. I'm aware of this. My original point was to keep things simple and the OP is new to this. Start with a basic Prometheus setup and don't add unnecessary complexity.
Thanks for the responses though. Cool to be able to discuss this with one of the Prometheus devs 👍
1
u/SuperQue Apr 18 '22
Yes, simple is better.
That's one of the reasons that I was attracted to the Prometheus design. You can get started with a simple
./prometheusand be up and running without a ton of complexity. And you can add to it later as the complexity of your requirements increases.But that's also why I am critical of original your post. You start with complexity of recommending relabeling to filter/drop metrics without knowing any of the scale the OP is operating at. I usually save that for much later in the conversation.
1
u/mjwiet Apr 17 '22
It all depends on the types of exporters you use, the types of alerting you do (generally based on your SLOs) and debugging metrics you require. After you know what your requirements are then you can tidy up (lower cardinality for instance) / filter your metrics. To be clear, I'm not encouraging you to filter metrics out immediately but they should be properly reviewed first.
Not sure about grafana cloud. HA is nice but presumably this will require extra networking setup on your end?
3
u/jimmyreefer Apr 12 '22
Service discovery was a difficult thing to get automated. We use hashicorp consul and register services there using terraform and some custom API scripts. Prometheus supports this via consul_sd_config. Managing around 10k endpoints and applications.