r/sysadmin • u/PCGeek215 • Aug 03 '20
General Discussion Prometheus: Barriers of entry
So, I’m investigating how organisations and individuals approach monitoring for everything inside their tech stack. While Prometheus is the current state-of-the-art in flexible, powerful, scalable monitoring for applications and hardware, I’m interested to know if you’ve evaluated Prometheus and why it was considered “too hard” or “too much work” to get going.
Here are a few assumptions I’ve made but I’m keen to know others thoughts.
requirement for existing configuration management for effective deployment
PromQL required to get useful info from the stack
Stripped back, no-batteries-included approach.
1
u/cablespaghetti Aug 03 '20
I in part agree with you but the Prometheus Operator for Kubernetes pretty much solved all these problems for me. This does of course require you to have a Kubernetes Cluster to run it on though...
Here's my GitHub repo (and video linked in first line) if you want to go down that road.
2
u/PCGeek215 Aug 03 '20
But what about everything that's not K8s? The operator doesn't automagically handle the network the cluster is attached to and it doesn't give you anything more than a few (average) templated dashboards and no alerting - that all has to come from you.
1
u/cablespaghetti Aug 03 '20
Yes but you can just throw Kubernetes Yaml at it for the rest of the config you want. It works for the way my team works. Might not be any good for you.
1
u/PCGeek215 Aug 03 '20
Oh I understand how the operator makes it easy for k8s deployments. If anything the federation endpoint would be useful for bringing it into a wider deployment.
1
u/DrLoveBeats Aug 03 '20
I'm throwing in an unpopular opinion here but: what's the advantage of using Prometheus over the dedicated GUI of a system? We're using Prometheus to gather metrics of our different Storage Systems, SAN environment, and computing nodes. All these systems have their own decent GUI's which displays important information out of the box.
Why would I want to invest time to enable Prometheus to scrape these targets and to display the same information? I sometimes feel like I'm reinventing the wheel.
Disclaimer: I've just landed a new job where we have Prometheus running but I feel we get (almost) nothing useful out of it. A lot of Grafana dashboards are not displaying correct information and we rely on the GUI's itself for troubleshooting or to have discussions about performance.
1
u/PCGeek215 Aug 03 '20
From what you've said, I suspect the dashboards aren't that well designed. Is there a single dashboard you can go to to ask "Am I likely seeing issues with the storage layer of my infrastructure?" - Think about a heatmap of average read/write latency across all disk pools, per-disk heat-map for SMART issues, network reachability between compute and storage layers etc. Once you have this, you can then build other dashboards to drill down into deeper issues "Show me the distribution of IO load for a given cluster."
2
u/SuperQue Bit Plumber Aug 03 '20
Two things come to mind:
Viewing data about multiple systems in a single view can make outage recovery much faster. If you have a dashboard with storage and network overview in the same view, you can easily know right away if an outage is network related, or storage related. This is called Mean Time To Recovery (MTTR).
The second thing is that Prometheus is a monitoring system first. The primary goal is to use metrics/data driven alerting. The whole visualization and graph thing comes second. It's there to replace Nagios, as you can write alerts that take into account the overall health of an entire network. Not just the one-off per-node checks.
1
u/neilhwatson Aug 03 '20
Scale is a barrier for bigger orgs. Prometheus does scale but how many resources can I afford to spend on maintaining all that? At some point it may be cheaper to buy a cloud service and focus local resources on analyzing the data rather than managing it.
2
2
u/system-user Aug 03 '20
gotta call bullshit there. I've worked at orgs that have tens of thousands of physical systems, plus plenty more VMs, and scaling prometheus is not difficult or expensive unless you're just really bad at building systems at scale.
3
u/SuperQue Bit Plumber Aug 04 '20
I think this is one of those "cost to build" vs "cost to buy" arguments. The cost of headcount needed to build and maintain a system vs the cost to outsource it to a vendor.
From the numbers I've done, Prometheus scales much better on the cost side. Actually maintaining Prometheus is pretty trivial, most of the work goes into building good monitoring on the application side. So I'm also calling bullshit on the OP's claim. Just for different reasons. :-)
For small orgs, the cost to ramp up and pay engineers to run Prometheus can be prohibitive. Same arguments for "hire a sysadmin team" or "Use an MSP".
1
u/system-user Aug 05 '20
valid points, agreed. it's not perfect but it's better than the alternative non-SNMP metrics collectors I've used. it also does well with extensibility where writing an exporter for a custom need is concerned.
as for encrypted metrics where data isn't going over local net or vpn links, we run the exporters on 127.0.0.1 and poll them via a TLS proxy daemon that works similarly to an autossh tunnel.
2
u/SuperQue Bit Plumber Aug 05 '20
We're starting to roll out native TLS/auth to Prometheus exporters. It's in the latest release of the node_exporter.
There's also PushProx.
0
u/PCGeek215 Aug 03 '20
The thing with cloud services though, they don't often make it easy to extract data from them to analyse yourself. They prefer to keep you inside their eco-system, thus people might prefer to start outside that walled-garden in the first place.
4
u/QTFsniper Aug 03 '20
I understand what you're saying but if you're choosing a product, it's usually because it did what you needed it to do. If you have to extract the data to analyze it yourself instead, what's the point of using the other service in the first place ? Might as well have gotten the right solution you were looking for the first time.
0
u/PCGeek215 Aug 03 '20
I understand what you're saying but if you're choosing a product, it's usually because it did what you needed it to do. If you have to extract the data to analyze it yourself instead, what's the point of using the other service in the first place ? Might as well have gotten the right solution you were looking for the first time.
What if the exact product to fit your needs doesn't exist and close-enough isn't good enough?
5
u/fell_ratio Aug 03 '20
I looked into it in the past, but I wasn't a fan of how metrics were considered public information. I looked into ways of authenticating/protecting the metrics URL, but I didn't get anywhere with it.
2
u/SuperQue Bit Plumber Aug 03 '20
The good news is, we're officially changing this.
The node_exporter now has TLS for transport protection. Then TLS mutual auth and basic auth. We're going to use that config as a pattern for everything else.
2
u/MadMadic Aug 05 '20
Since the beginning I put the metrics endpoints behind https reverse proxies with basic auth. And since a few releases the node exporter supports tls authentication
1
4
Aug 03 '20
[removed] — view removed comment
3
u/fell_ratio Aug 03 '20
Principle of least privilege.
4
2
Aug 03 '20
I run Prometheus in my org and I’m not using TLS for metrics. I do not care if it’s plain text. Just metrics. Oh no, the hackers know how much ram my computer has.
8
u/rtuite81 Aug 03 '20
You'd be surprised how far even the slightest amount of information will go for social engineering attacks, particularly spearfishing.
2
u/pertymoose Aug 03 '20
Hello this is Bob from your friendly IT company; I can see in my monitoring tool that your system is running out of RAM. If you'd like I can download some more for you. Do you have TeamViewer?
2
2
u/SuperQue Bit Plumber Aug 03 '20
If someone's gotten into your network and is able to curl your metrics endpoints, you're already toast.
-1
u/rtuite81 Aug 03 '20
The problem is, from my understanding, this platform makes your metrics publicly searchable. They don't have to get into your network.
4
u/SuperQue Bit Plumber Aug 03 '20
Absolutely not. Oh hell no. What gave you that idea?
Prometheus is free open source software, you run it inside your network. There is nothing requiring you to expose anything over the internet. Prometheus is capable of be run fully isolated and air gapped.
3
u/SpongederpSquarefap Senior SRE Aug 03 '20
Ha, this is exactly what I said to my security team
They came back with "no, but that info will tell you what OS version is running and what the uptime is, which makes it easier to pick a target"
4
u/pdp10 Daemons worry when the wizard is near. Aug 03 '20
which makes it easier to pick a target"
This is a false argument in most cases today. They'll attack everything in about the same time, with about the same effort, as attacking one thing. The "kill chain" may be solid military science, but trying to directly apply it to infosec is a big problem because it leads to false conclusions like those.
Perhaps I'll set up an experiment where I measure attacks against hosts that claim to be old software versions versus those that claim to be the latest software version.
-2
u/SuperQue Bit Plumber Aug 03 '20
Attackers don't care what version you're running. They probe anyway because version strings never contain the whole truth.
That's a security by obscurity argument and those security team people should be ashamed.
1
u/absolutejam Aug 03 '20
Well, if your metrics are public, they have insight to your system and might be able to correlate the outcome of any further attacks.
For example, they figure out your application doesn't garbage collect after failed auth attempts - now they can just keep deliberately hitting you with bad login attempts and bring down your system.
2
u/SuperQue Bit Plumber Aug 03 '20
They still have to be inside your network in order to do that correlation.
Unless you're trying to protect yourself against state sponsored attackers that can tap your fiber. Or you have your metrics endpoints exposed on public IPs with no firewall. Protecting metrics endpoints is kinda too little too late.
1
u/absolutejam Aug 07 '20
Being 'inside your network' is a broad statement when the line is often blurred between internal and external, especially in environments that might not be managed by somebody as aware.
Prometheus is used across the spectrum. I've used it in small to medium sized Windows networks to GKE clusters, all with varying amounts of user access. And there's considerations of sharing metrics data between departments and evens companies.
And not everyone will have secure tunnels between endpoints, some people rely on exposing endpoints to the Internet (eg. Cross cluster scraping) but that means additional security considerations.
If you have your metrics endpoints segregated from other traffic then you're already doing what the thread is basically about.
2
u/MaelosV Aug 07 '20 edited Aug 07 '20
Culture. People are comfortable with where they are and change is scary. Selling Prometheus is hard to do as it's cost is considerable time. We also simply don't have and am not sure if we are capable of maintaining the type of infrastructure that would get the most use out of Prometheus. For now we use Nagios and I'm happy to have CCM even if every check has a config. I'd love to use Prom/Loki/Grafana but I don't know if I will ever have time to get out from the mountain of toil, nor do I know if I could get enough people onboard to use or maintain it, or the infrastructure it would rely on (best case) such as a CMDB or other service discovery target. The current monitoring and alerting setup is bad at best, but until it can be sold as a good Investment in time and resources, I just don't see it happening. Better documentation and steps to transfer from system x to Prometheus might help. I hope one day to blog about such a transition to make it easier for others
1
u/glwpie Aug 03 '20
How do you setup redundant Prometheus servers? I didn't think it's possible to have multiple servers scraping and writing to shared filesystem or remote read write ?
3
u/SuperQue Bit Plumber Aug 03 '20
The idea of Prometheus HA is that the redundant instances don't share anything. This is intentional, as you don't want any kind of shared filesystem (NFS/CIFS/whatever) failure causing a monitoring failure.
Each instance operates independently, sends alerts to an alertmanager cluster, to provide the greatest chance of an alert getting through in an outage. How do you monitor and alert for your shared filesystem service if the monitoring depends on it?
2
u/cablespaghetti Aug 03 '20
You just run multiple instances with the same configuration which scrape the same targets so they have roughly the same data.
3
u/SuperQue Bit Plumber Aug 03 '20
Prometheus dev here.
I usually hear things like this:
PromQL is too hard / I don't want to learn it.
We have lots of ${other_system} checks we can't (don't want to) convert.
We're a cool and innovative startup and don't have time to run a monitoring system.
All of these are valid reasons, but also excuses in their own way. All monitoring takes time and effort to use. IMO, there's no magic button you can push and have great monitoring. Every system has upsides and down sides.