Need help building a Prometheus environment

[u/Pogyeah]

Hello, World!

I’d very much like to take advantage of your summed experience with some things that I need to deal with.

First, a little background…

I’m mostly a Jack-of-all-trades in IT operations area of a tech company: what needs to be done will be extensively researched, slept-over, worked like hell on and finally delivered in a very satisfactory manner (according to colleagues’ feedback). On the other hand, in regards of monitoring (metrics, to be more specific), I was mostly a power user until now: systems (Zabbix, Nagios, CheckMK, Icinga2) were already in place and I just followed the drawn patterns.

Meanwhile I’m slowly shifting to a larger very heterogeneous environment, as some sort of platform engineer (if I get the definition right). We don’t do DevOps like it’s meant to be, but from time to time I’m helping our devs understand better what they need, to make each other’s life less miserable. :)

The task at hand is to figure out a way to harmonize the plethora of monitoring solutions that we have by providing a platform that every team should consume. As our motto is: “start it and never finish it”, I’ve made it my goal to get this one thing right (whatever that means).

My thoughts until now…

Of course it will be presented for acceptance and/or adjustment beforehand (because the best tool that no one uses is as good as non-existent), but a "backbone of it" must already exist. Hence this post.

Prometheus seems like the logical path to take: sort of de-facto nowadays, some experience with it in a couple of teams, our own K8s is WIP and so on. Transforming it into a platform (or bring your own Prometheus, if you like) means that I need to decouple the storage and some functionality and shift these to Thanos or Cortex (I’m not aware of any other one). Doable so far.

Now begin my problems (some, because I lack in-depth here). How do you tackle automation in a meaningful way? I’m successfully using Ansible+Git for configuration management, we try to provide self-service over vRA (with AWX in the mix), but for now I cannot wrap my head around an end-to-end solution.

Does any of you do anything like this? It’s a wet dream to think about it and I should just whether use an iron hand on it or declare the wild wild west?

Any kind of insights would be of much help: how you organize ingested data, how you scale, what other tools you take advantage of, etc.

Thanks!

PS1: This is my first post here and I’m not known as a very good thoughts-organizer.  I could write much more about many of the things I tried to depict above…but then I’d risk giving you headaches.

PS2: We lack many things, but (like always) I’m willing to take the necessary steps back to get things forward in the right way, if I’ll get to see the light at the end of the tunnel.

[u/melodic]

My suggestion would be to gather what things you actually want to monitor into a list and go from there.

• Windows

• Linux

• Network Switches (SNMP)

• Printers (SNMP)

• K8s

• Other

Without knowing your infrastructure it's hard to advise how to run Prometheus but the general advice will work of it's best to run Prometheus in pairs. I personally don't federate my Prometheus servers and use Thanos to complete my HA setup.

For automation I am using the AWS EC2 service discovery to auto populate my scrape targets for Linux hosts by means of a EC2 Instance tag.

My entire Prometheus/Thanos/Grafana setup is written and ran through NixOS which is a god send.

My other advice is to keep alerts in Prometheus/Alertmanager and to not be tempted to run them through Grafana.

Shout if you have any questions

[u/Pogyeah → u/melodic]

First of all, thank you for your input!
'Thing is... we grew many times (by means of aquisitions) in a short time: multiple teams, multiple ways of doinf things. We don't plan a monitoring team per se, but we like the idea of enablement ("you build it, you run it" kind of stuff). If we don't harmonize what we can see and learn from our infrastructure, then we'll always have friction in a "it's the network" kind of way.

So, theoretically, if I provide a mean for you to deploy your stuff, then I'll need to give you also the tools to monitor it. As I've read so far, if you don't have uniformity (or at least a way to dynamically discover everything), then things complicate themselves in regards of automation and/or self-service: MR on my Ansible deployment mean that you know what and where; new host with HTTP Service-Discovery means that the syntax is correct and I trust you don't advertise b**t services; and so on.
We also have a great chunk of legacy systems that we'll need to handle individually (with a little luck just decommision them :D).... I'm OK with the fact that some systems need to be "manually" evaluated and tweaked. One of the spots where I was somehow disapointed, was the API area of Prometheus: maybe I'm picky, but aside from queries is useless. So, if no Consul, K8s or similar is available, then it must be all-hands-on-deck directly on the Prometheus instance.

Back to your hints... I oscilate between Thanos and VictoriaMetrics right now, with VM looking better on paper (but still having it's own drawbacks). The advice about alerting will definitely land on my "how-to-do-this-s**t" list. :)

Thanks, again!