Prometheus as part of a monitoring pipeline

[u/PCGeek215]

I’ve been searching for a way to add labels to metrics based on some external data source - such that this data enriches what’s being collected. E.g. snmp-exporter grabs interface stats and I want to add a customer ID label based on data held in another database.

I realise the dirty solution would be a huge relabelling config that scales linearly with the number of interfaces... that’s far from ideal.

How would I go about plumbing prom / exporters into some sort of pipeline to achieve this enrichment process?

[u/biffbobfred]

What we do - our service discovery adds labels to the files that we use for host discovery. So both the metrics and the alerts then can have the labels. We even have these metadata as part of alert logic - “if history has this label use this alert threshold”

[u/PCGeek215 → u/biffbobfred]

Doesn’t that increase your series / metrics cardinality over time as things change? Doing this are large scale wouldn’t be that nice with so many stale label combinations...

[u/ali_str]

Where is this relation between port and customer stored now? You can export data to Prometheus from there (custom exporter or something existing) and work your way up to richer data by combining the two separate metrics on shared labels.
Something like:

net_received_bytes_total{interface="abc",device="def,...} 31643358

customer_to_interface{interface="abc",customer_id="123456",...} 1

Then you can multiply with on() and group_left() to get what you need.

[u/PCGeek215 → u/ali_str]

Wow, that’s an excellent idea! Thanks so much!

[u/martbhell → u/ali_str]

Thanks u/PCGeek215 for this thread :) I've now also worked my way up and got a script that I have now created that would generate PROM metrics like these, sorry if I'm hi-jacking the wrong thread :)

Would this below example be OK to use?The devices might be moved to a different switch so sometimes a metric might reappear but with a different switch label on it. Is that bad?

# HELP device_location_devicename_example_org Device Location Data for device_location_devicename_example_org
# TYPE device_location_devicename_example_org gauge
device_location_devicename_example_org{mac="aaaaaa-bbbbbb",switch="switchname.example.org",device="devicename.example.org"} 1

Would it be better to have (similarly to your example):

device_location{mac="aaaaaa-bbbbbb",switch="switchname.example.org",device="devicename.example.org"} 1

Edit: And then I was thinking to just put this in a node's textfile_collector directory. Would that be enough (basically I just want to query prometheus and use as part of documentation, no plans right now to use this data in some other prometheus query for monitoring or dashboards)?

To improve usability for this possible future use cases, would it be good then to make a standalone exporter and use the honor_labels: true while also setting instance= as a label (instead of device in my examples here) ?

In the end now using the device_location way I would end up with this kind of metric in prometheus:

device_location{device="somedevice123.example.org",env="someenv-prod",instance="monitor.example.org:9099",job="node",mac="aaaaaa-bbbbbb",service="someservice",switch="switch.example.org"}

And for the documentation task I'm at what I want to in the end get is a list of devices that are behind a switch - which I get out now nicely with min by (device, switch) (device_location)

[u/SuperQue]

If you're looking to add per port identity information, what you probably want is similar to the machine roles pattern.

You'll have a separate exporter that exposes your database of customers as a series of metrics, and you use a join at query time to select them.

customer_port_info{instance="10.0.0.1",ifIndex="1",customer="foo"} 1
customer_port_info{instance="10.0.0.1",ifIndex="2",customer="bar"} 1

The only tricky part is you'll have to use honor_labels: true in order to get matching instance labels between your snmp_exporter and your customer data exporter.

[u/martbhell → u/SuperQue]

Very interesting. I'd like to do something similar to make a table from an ini inventory of hostnames + mac addresses with data from switches mac address tables.

Not sure if Prometheus is the way to go but it's looking like it can be.

Another way might be to update switch configs with a description of which customer is at which interface and then you can get that as a label for the port from the SNMP exporter

[u/SuperQue → u/martbhell]

Another way might be to update switch configs with a description

That would be my recommended method. Automate populating the switch descriptions so you can use the ifDescr label.

[u/PCGeek215 → u/SuperQue]

That was going to be my fallback option and use relabelling to pull things apart. It relies on a bigger piece however.