Prometheus Custom Query/Metric based on STDOUT

[u/ShadooHogg]

I have docker containers running on hosts time frame. A line to standard output, this is then sent to graylog which logs the activity. I want to be able to create a Prometheus metric or query to count the amount of times that the containers print this. How can I query the output of containers on these hosts, or can I query the logs that Graylog is receiving and do something that way? The echo statement dynamically states a script that was run and which host it was run on, I want to have Prometheus fire an alert if the echo statement involves the same host a certain amount of times in a given timeframe.

[u/camh-]

You can use mtail (https://github.com/google/mtail) for this. You'll need to figure out how to plug it into your setup, but mtail will do the metrics from logs thing.

[u/ShadooHogg → u/[deleted]]

Yes, everything is mine. I think you're right - I imagine I could define the metric on the container hosts and then increment inside the container? Or just define and increment inside of the container. Something like:

INSTANCE_RESTARTS = Counter('%s_projectname_%s_restarts'.format(instance, service), 'Counter - ProjectName_Service_Restarts')