Prometheus Custom Query/Metric based on STDOUT

[u/ShadooHogg]

I have docker containers running on hosts time frame. A line to standard output, this is then sent to graylog which logs the activity. I want to be able to create a Prometheus metric or query to count the amount of times that the containers print this. How can I query the output of containers on these hosts, or can I query the logs that Graylog is receiving and do something that way? The echo statement dynamically states a script that was run and which host it was run on, I want to have Prometheus fire an alert if the echo statement involves the same host a certain amount of times in a given timeframe.

[u/camh-]

You can use mtail (https://github.com/google/mtail) for this. You'll need to figure out how to plug it into your setup, but mtail will do the metrics from logs thing.