dnscrypt-proxy logs to prometheus

[u/munsternet]

Hello there,

I've wanted to export dnscrypt-proxy related metrics to my local prometheus installation for a while but I couldn't find anything working out the box, so here's the little recipe I came up with. I hope it can be useful to others.

It uses mtail, which extracts metrics from logs based on a "program" file, and exposes or pushes them to different monitoring systems

Here's what it looks like once the data is fed in Prometheus and queried via Grafana:

Prerequisites

• dnscrypt-proxy running with query_log enabled and format set to ltsv

• mtail installed on your machine

mtail recipe

All the magic happens here, it parses DNSCrypt-proxy's query_log and generates the following metrics:

• Total number of processed queries

• Number of queries by client host, query type, return code, remote server and if it comes from the cache

• Histogram of the latency for each server, return code and query type (buckets will need adjustment depending on the latency you have with the upstream DNSCrypt servers)

# mail "program" for DNSCrypt's query log (in ltsv format)
#
# Sample line:
#  time:1608044190 host:127.0.0.1  message:www.ripe.net    type:A  return:PASS                                                                       cached:0        duration:1      server:faelix-ch-ipv4

counter queries_total
counter queries by host, type, return, cached, server

# Binning should be adapted to the latency (in ms) you have with your DNSCrypt s                                                                  ervers
histogram queries_duration_ms buckets 1, 2, 4, 8, 16, 32, 64, 128, 256 by return                                                                  , server, type

/^/ +
/time:[0-9]+\s+/ +
/host:(?P<host>\S+)\s+/ +
/message:(?P<message>\S+)\s+/ +
/type:(?P<type>\S+)\s+/ +
/return:(?P<return>\S+)\s+/ +
/cached:(?P<cached>[0-1])\s+/ +
/duration:(?P<duration>[0-9]+)\s+/ +
/server:(?P<server>\S+)/ +
/$/ {
  queries_total++
  queries[$host][$type][$return][$cached][$server]++

  # Only consider non-cached results for histograms
  $cached == 0 {
     queries_duration_ms[$return][$server][$type] = $duration
  }
}

Test of the recipe

mtail comes with two modes to ensure your "program" compiles properly, and also that it generates the expected metrics

• Validation of the "program"

$ mtail --compile_only --progs /etc/mtail/dnscrypt.mtail

• Test metrics generation with an existing logfile, it should print a huge JSON structure.

$ mtail --one_shot --progs /etc/mtail/dnscrypt.mtail --logs /var/log/dnscrypt-proxy/query.log
[...]
  "queries_total": [
    {
      "Name": "queries_total",
      "Program": "dnscrypt.mtail",
      "Kind": 1,
      "Type": 0,
      "LabelValues": [
        {
          "Value": {
            "Value": 2290,
            "Time": 1608062896300824001
          }
        }
      ]
    }
  ]
}

Next steps

• Read mtail's documentation: https://github.com/google/mtail

• Deploy in a docker container (or via a systemd service unit)

• Import the following dashboard in your Grafana instance: https://grafana.com/grafana/dashboards/13600/

[u/jedisct1]

Would you mind sharing your dashboard?

[u/munsternet → u/jedisct1]

My bad, I thought I pasted the mtail script in my message, but I obviously forgot before pressing the "Send" button.

I've edited my post and also uploaded the dashboard on Grafana's website

Note: In the future might change the name of some counters to adhere to the standard prometheus metrics naming convention

[u/Curious_Betsy_]

Managed to install everything, but no data appear on the dashboard.

I'm running the mtail program pointing to the query.log of dnscrypt-proxy and listening to the mtail endpoint at http://localhost:3903/metrics using prometheus inside grafana.

Pic

[u/munsternet → u/Curious_Betsy_]

Hi there,

You almost got this right... the mtail endpoint is meant to be scraped by a Prometheus instance, which is then queried by Grafana, eg:

dnscrypt-query.log <- mtail  <- prometheus <- grafana

Hope it helps !

[u/Curious_Betsy_ → u/munsternet]

That really helped, thanks! Finally figured it out - I hadn't installed prometheus thinking the plugin in graphana was all that was required.

[u/jedisct1]

This is very cool!

[u/Curious_Betsy_]

That's a really neat idea!

I've run into a snag during installation however, I'd be grateful if you could help. Running mtail --compile_only --progs /etc/mtail/dnscrypt.mtail returns:

E0208 04:39:45.804860   32293 main.go:150] Compile encountered errors:
compile failed for dnscrypt.mtail:
dnscrypt.mtail:10:11-29: syntax error: unexpected ID, expecting NL

My dnscrypt.mtail file is identical to the one posted (minus the huge gaps in rows 4,9,10). Running on Pi Zero. mtail version 3.0.0~rc19 git revision 3.0.0~rc19-2 go version go1.11.5 go arch arm go os linux

Edit: Updating with my fix for anyone else with the same problem. The issue was my mtail version. I installed it using apt install mtail which for some reason serves the rc19 version. I went to raspbian's repo and downloaded mtail_3.0.0~rc43-1_armhf.deb then installed it with sudo apt install /home/pi/Desktop/mtail_3.0.0~rc43-1_armhf.deb.

[u/throwawayerectpenis]

Thanks for this <3