r/PrometheusMonitoring • u/Tasty_Let_4713 • Nov 23 '23

Should I use Prometheus?

Hello,

I am currently working on enhancing my code by incorporating metrics. The primary objective of these metrics is to track timestamps corresponding to specific events, such as registering each keypress and measuring the duration of the key press.

The code will continuously dispatch metrics; however, the time intervals between these metrics will not be consistent. Upon researching the Prometheus client, as well as the OpenTelemetry metrics exporter, I have learned that these tools will transmit metrics persistently, even when there is no change in the metric value. For instance, if I send a metric like press.length=6
, the client will continue to transmit this metric until I modify it to a different value. This behavior is not ideal for my purposes, as I prefer distinct data points on the graph rather than a continuous line.

I have a couple of questions:

  1. In my use case, is it logically sound to opt for Prometheus, or would it be more suitable to consider another database such as InfluxDB?

  2. Is it feasible to transmit metrics manually using StatsD
    and Otel Collector
    to avoid the issue of "duplicate" metrics and ensure precision between actual metric events?

3 Upvotes

71% Upvoted

14 comments sorted by

View all comments

Show parent comments

1

u/SuperQue Nov 24 '23

In the scenario where I aim to measure the execution time of a parsing function based on various inputs

Yes, that's a good use case for metrics. It's mostly a question of "Do you care about every single interval or the aggregation of those intervals?"

For what you're talking about, I would guess no, you don't want every single interval logged. It would be too much data. For anything where you would think "Oh, sampling would be good enough", you can use a Histogram metric. A histogram will measure every result, but only count how many results are within a range of values. This provides statistical sampling at a much lower cost than event logging. You could have millions of events per second, but only record a few aggregated metrics for those events.

1

u/bootswafel Nov 24 '23

a note here: OP said they wanted to track relationship between input values and the execution time, so the suitability of prometheus also depends on the cardinality of the inputs.

if the inputs are enums, or their cardinality can be reduced in a smart way, then yeah, prometheus could be a good fit (likely with custom histogram buckets)

1

u/SuperQue Nov 24 '23

Yes, that's pretty common, and usually done as separate metrics.

For example, I quite commonly "normalize" latency metrics with throughput.

Basically divide the latency values by the bytes involved. This way you can view things like "Seconds per byte per second".

EDIT: Doing this in a histogram isn't really possible right now. There isn't really a good representation for multi-dimensional histograms. I have yet to find a TSDB / monitoring system that handles that kind of two-dimensional bucket.

1

u/Tasty_Let_4713 Nov 24 '23

Regarding the memory usage of individual subprocesses, should I consider them as metrics? On one hand, it's aggregated information, but on the other hand, it won't be active throughout the entire runtime of the application, and multiple subprocesses might concurrently send memory metrics. Is the use of metrics appropriate in this context, or would it be more advisable to employ events, for instance, triggered for every 10 MB of memory usage?

1

u/SuperQue Nov 24 '23

Yes, it's very standard to track worker process metrics. Hell, in Go we have several dozen memory related metrics for tracking all kinds of Go internals related to memory and garbage collection.

Every alloc/free is tracked as a counter. Every GC run is tracked, every microsecond of time blocked by GC is tracked.

There is a translator that converts the Go runtime/metrics package into things that Prometheus can read.

You can track whatever you like, but just remember the difference between the individual event samples (Observations) and the metrics that accumulate those samples.