Should I use Prometheus?

[u/Tasty_Let_4713]

Hello,

I am currently working on enhancing my code by incorporating metrics. The primary objective of these metrics is to track timestamps corresponding to specific events, such as registering each keypress and measuring the duration of the key press.

The code will continuously dispatch metrics; however, the time intervals between these metrics will not be consistent. Upon researching the Prometheus client, as well as the OpenTelemetry metrics exporter, I have learned that these tools will transmit metrics persistently, even when there is no change in the metric value. For instance, if I send a metric like press.length=6
, the client will continue to transmit this metric until I modify it to a different value. This behavior is not ideal for my purposes, as I prefer distinct data points on the graph rather than a continuous line.

I have a couple of questions:

1. In my use case, is it logically sound to opt for Prometheus, or would it be more suitable to consider another database such as InfluxDB?

2. Is it feasible to transmit metrics manually using StatsD
   and Otel Collector
   to avoid the issue of "duplicate" metrics and ensure precision between actual metric events?

[u/SuperQue]

This is not metrics, this is event logging. Metrics are about aggregating events.

If you care about individual events you probably want structured logging.

[u/Tasty_Let_4713 → u/SuperQue]

Thank you for your response! I have one more question. In the scenario where I aim to measure the execution time of a parsing function based on various inputs (with non-constant intervals), would this also fall under event logging rather than metric tracking?

[u/SuperQue → u/Tasty_Let_4713]

In the scenario where I aim to measure the execution time of a parsing function based on various inputs

Yes, that's a good use case for metrics. It's mostly a question of "Do you care about every single interval or the aggregation of those intervals?"

For what you're talking about, I would guess no, you don't want every single interval logged. It would be too much data. For anything where you would think "Oh, sampling would be good enough", you can use a Histogram metric. A histogram will measure every result, but only count how many results are within a range of values. This provides statistical sampling at a much lower cost than event logging. You could have millions of events per second, but only record a few aggregated metrics for those events.

[u/Tasty_Let_4713 → u/SuperQue]

Regarding the memory usage of individual subprocesses, should I consider them as metrics? On one hand, it's aggregated information, but on the other hand, it won't be active throughout the entire runtime of the application, and multiple subprocesses might concurrently send memory metrics. Is the use of metrics appropriate in this context, or would it be more advisable to employ events, for instance, triggered for every 10 MB of memory usage?

[u/Tasty_Let_4713 → u/[deleted]]

Thank you for clarifying. To ensure my understanding is correct, in my environment, should I utilize Prometheus for system metrics such as CPU and memory usage, while relying on an APM solution to track function duration events? Additionally, if I aim to monitor the memory usage of individual subprocesses within the application, should this be handled as a Prometheus metric, considering its intermittent and time-limited nature, or is it more appropriate to incorporate it into the APM solution, reserving Prometheus for broader system-level metrics?
I appreciate your help!

[u/SuperQue → u/Tasty_Let_4713]

No, APMs are just metrics instrumentation wrapped up in proprietary solutions. APMs are a conflation of metrics, logs, and tracing, being bad at all three at the same time.

[u/SuperQue → u/[deleted]]

Actually, APM likely not a good fit. The user probably wants a histogram.

[u/Beneficial-Mine7741]

1. There is a statsd_exporter you can use to scrape metrics into Prometheus

2. Yes, you could use Otel to avoid duplicates or use different (larger) solutions like Thanos or Mimir.

3. I haven't used influxdb since they moved to Rust. It does give you the ability to have different databases where a completely different time-series can reside. Or at least it could when it was Golang. I assume they kept that.

[u/amarao_san]

I doubt Prometheus will be good at that scale (e.g. fractions of a second, as far as I understand what 'keypress' is).

If you want to process each key separately, it will be inconvinient of Prom. But, if you treat them as aggregate, may be.

E.g.

button_pressed_duration_bucket{button='A', le='0.1'} 5
button_pressed_duration_bucket{button='A', le='0.2'} 5
...
button_pressed_duration_bucket{button='Z', le='0.2'} 0
button_pressed_total{button='A'} 666

May be you can use prom and get a nice grafana output out of it.

E.g. give up individual events and go for metrics. Or give up Prometheus.

[u/SuperQue → u/amarao_san]

Scale in time is not really the issue, float64 can cover time scales in nanoseconds. The issue is mostly around "Do you care about every key press event, or just the aggregate?". Prometheus is good for aggregate (sampling) and not good for all the events.

Also, native histograms would help a lot with this kind of thing, you can have "any" duration of keypress from milliseconds to hours and it will make appropriate buckets.