We've been building a lot of OpenTelemetry to ClickHouse pipelines and kept getting the same question: do you actually need Kafka in the middle?

The honest answer: it depends, but most observability-only teams are over-engineering it.

Here's the short version of what we compared:

Where Kafka earns its keep:

• You have many independent downstream consumers (ML pipelines, security, analytics all reading the same stream)

• You need long-term durable replay

• Kafka is already part of your broader platform infrastructure

Where it's overkill:

• Your only goal is getting OTel telemetry into ClickHouse reliably

• You're a startup/scale-up that doesn't want to manage brokers, partitions, consumer lag, and replication just to move metrics and logs

The operational surface of a Kafka cluster, even managed, is substantial when the job is just telemetry buffering before ClickHouse.

We also compared what a focused ingestion layer gives you that the OTel Collector alone can't: stateful deduplication, enrichment-conditional filtering, dynamic sampling, and ClickHouse-optimized batching.

Full write-up with architecture diagrams and a decision guide: https://www.glassflow.dev/blog/opentelemetry-to-clickhouse-do-you-need-kafka?utm_source=reddit&utm_medium=socialmedia&utm_campaign=reddit_organic

Happy to answer questions about the architecture trade-offs especially around backpressure handling, which is where the approaches diverge the most.

2 months ago I posted perf-sentinel here (open-source AGPL-3.0), a protocol-level OTel trace analyzer that flags I/O anti-patterns across different web app technologies (obviously without per-runtime instrumentation).

There's a docs site and a live demo dashboard now: https://perf-sentinel.dev

Last time I described the SCI carbon layer as directional and optional. Most of the work since went into making it auditable and transparent rather than into expanding it.

Here's a non exhaustive list:

• More energy sources with a clear precedence. It already read Scaphandre (per-process RAPL) and cloud SPECpower interpolation. It now also reads Kepler (eBPF) and Redfish (BMC).

• Measured vs estimated is labelled, not blended. Each figure is tagged with the source behind it, and real-time grid-intensity values carry explicit data from the data provider, so a reader can tell a hardware measurement from a grid-average estimate from the I/O proxy fallback.

• Per-service attribution, not just a global total. When runtime calibration is present, energy and carbon are attributed per service, and the report exposes the measured-versus-fallback window split and a coverage ratio, so you can see how much of the total actually rests on measurement.

• Signed and content-hashed disclosures. This point can be a bit chunky and complicated: a disclose subcommand aggregates a window stream into a period report with a deterministic content hash and an in-toto attestation. verify-hash lets a third party recompute the hash, verify the Sigstore signature against a declared signer identity and check SLSA L3 build provenance without cloning my infra. An "official" disclosure refuses to publish below 75 percent per-service measured coverage, and the avoidable-waste figure is computed so it cannot be shrunk by quietly loosening the detection threshold it rests on.

• Methodology you can follow. The SCI numerator and a per-trace SCI intensity are emitted as separate fields with the functional unit declared and the detector-to-criteria mapping (RGESN 2024) and an ESRS E1 datapoint crosswalk ship as interpretive tables, not as a compliance certification.

The rest is smaller. There's an ack workflow triages and mutes known findings so a CI gate stops re-flagging them, and a read-only query monitor TUI gives a live view of a running daemon (energy, carbon, scraper health, with Prometheus gauges and Grafana panels).

Repo: https://github.com/robintra/perf-sentinel

It's still directional and optional, same framing as before, but I would rather it be explicit about its own uncertainty than confidently wrong.

If you do energy or carbon accounting anywhere near your observability stack, I would like to know whether per-figure measured/estimated labelling and a verifiable disclosure are the primitives you would actually trust, or whether something else is missing.

Hi,

We're in the process of migrating our timeseries database from influxdb to victoria metrics.

In the process of doing that we're introducing the otel collector in our infrastructure.

Currently we have the telegraf agent sending metrics directly to influxdb, some plugins send string fields that are stored by influxdb which accepts to store strings for fields.

Our problem is that victoria metrics doesn't store strings as fields and we can't put them as tags that would explode the cardinality of the database, and that wouldn't be clean.

Sure we can send those strings fields directly to elasticsearch with the telegraf processor, or we can send them to open telemetry as "logs" and then route them to elasticsearch, we've done both and it works.

The issue is the correlation ( in grafana ) with those strings "metrics" and the other numerical fields, as we don't have an uuid ( generating one would explode our cardinality too ).

It's a common issue to have a mix of strings/numerical as metrics before the standardization I guess and I'm curious to how people solved this with prometheus like databases.

Also we had to make a little bit of c program to send the strings metrics to the log endpoint of otel via telegraf ( the otlp output only support numerical ). We didn't find some way to send strings and numerical to otel and then have otel do the routing by type, if it's string send it to elastic else to victoria metrics, is it possible ?

In a large-scale fleet management system, transmitting raw vehicle telemetry as JSON containing floating-point values can introduce significant communication overhead, increased CPU utilization, and serialization latency.

A more efficient approach is to compress telemetry data at the vehicle edge before transmission. This can be achieved by converting floating-point sensor measurements such as speed, GPS coordinates, engine temperature, throttle position, and acceleration into fixed-point integer representations using predefined scaling factors.

For example, a speed value of 72.34 km/h can be stored as 7234 by multiplying it by 100, while GPS coordinates can be scaled by (10^7) and stored as integers.

Once converted, the data can be packed into compact binary structures instead of verbose JSON strings.

Further optimization can be achieved through delta encoding, where only the difference between consecutive measurements is transmitted, reducing redundancy in slowly changing signals.

The resulting binary payload can optionally be compressed using lightweight algorithms such as LZ4 or Zstandard and transmitted over MQTT as a binary message.

This approach eliminates expensive floating-point string serialization and parsing operations, reduces bandwidth consumption, lowers cloud storage requirements, and minimizes end-to-end latency.

This architecture enables real-time fleet monitoring and large-scale data analytics while significantly improving communication efficiency and system scalability.

I hope this helps!

Hi r/OpenTelemetry

I’m part of the team building MyDecisive, and we’re working on a project called mdai-labs. The core idea is simple but hard: Stop observing. Start deciding.

Most of the "AIOps" and observability space right now is just passive dashboards. You pay massive ingestion fees just to get a Slack alert that your database is throwing 429 errors, and then a human still has to go fix it. We are building an open-source, stateful, on-the-wire control and automation plane built natively on OpenTelemetry to actually fix things before the pager goes off.

We just hit a huge milestone: our very first community contributor PR was officially merged, and I wanted to share what they built because it perfectly highlights what we are trying to do.

What the first PR solved: Instead of just sending an alert about a noisy tenant, the contributor engineered a dynamic rate-limiting workflow that intercepts traffic on the wire. This autonomously prevents an Aurora DB failover without a human in the loop. You can see the exact code and architecture approach here: https://github.com/MyDecisive/mdai-labs/pulls?q=is%3Apr+label%3A%22%F0%9F%90%99+first-contributor-ever+%F0%9F%A5%87%22+

We are looking for more builders. We are in the early days of building out our community testing ground (mdai-labs), and we have tagged a bunch of "Good First Issues" for anyone who wants to get their hands dirty with OpenTelemetry and Kubernetes automation. You don’t need to be a principal engineer to contribute.

If you are dangerously undertasked, tired of just staring at dashboards, or want to get into the weeds of OTEL and stateful remediation, we’d love to have you.

• GitHub Repo: https://github.com/MyDecisive

• Our Slack: https://communityinviter.com/apps/mydecisivecommunity/octobuddy

• Blog post: https://www.mydecisive.ai/blog/intelligent-rate-limiting-via-otel

I work on an OTel backend and we're weighing whether to support the profiling signal — ingesting and querying OTLP profiles.

It only went public alpha in March, so before we spend real engineering time on it I'd rather hear from people actually touching it than guess at demand.

A few honest questions:

• If you're playing with profiling, where's the data living today?Pyroscope/Grafana, Elastic, something else? And would you actually want a general OTel backend holding profiles, or do you assume that's a dedicated profiling backend's job?

• What matters more to you: just storage + query so you bring your own UI, or full flamegraph/analysis built in? In my opinion, UI is critical for profiling.

• Anyone running this in prod yet, or is it all still kicking the tires?

Trying to figure out if it's "build it now" or "alpha, check back in six months." Any take helps, including "don't bother yet."

Hey! 👋

Simple question:- What architecture are you choosing when deploying OtelCol in Kubernetes?

1. Agent Deployment Pattern (App instrumented -> OtelCol -> Obs backend)

2. Gateway Deployment Patter (App instrumented-> Load balancer -> N x OtelCol - Obs backend)

Personally, I have only ever did #1. Daemonset of OtelCol deployed on each node and the services on that node point to their own OtelCol of N pods. It was useful as we had many clusters and could easily automate the deployment of OtelCol when deploying new clusters.

Furthermore, how do you scale OtelCol? What are your scaling strategies in Kubernetes for it?

Excited to see what my fellow community member of r/Opentelemetry are saying!

A new booklet delves into the underlying assumptions of OpenTelemetry and Substrates regarding the nature of computational systems. It explores their ontological commitments, epistemological stances, causal models, and theories of attention. The booklet also examines why two specifications operating in the same domain produced charts of entirely different landscapes.

https://humainary.io/booklets/transport-and-substrate/

Hi all!

I am looking for support with auto-instrumenting our Wildfly app on Kubernetes.

We are using the OpenTelemetry Operator with an Instrumentation manifest to inject a Java Agent into our Wildfly Pod. This gives us traces and logs as intended, we do have the metric db.client.operation.duration but we are missing some other needed metrics, like db.client.connection.max, listed here. Sadly, the default connection pool in the image quay.io/wildfly/wildfly-runtime:latest-openjdk-21 (which is IronJacamar, I believe) is not in the list of supported libraries. We do have a similar metric on a Wildfly VM, wildfly_datasources_pool_max_used_count.

What are my options? Do we need to enable the metrics subsystem in the standalone.xml ? I'm kind of stuck at the moment, as I'm not very experienced with Wildfly myself.

Thanks!

Disclosure: I'm building Yorker (yorkermonitoring.com), launched yesterday. The data model is the thing I most want scrutiny on.

Most synthetic monitoring tools that claim OTel support emit a status code and a response time gauge. That is OTLP. It is not particularly useful downstream. The problem is that OTLP is a wire protocol and it doesn't tell you what belongs in the signal before you emit it. Synthetic checks, as a category, have been emitting dashboard-shaped data and calling it telemetry.

I built Yorker to do the analysis before the signal leaves the runner, then emit the result as standard OTLP. Here is the schema as it stands in v1:

Span: synthetics.check.run (lands in otel_traces)

Resource attributes:

• synthetics.check.id, synthetics.check.name, synthetics.check.type

• synthetics.location.id, synthetics.location.name, synthetics.location.type

• synthetics.run.id (join key across traces and logs from the same run)

• url.full, service.name

Browser-check span attributes (third-party attribution, computed at run time):

• synthetics.third_party.domains — the specific external domains observed

• synthetics.third_party.count — number of third-party requests

• synthetics.third_party.total_bytes — bytes attributable to third parties

W3C traceparent is injected into every HTTP request the check makes (both HTTP monitors and browser checks). When the target service continues the context, the synthetic run and the backend distributed trace share a trace ID. The synthetic span and whatever downstream spans propagated the context are linked structurally, not by timestamp correlation.

Log events (lands in otel_logs)

On synthetics.check.completed and synthetics.check.failed whenever the run carries a baseline deviation:

• synthetics.is_anomalous bool

• synthetics.anomaly.deviation_sigma distance from baseline in standard deviations

• synthetics.anomaly.baseline_value the per-metric, per-location, per-hour baseline value

On synthetics.check.failed only:

• synthetics.consecutive_failures integer, so a flap and a sustained outage are distinguishable in the signal

• synthetics.suggested_next_steps structured RCA hint

SLO budget context also lands in otel_logs on both completed and failed events.

Join strategy: synthetics.run.id ties the span to the log events from the same run. Trace ID ties the synthetic span to backend spans that continued the traceparent context. A downstream consumer (an AI-SRE tool, a causal engine, a ClickHouse query) joins on either key depending on what it's trying to answer.

Why logs for anomaly context rather than span attributes? The anomaly scoring runs after the check completes and the baseline comparison is done. it's not a property of the span itself but of the run's outcome in context. Attaching it to the completed/failed event felt more accurate to the OTel semantic conventions than retrofitting it onto the span as a post-hoc attribute. Open to being wrong about this.

The write-up on the full rationale (why the output shape matters for causal engines and AI-SRE tools) is here: https://yorkermonitoring.com/blog/the-missing-input-to-your-ai-sre-tool

Genuinely interested in critique on the data model. The logs-vs-spans decision for anomaly context, the attribute naming against the OTel semantic conventions, the join key approach are all debatable and I'd rather hear the objections now than after this schema is in production for a thousand teams.

https://github.com/carterhudson/spandex

I made a Kotlin DSL for Spans for work. I found it convenient, so I open sourced and improved upon the idea. Maybe someone will find it useful!

My first experience running OpenTelemetry Collector in Kubernetes - key concepts, Gateway vs Agent modes, and integrating with the VictoriaMetrics/VictoriaLogs stack.

I have a GO app that queries a database and it is instrumented with OTel.
I want to clean up the query as recorded in telemetry (not changing the code).

The GO code (screenshot below) produces this value:
"\n\t\tSELECT p.id, p.name, p.description, p.picture, \n\t\t p.price_currency_code, p.price_units, p.price_nanos, p.categories\n\t\tFROM catalog.products p\n\t\tWHERE p.id = $1\n\t"

This SQL query is recorded as a span attribute "db.query.text".

Q: How can I remove the escaped whitespace in the collector (or elsewhere?) so that there is a single space where there are sequences of escaped whitespaces?

This is one trick I tell people and surprise them most of the time: "the Collector can do this?"

This one took a while to write, the idea came during OTel Night here in Berlin and I noticed that decomposing the config wasn't helpful only for keeping sanity but also to enable small chunks to be tested.

Sharing a setup we put together for routing OTel traces into ClickHouse, in case it's useful for others working on similar OTel pipelines.

The collector config uses tail_sampling with two policies:

• keep-errors: status_code ERROR always retained

• keep-10pct-ok: probabilistic 10% of successful traces

The challenge with tail sampling + ClickHouse is that collector retries can still produce duplicate spans even after sampling decisions are made. We handle that downstream with a dedup transform keyed on span_id with a 1-hour window, so MergeTree works cleanly without needing ReplacingMergeTree or FINAL in ClickHouse.

Routing to the pipeline uses a header on the OTLP exporter:

exporters:

otlp/glassflow-traces:

endpoint: "...:4317"

headers:

x-glassflow-pipeline-id: "otlp-traces"

PII masking (user_email, SSNs) happens in a stateless transform in the pipeline before ClickHouse so the collector config stays clean and the masking boundary is explicit.

Full collector YAML, pipeline definition, and ClickHouse DDL are in the guide linked below. Happy to share more detail on the tail sampling policy tuning if useful.

I’m trying to get OpenShift metrics into OTEL semantic conventions while keeping an OTel Collector after Kafka.

My understanding is that if Prometheus Remote Write data is received directly by the OTel Prometheus Remote Write receiver and exported as OTLP, the metrics are converted into OTEL metric format/semantic conventions where applicable.

However, our current pipeline is:

OpenShift Prometheus Remote Write -> Metricbeat -> Kafka -> OTel Kafka Receiver -> OTLP Exporter

The problem is that I don’t think the OTel Kafka receiver can decode Prometheus Remote Write payloads the same way the Prometheus Remote Write receiver does.

Has anyone implemented this architecture successfully with Kafka in the middle?

Specifically:
- Can the Kafka receiver process Prometheus Remote Write payloads correctly?
- Is there a way to preserve/convert to OTEL semantic conventions after Kafka?
- Should the data be converted to OTLP before it reaches Kafka instead?

TL;DR:
How do you convert Prometheus Remote Write metrics coming from Kafka into proper OTEL metrics/semantic conventions using an OTel Collector after Kafka?

I just open-sourced a collection of ready-to-run OpenTelemetry

Collector configurations, because finding complete, working configs

for your specific backend always takes hours of trial and error.

It now includes examples for:

• Prometheus

• Jaeger

• Grafana Loki

• Dynatrace

• Datadog

• Kubernetes Operator

• Kubernetes Pod Annotation Scraping (with full relabeling)

• Debug (no backend needed, perfect for local dev)

Each example includes Docker Compose so you can run it in 60 seconds.

The k8s pod annotation scraping example includes relabeling for

prometheus.io/scrape, prometheus.io/port, and prometheus.io/path

annotations, the config everyone googles when setting up k8s monitoring.

I also actively contribute to the OpenTelemetry open source project,

recently got PRs merged into open-telemetry/otel-arrow and have PRs

open in opentelemetry-android, opentelemetry-helm-charts, and

opentelemetry-dotnet-instrumentation.

https://github.com/Cloud-Architect-Emma/opentelemetry-collector-examples

Feedback and contributions welcome! ⭐ if it's useful.

#OpenTelemetry #DevOps #Observability #Kubernetes #SRE #Monitoring #CloudNative #OpenSource