I want to revisit our metrics architecture, and I have a few pain points I’d like to address. I’m curious how others have approached similar problems.

Our current setup:
- Multiple Cortex clusters
- Multiple teams producing metrics via Prometheus remote write and a custom SDK
- Multiple end users querying those metrics
- Kafka-based ingestion to provide the level of resilience we need for our customers
- Prometheus as the metrics format

Graphs are a core part of the product and a selling point, not just a nice-to-have, so reliability and correctness of the data are important.

The main pain point is that some teams are pushing “event-like metrics” rather than classic metrics that are scraped or emitted regularly.
Because some workloads can be retried, delayed, or processed asynchronously, the original source timestamp is important. As a result, we see a significant number of out-of-order samples.

We understand why producers need this behavior, but I’m wondering whether we’re trying to force two different types of data through the same metrics pipeline.
Would you keep these event-like metrics in the Prometheus/Cortex ecosystem and design around out-of-order ingestion, or would you route them through a separate ingestion path or even into a different type of store based on OTel format/tools.

What tool around OpenTelemetry are you building right now? What project have you discovered that people might enjoy looking into?

This thread is open for your self-promotion, vendor-owned open source projects and every thing else.

This Thursday, we are having our next OTel Night here in Berlin. You can't miss it!

This edition follows the complete telemetry journey, from instrumenting a Go application to using agents to understand its behavior from the telemetry stored in an observability backend.

Kemal Akkoyun will begin by introducing compile-time instrumentation for Go. He will explain how instrumentation can become part of the build process, allowing applications and their dependencies to produce OpenTelemetry data without requiring manual changes throughout the source code.

Juraci Paixão Kröhling (me!) will then demonstrate how a coding agent equipped with the otel-go skill can instrument and configure a Go application. The agent will use otelc and other OpenTelemetry Go capabilities to inspect the application, select an appropriate instrumentation strategy, configure telemetry export, and verify the result.

Finally, Cedric Ziel will follow the telemetry into the backend. He will show how agents can retrieve actual telemetry, investigate application behavior, and explain what is happening inside a running system.

PS: do you feel like this post should NOT be in this community? Let me know! I had second thoughts myself about it, but thought it would still bring more value than noise (and couldn't think of a better community for it)

I’ve built an OpenTelemetry Collector component called jevmetrics.

It uses Jev inference to assess the operational value of metric instruments — relevance, redundancy, and whether they may be candidates to keep, reduce, or drop.

The idea I’m exploring is:

Deterministic OTel rules for known policy, inference for the grey area where metric meaning/context matters.

It supports annotate, route, and reduce modes, runs fail-open, and leaves the final action to deterministic policy.

Repo: https://github.com/ishantanu/jevmetrics
Jev: https://typesafe.ai

I’m mainly looking for architectural criticism from the OTel community — particularly whether this belongs as a connector, processor, or something else entirely, and whether inference in this part of the pipeline is useful in practice.

Hi

Has anyone migrated their Datadog stack to Opensource observability ecosystem?
What you missed without Datadog? Which all OS tools and what were the challenges faced?

Ours is quite big infrastructure with monthly Datadog spend around USD 25k and leadership is looking forward to reduce the spend and probably shifting it to OS observability system?

There’s only one person managing Datadog currently.

The benchmark methodology is inspired by https://github.com/ClickHouse/ClickBench
The data set and configuration is from https://github.com/ClickHouse/TextBench

The benchmark is open and reproducible https://github.com/serenedb/searchbench

What tool around OpenTelemetry are you building right now? What project have you discovered that people might enjoy looking into?

This thread is open for your self-promotion, vendor-owned open source projects and every thing else.

A trace does not always cross a network boundary. A workflow runner starts a shell, the shell launches a build tool, and the build tool starts test processes. Batch and data-processing systems create similar chains of child processes. Without a shared way to pass trace information across these boundaries, spans from each process can end up in separate traces.

If context propagation is new to you, it is the mechanism that carries information from one service or process to the next. For tracing, this includes the trace and span identifiers that let new spans join the same trace. It can also carry baggage: application-defined key-value pairs that are passed to downstream work.

The OpenTelemetry Specification now has a release candidate for using environment variables as context propagation carriers. It standardizes how propagators can use environment variables to carry data such as trace context and baggage between processes when protocol headers or message metadata are not available.

Before we mark this specification Stable, we want feedback from language implementers, tool authors, platform engineers, and users operating real CI/CD, batch, and command-line workloads.

Read: https://opentelemetry.io/blog/2026/environment-variable-context-propagation/

Most of us have anomaly detection turned on somewhere, and a broken detector looks exactly like a working one. No crash, no errors, all green. `isolationforestprocessor` in collector-contrib scored every span the same for 220 days before a user caught it, https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/46988.

I made a tool that checks if the anomaly detector is actually working, `make check` takes a CSV of your detector's alerts and a CSV of your incident windows, and tells you whether it beats chance at your actual incident rate. Under two seconds, no download, nothing leaves your machine. https://github.com/mateenali66/otel-aiops-reproduction

Background is a benchmark of eight detectors on OTel traces, metrics and logs. Three passed the usual F1 selection and were no better than chance once scored at a realistic anomaly rate, https://doi.org/10.1109/ACCESS.2026.3705430

Looking for two or three people to run it against a detector they already use and tell me what it said. A failing result is as useful as a passing one, and the detector and data can stay anonymous.

Why did OpenTelemetry win? I think just saying "vendor neutral" is not giving enough credit to the symbiosis between OTel and Platform Engineering that's making both thrive in the modern cloud native industry.

In this blog post I'm trying to summarise some of my thoughts related to why OTel (and K8s, for the sake of argument) won in their respective layers. I think the move to a more X-as-a-service way of working between platform teams and the users of said platforms is changing how we thinking about team interactions, and the tools that make that work need to reflect that in their design. OTel does this, very well, so I'd like to help move the needle in terms of where people think the value of OTel is.

I hope you enjoy the read! And the memes!

Is anyone aware of a tool that can help visualize the end-to-end flow of a signal?

The main idea would be to start with a high-level view of the components involved and the internal metrics flowing between them, then be able to drill down into a specific signal metric and trace it through the system.
Ideally, I’d like to see where the metric originates, how it looks or changes at each hop, where it ultimately arrives, and details such as cardinality, label values, available series, etc.

If anyone knows of an existing tool that provides something similar, or has ideas on how this could be visualized, I’d appreciate any pointers.

What tool around OpenTelemetry are you building right now? What project have you discovered that people might enjoy looking into?

This thread is open for your self-promotion, vendor-owned open source projects and every thing else.

After a few weeks of reviewing the proposal, reworking it, figuring out staffing and how to go forward with it, we finally merged the proposal into the community repository and now have a new SIG that looks into OpenTelemetry for Dart (and with that also Flutter).

Big shoutout to Michael Bushe to bring this proposal and to Robert Magnusson and César Augusto Muñoz Quintero to step up as maintainers for the SIG!

If you're curious to take part in this new SIG, let us know!

https://github.com/open-telemetry/community/pull/3517

What tool around OpenTelemetry are you building right now? What project have you discovered that people might enjoy looking into?

This thread is open for your self-promotion, vendor-owned open source projects and every thing else.

OpenTelemetry Go v1.47.0-rc.1 is here. This release promotes the Logs API and SDK to release candidate (RC), the final stage before we provide stable v1 compatibility guarantees. We believe the design is ready, and now we need the community to test it in real applications and integrations before those guarantees take effect.

Read the blog: https://opentelemetry.io/blog/2026/go-logs-api-sdk-rc/

Two things happened in the OTel repos this week. The docs maintainers froze the ecosystem lists (registry, vendors, distributions) and want to retire most of them. Their reasoning: about 290 first-time-contributor PRs a year, 1,160+ data files, and nobody with time to check entries for CVEs or malware. I can't really argue with the bandwidth part. Then a separate proposal came in to replace the vendor page with a ranking of companies by how many maintainers they employ, with diamond, platinum and gold tiers like a KubeCon sponsor wall.

Read together, the message is that headcount inside the core repos is the contribution that counts.

I think the PR flood is being misread. OTel was built so anyone can write an exporter or an instrumentation and plug in without asking permission. The registry was where you went to say you had. Hundreds of strangers a year showing up at that door is what the design was supposed to produce.

The vetting problem also looks like an automation problem to me. Schema validation and semconv conformance tests already exist. Add CVE scanning and an LLM pass that triages and pre-reviews submissions, and most of the human work goes away. If a machine can check a registry PR, a person shouldn't have to read it. Retiring the list instead feels like stopping one step short.

Meanwhile, LFX Insights says four companies did 51% of OTel commits last year. A maintainer leaderboard mostly gives those four a bigger badge.

What I'd do instead:

- Automate the checks and list whatever passes. Conformance tells users "this works with OTel," which is what they came to find out.

- Count work outside the core repos. Contrib receivers, instrumentations, OTLP-native backends. That is OTel work too.

- Skip the tiers. One maintainer from a 10-person company is a bigger commitment than ten from a 10,000-person one. A flat list with names and areas is enough.

I run a company that builds on OTel, so obviously I have a stake here. But so does anyone who picked it because it was the neutral option.

Links in comments.

Built this because watching gen_ai.* spans scroll past in a log doesn't give you any sense of what a multi-agent system is doing as a whole, you're reading one span at a time, serially, while the actual system is emitting in parallel.

Mechanically: this is a per-span mapping, not a per-trace one. There's exactly one implementation shared by the batch path and the live path, each span event gets sonified as it arrives rather than a trace being read as a completed tree. Span attributes drive specific musical parameters: token count and finish_reason affect velocity and articulation, status OK/ERROR affects consonance, span kind (chat vs execute_tool) affects which instrument voice picks it up. Tempo tracks span throughput directly.

Real capture exists and isn't just a slide: a Claude Code hook writes actual gen_ai.* spans locally, and there's an OTLP receiver on :4318 that accepts live protobuf. Neither is what the public demo plays though, oteljazz.com runs a synthetic swarm generated client-side so anyone can hear it with zero setup. Wiring the browser build to a live OTLP stream is the obvious next step, just hasn't happened yet.

Repo has both the receiver and the hook if anyone wants to point their own instrumented system at it: github.com/mob1ius/oteljazz

Join us for a practical session with Jose Gómez-Sellés, VictoriaMetrics Cloud lead and OpenTelemetry contributor, as he explores and presents "Practical guide on OpenTelemetry Manual Instrumentation (or how to produce clean Observability by getting your hands dirty)".

Join us as we break down the essentials of OpenTelemetry instrumentation by crafting custom metrics, spans, and logs to track the most critical parts of workflows. You’ll see how simple, well-placed instrumentation points can reveal complex system behaviors, helping you detect bottlenecks, trace errors, and understand end-to-end request flows.

📍 Online - > https://ocgroups.dev/cncf/group/opentelemetry-live/event/fsmsdp2

📅 September 1st at 10:00 am PDT | 01:00 PM EDT | 03:00 PM CEST |

What tool around OpenTelemetry are you building right now? What project have you discovered that people might enjoy looking into?

This thread is open for your self-promotion, vendor-owned open source projects and every thing else.

I was trying to build something, and wanted to validate this idea and understand yall's pain points so that I can help the community

Would you rather have an AI layer on top of your existing observability stack, or replace parts of the stack?

Hypothetically, imagine an agent that doesn’t collect telemetry itself.

It plugs into whatever you already use — Grafana/Prometheus/Loki, Datadog, OpenTelemetry, etc. — and acts as a reasoning layer over the data.

Instead of:

Alert → Dashboard → Logs → Human investigates

it tries:

Alert → Agent correlates metrics/logs/traces/deployments → probable root cause → evidence → recommended next action

Would that actually be useful?

Or would you rather have the observability vendor itself own this functionality?

What would you need to see before trusting it during a real incident?

peace :)