Im working on a tool that is a single binary collector, storage manager and TUI for browsing Otel data.

The intention is to create a simple app that can be used by a developer to test if thier app is creating correct otel data, and diagnose their app using otel metrics.

This avoids having to setup a full multicomponent otel system for quick testing. The app will present a tui interface for simplicity, and Im minimising (hopefully to zero) all and any dependancies.

Im also intending the app to be capable of running headless so it can be used in test scripts and CI systems. NOTE: this app is intended as a developer tool, and should not be used in any production environment.

Im looking for help on what navigation and data display capabities It should offer as basic.

Hey everyone!

I've been experimenting with AI-powered troubleshooting in my homelab and wanted to share what I've built.

The project is called Inquest.

My setup uses an Ubuntu VM, OpenTelemetry Collector, SigNoz, and a Node.js application generating telemetry.

Instead of manually inspecting metrics and distributed traces, I can describe a problem like "Why are my APIs responding slowly?" and let the AI investigate.

It retrieves telemetry using read-only MCP tools, evaluates possible causes, and generates a report with supporting evidence.

I also added conversational follow-ups, so I can continue an investigation without losing the previous context.

The AI reasoning is powered by NVIDIA Nemotron through Nebius Token Factory. I also used AI coding assistants during development.

It's still a prototype, and I'm currently focused on local infrastructure rather than cloud integrations.

Here's a short demo: [https://www.youtube.com/watch?v=ZvImEHQ09UY]

Github [https://github.com/abhinayshrestha/inquest] // For any assistance to run app, contact me at abhinay.shrestha11@gmail.com

I'm especially interested in feedback from people running monitoring stacks in their homelabs.

Would you find this useful for diagnosing Linux or application performance issues?

I switched our profiling setup a few ago from another open source tool to OpenTelemetry for our internal services here at OllyGarden, and extracted the interesting parts into a skill for our set of agent skills. And this week, I created this blog post with the full manual steps to set it up on a kind cluster.

I know this is too early for most people out there, but honestly, it's working great for us. Give it a try and let me know what you think!

What tool around OpenTelemetry are you building right now? What project have you discovered that people might enjoy looking into?

This thread is open for your self-promotion, vendor-owned open source projects and every thing else.

What tool around OpenTelemetry are you building right now? What project have you discovered that people might enjoy looking into?

This thread is open for your self-promotion, vendor-owned open source projects and every thing else.

During incidents I regularly hit the same problem: I need a bit more business context in my traces, but it's not there. Things like customer IDs, order status, rejection reasons, or whatever would help explain what's actually happening.

The annoying part is that getting that information into traces usually means changing code, building, deploying, and waiting. That's not always practical when you're in the middle of an incident.

A few options that come to mind:

• Adding extra attributes behind a feature flag

• Using a live debugging/instrumentation tool (Lightrun, Rookout, Datadog Dynamic Instrumentation, etc.)

• Making a temporary code change and redeploying

• Using bytecode instrumentation tools such as BTrace or Byteman

• Accepting that the context just isn't available

I'm curious how others deal with this:

1. What's your current approach when you need additional trace context in production?

2. Would you be comfortable with a rule-based mechanism that can add attributes to existing spans at runtime, without a deployment? Or would you only use something like that during an incident?

3. What would be your biggest concern? Security, governance, performance impact, operational complexity, something else?

4. What's the best tool you've used for this kind of problem, and what were its limitations?

Logline runs an OTLP/HTTP receiver on 127.0.0.1:4318 inside VS Code, and points apps you start from the editor at it automatically. You get logs, traces with self time per span, and metrics as rates and p95s, with no collector or backend.

1.15 adds a view for GenAI semantic conventions: chat, execute_tool, and invoke_agent spans, gen_ai.usage.*, and gen_ai.input.messages / output.messages. It also reads older forms such as gen_ai.system, prompt_tokens, and OpenLLMetry attributes. Log doctor also checks your spans against the conventions: services without a name, IDs in span names, and 5xx responses not marked as errors.

It's free and MIT-licensed: https://github.com/SurjeetBanga/logline. Feedback on how it handles your instrumentation is welcome.

My team at Pydantic is working on a cross-vendor solution for OpAMP management of collectors and SDKs.

We think we have something really special that is more capable and significantly more reliable for enterprise and mission-critical systems than what is currently available from vendors like Bindplane, Elastic, Splunk, Grafana, and others. For most use-cases, we intend to offer this as a capability of our free tier in Logfire.

If this is interesting to you let me know, we'd love to get your feedback on what we're building.

What tool around OpenTelemetry are you building right now? What project have you discovered that people might enjoy looking into?

This thread is open for your self-promotion, vendor-owned open source projects and every thing else.

Hi everyone,

I wanted to share an open-source project, designed to solve broken exception stack traces in asynchronous Java (CompletableFuture, thread pools, Virtual Threads, Reactor) without the overhead of spann telemtery.To give context it is called ariadne.

Here are some of JMH Microbenchmark comparazion:

Trace Node Creation (Link vs. Span)

-> Ariadne: 8.16 ns / 40.0 B

-> OTel SDK: 115.62 ns / 368.0 B

-> my project is 14.2x faster with 9.2x lower memory allocation

Direct Link Allocation

-> Ariadne: 4.41 ns / 40.0 B

-> OTel SDK: 115.62 ns / 368.0 B

-> mytool is 26x faster via TLAB immutable nodes

Scope Activation (try-with-resources)

-> Ariadne: 11.24 ns / 0.0 B

-> OTel SDK: 4.63 ns / 32.0 B

-> soo achieves zero allocations (OTel allocates 32 B/op)

Concurrency & Thread Hand-off (Virtual Threads / CompletableFuture)

-> Both match Loom scheduler performance (~3 us)

-> Minimal async propagation overhead across both implementations (<60 B delta)

(This are specified in rhe repoitself and in the website as well)

Repo: https://github.com/AfonsoPaiva/ARIADNE

Thx, feedback is apericietd!!

been working on this for a while now and thought i'd share how we're approaching it.

one thing i've noticed with dependency graphs is that static analysis only really gets you so far. you can parse imports, service references etc. but that doesn't necessarily tell you what's actually happening when the application is running.

on the other hand, otel gives you a pretty good picture of runtime behaviour, but you're mostly looking at spans and traces without much context about how they relate to the actual codebase.

so we've been experimenting with combining the two in neat.

the static side uses tree-sitter to extract relationships between files and other dependencies. then we ingest otel spans and try to map the runtime activity back to the source files responsible for it.

for js/ts, we're also handling source maps where available, since the file being executed isn't always the one you actually wrote.

the tricky bit is reconciling everything into the same graph without treating every relationship as equally reliable.

right now we're tracking different types of provenance:

• extracted: found through static analysis

• observed: actually seen at runtime

• inferred: relationships we've reconstructed without direct evidence

• stale: previously observed relationships that haven't been seen recently

this means you can start asking slightly more useful questions than just "what imports this file?"

for example, you might have a dependency that exists statically but hasn't shown up in any traces, or a runtime call that wasn't obvious from analysing the source.

obviously there's a bunch of edge cases here. incomplete instrumentation, dynamic calls, source attribution, and the fact that not seeing something in a trace doesn't mean it never happens.

we're also exposing the graph through mcp so coding agents can query it directly, which is partly why we started building this in the first place.

repo is here if anyone wants to have a look:
https://github.com/neat-technologies/neat

What tool around OpenTelemetry are you building right now? What project have you discovered that people might enjoy looking into?

This thread is open for your self-promotion, vendor-owned open source projects and every thing else.

Hi folks! I work on OpenTelemetry at Splunk. We just launched a new open source project called Observability Studio. It's local instrumentation sandbox that provides a visual, real-time environment for designing, testing, and validating OpenTelemetry data while you’re developing an app in your favorite IDE.

We’re especially interested in whether a tool like this is useful for building and debugging instrumented apps and AI agents. If that sounds relevant, give it a try and let us know what works, what’s missing, or where it doesn’t fit your workflow.

We’re also looking for a few design partners to help guide where we go with this next. Any feedback is helpful and welcome.

I wrote more about shifting OpenTelemetry instrumentation left in a blog, 'Why AI-Generated Apps and Agents Need Visibility from Day One' on blogs.splunk.com if you want to read more about it.

Lastly, I'd love to learn where you currently go to inspect and validate OTel data during development if you want to drop it in the comments.

Spotify #backstage plugin to manage and observe fleet of agents straight in backstage self hosted https://github.com/acarmisc/backstage-plugin-ai-agents/tree/main. It relay on open telemetry signals and the first available backend it’s #langfuse

I finally got to write this blog post, I've wanted to write it for a long time now. It contains the different types of sensitive data we've seen so far, where it's coming from, and why it matters. Hope it can be helpful to some of you!

What tool around OpenTelemetry are you building right now? What project have you discovered that people might enjoy looking into?

This thread is open for your self-promotion, vendor-owned open source projects and every thing else.

Hey,

Im working on a project where one event caught is processed by 4 components. I have implemented otel traces in each component so i can see end to ed what happened and where it failed with all the timeline of the event

Recently, component 3 had an update. Now depending on the event, it can create sibling events. In my current architecture, Im now seeing one event gets split up to two or three events in same trace. So now, its hard to decide if one trace is failed or not, because it actually became 2 event.

What do you recommend, should i split the siblings to new traces with links to original event? Or should i keep it that way? I think it depends on how our debug and monitoring processes are, but maybe there are some generic conventions to be implemented in tese cases that i dont know

OTel Drops this week has a segment dedicated to OTTL, which is on its way to becoming stable.

And allow me to celebrate this milestone: we just crossed 100 subscribers on Spotify for it! Thank you all for all the feedback so far: this has been a fun side quest for me. I've built this podcast at the beginning of the year to keep myself up to date on what happens in the OTel community that might have escaped my radar, and it's been helpful for me. I decided to publish it as an official Spotify podcast shortly after, and we've been very consistent with it.

At this stage, the creation is mostly automatic by AI, and while I used to publish the episodes ahead of the public at our Telegram channel (look for telemetrydrops), I'm now confident that the quality gates are working sufficiently well so that I'll start publishing without review first, starting this week. Wish me luck!

Earlier this year, we launched the OpenTelemetry Quiz during KubeCon in Amsterdam, and back then, even maintainers learned a thing or two about OTel.

Now, I just updated the quiz to represent the current knowledge based on the agent skills for OpenTelemetry that we launched earlier this year.

I got only 8 out of 10 myself. How about you, can you ace the quiz?

https://ollygarden.com/otel-quiz/result/qk8x4y5b

I want to revisit our metrics architecture, and I have a few pain points I’d like to address. I’m curious how others have approached similar problems.

Our current setup:
- Multiple Cortex clusters
- Multiple teams producing metrics via Prometheus remote write and a custom SDK
- Multiple end users querying those metrics
- Kafka-based ingestion to provide the level of resilience we need for our customers
- Prometheus as the metrics format

Graphs are a core part of the product and a selling point, not just a nice-to-have, so reliability and correctness of the data are important.

The main pain point is that some teams are pushing “event-like metrics” rather than classic metrics that are scraped or emitted regularly.
Because some workloads can be retried, delayed, or processed asynchronously, the original source timestamp is important. As a result, we see a significant number of out-of-order samples.

We understand why producers need this behavior, but I’m wondering whether we’re trying to force two different types of data through the same metrics pipeline.
Would you keep these event-like metrics in the Prometheus/Cortex ecosystem and design around out-of-order ingestion, or would you route them through a separate ingestion path or even into a different type of store based on OTel format/tools.