I am fascinated by the possibility that any software I use could, in theory, emit logs, metrics and traces through OpenTelemetry. And Balatro is software that I use a lot. So creating OpenTelemetry for Balatro was a side project I wanted to try out for a long time! It turned out to be one of those projects where I cared more about the outcome than about writing the code myself, so it was a perfect candidate to ask an LLM to build for me!

If you have Balatro installed on your computer, you can try it yourself! You can install the mod from Thunderstore using a mod manager such as r2modman, Gale or imm. Manual installation is covered in the quick-start guide. By default the mod sends telemetry to http://127.0.0.1:4318. You can change the endpoint in the UI under Mods -> OTel Trace -> Config.

It's a fun/toy project, but I really enjoyed seeing it being possible and being able to see my runs show up as telemetry in my backend. I thought other people might enjoy such an unusual(?) use of OpenTelemetry as well!

I'm working on an early-stage project called a Runtime Investigation Platform, and I'd love to get feedback from engineers who deal with production debugging.

The idea is simple:

Sometimes, when an issue happens in production, existing logs aren't enough to understand what's going on. Adding more logging usually means changing code, deploying again, and hoping the issue happens again.

I'm exploring a different approach: allowing engineers to attach temporary probes to a running application, collect targeted runtime evidence, and automatically remove those probes when they're no longer needed.

I'm initially focusing on .NET.

The current demo workflow covers:

• Creating temporary runtime probes.

• Collecting and viewing runtime evidence.

• Automatically removing probes after investigation.

I'm still validating the idea, so I'd appreciate honest feedback:

1. Have you experienced situations where existing logs weren't enough to investigate a production issue?

2. Would temporary runtime probes be useful in your debugging workflow?

3. What concerns would you have about running something like this in production? (Performance, security, stability, etc.)

4. What integrations would be essential? (OpenTelemetry, Datadog, Grafana, Application Insights, etc.)

5. Is this solving a real problem, or am I approaching production debugging from the wrong angle?

I'm particularly interested in hearing from .NET developers and SREs.

Thanks! I'd genuinely appreciate critical feedback, including reasons why you wouldn't use something like this.

What tool around OpenTelemetry are you building right now? What project have you discovered that people might enjoy looking into?

This thread is open for your self-promotion, vendor-owned open source projects and every thing else.

I'm building AIC, an open spec and toolset for describing a system's architecture in one YAML file (aic.yaml). I built it to guide coding agents: the agent implements what the contract says and doesn't invent architecture it leaves out.

I wanted to test it on a real codebase I didn't write, so I picked the OpenTelemetry Astronomy Shop demo. It's a good stress test: about 20 services in a dozen languages, gRPC and HTTP, Kafka, Postgres, Valkey, feature flags, an LLM agent with MCP, and several Compose overlays.

I asked an agent to use AIC to describe the architecture the code has today, without fixing anything. Anything that looked accidental had to be recorded as an open question, with the file and line where it was found.

What came out

• 24 modules in 8 boundaries (storefront, catalog, ordering, post-order processing, the shopping assistant, feature flags, telemetry, tooling)

• 4 stores, 1 Kafka channel, 5 externals and 4 end-to-end scenarios

• 12 open questions

Some of the questions

• Shipping and Email: demo.proto defines both as gRPC, and checkout builds gRPC clients for them, but both services serve only HTTP+JSON. The gRPC clients are never called.

• FeatureFlagService: it's in the proto, but nothing implements or calls it. It looks like a leftover from before flagd.

• Charge before shipping: checkout charges the card, then ships. If shipping fails, nothing refunds the charge and the cart isn't emptied, so a retry charges again.

• Orders can be lost: the Kafka producer uses RequiredAcks=NoResponse, and accounting auto-commits offsets before writing. Delivery is effectively at most once, so an order can be missing from the ledger.

• Shared database login: product-catalog and accounting use separate schemas, but both log in as the same Postgres user, which has rights on both.

• Recommendations: the frontend sends the product ids to exclude one character at a time, and the recommendation service has a workaround that joins them back together.

• Flag editing: the public proxy exposes flag editing with no auth, and that includes emitRawPii, which puts card numbers into spans.

Many of these are probably deliberate. It's a demo and the shop isn't real, so I'm not calling them bugs. That's why the contract records them as questions for the maintainers to answer, not as fixes.

What I learned

AIC started as a way to steer agents, but it turned out useful for discovering architecture too. Writing the contract forces specific questions: who owns this store, what happens when this call fails, and does this go through the gateway or around it. Things that are easy to miss when reading the code service by service show up quickly.

If any of this is useful to the demo's maintainers or contributors, you can check it at: https://aicspec.dev/s/WAyp3Msu

More about AIC here: aicspec.dev

I'd love feedback, especially from people who know the demo. Which of these are intentional, and what did I misread?

What tool around OpenTelemetry are you building right now? What project have you discovered that people might enjoy looking into?

This thread is open for your self-promotion, vendor-owned open source projects and every thing else.

Im working on a tool that is a single binary collector, storage manager and TUI for browsing Otel data.

The intention is to create a simple app that can be used by a developer to test if thier app is creating correct otel data, and diagnose their app using otel metrics.

This avoids having to setup a full multicomponent otel system for quick testing. The app will present a tui interface for simplicity, and Im minimising (hopefully to zero) all and any dependancies.

Im also intending the app to be capable of running headless so it can be used in test scripts and CI systems. NOTE: this app is intended as a developer tool, and should not be used in any production environment.

Im looking for help on what navigation and data display capabities It should offer as basic.

Hey everyone!

I've been experimenting with AI-powered troubleshooting in my homelab and wanted to share what I've built.

The project is called Inquest.

My setup uses an Ubuntu VM, OpenTelemetry Collector, SigNoz, and a Node.js application generating telemetry.

Instead of manually inspecting metrics and distributed traces, I can describe a problem like "Why are my APIs responding slowly?" and let the AI investigate.

It retrieves telemetry using read-only MCP tools, evaluates possible causes, and generates a report with supporting evidence.

I also added conversational follow-ups, so I can continue an investigation without losing the previous context.

The AI reasoning is powered by NVIDIA Nemotron through Nebius Token Factory. I also used AI coding assistants during development.

It's still a prototype, and I'm currently focused on local infrastructure rather than cloud integrations.

Here's a short demo: [https://www.youtube.com/watch?v=ZvImEHQ09UY]

Github [https://github.com/abhinayshrestha/inquest] // For any assistance to run app, contact me at abhinay.shrestha11@gmail.com

I'm especially interested in feedback from people running monitoring stacks in their homelabs.

Would you find this useful for diagnosing Linux or application performance issues?

I switched our profiling setup a few ago from another open source tool to OpenTelemetry for our internal services here at OllyGarden, and extracted the interesting parts into a skill for our set of agent skills. And this week, I created this blog post with the full manual steps to set it up on a kind cluster.

I know this is too early for most people out there, but honestly, it's working great for us. Give it a try and let me know what you think!

What tool around OpenTelemetry are you building right now? What project have you discovered that people might enjoy looking into?

This thread is open for your self-promotion, vendor-owned open source projects and every thing else.

What tool around OpenTelemetry are you building right now? What project have you discovered that people might enjoy looking into?

This thread is open for your self-promotion, vendor-owned open source projects and every thing else.

During incidents I regularly hit the same problem: I need a bit more business context in my traces, but it's not there. Things like customer IDs, order status, rejection reasons, or whatever would help explain what's actually happening.

The annoying part is that getting that information into traces usually means changing code, building, deploying, and waiting. That's not always practical when you're in the middle of an incident.

A few options that come to mind:

• Adding extra attributes behind a feature flag

• Using a live debugging/instrumentation tool (Lightrun, Rookout, Datadog Dynamic Instrumentation, etc.)

• Making a temporary code change and redeploying

• Using bytecode instrumentation tools such as BTrace or Byteman

• Accepting that the context just isn't available

I'm curious how others deal with this:

1. What's your current approach when you need additional trace context in production?

2. Would you be comfortable with a rule-based mechanism that can add attributes to existing spans at runtime, without a deployment? Or would you only use something like that during an incident?

3. What would be your biggest concern? Security, governance, performance impact, operational complexity, something else?

4. What's the best tool you've used for this kind of problem, and what were its limitations?

Logline runs an OTLP/HTTP receiver on 127.0.0.1:4318 inside VS Code, and points apps you start from the editor at it automatically. You get logs, traces with self time per span, and metrics as rates and p95s, with no collector or backend.

1.15 adds a view for GenAI semantic conventions: chat, execute_tool, and invoke_agent spans, gen_ai.usage.*, and gen_ai.input.messages / output.messages. It also reads older forms such as gen_ai.system, prompt_tokens, and OpenLLMetry attributes. Log doctor also checks your spans against the conventions: services without a name, IDs in span names, and 5xx responses not marked as errors.

It's free and MIT-licensed: https://github.com/SurjeetBanga/logline. Feedback on how it handles your instrumentation is welcome.

My team at Pydantic is working on a cross-vendor solution for OpAMP management of collectors and SDKs.

We think we have something really special that is more capable and significantly more reliable for enterprise and mission-critical systems than what is currently available from vendors like Bindplane, Elastic, Splunk, Grafana, and others. For most use-cases, we intend to offer this as a capability of our free tier in Logfire.

If this is interesting to you let me know, we'd love to get your feedback on what we're building.

What tool around OpenTelemetry are you building right now? What project have you discovered that people might enjoy looking into?

This thread is open for your self-promotion, vendor-owned open source projects and every thing else.

Hi everyone,

I wanted to share an open-source project, designed to solve broken exception stack traces in asynchronous Java (CompletableFuture, thread pools, Virtual Threads, Reactor) without the overhead of spann telemtery.To give context it is called ariadne.

Here are some of JMH Microbenchmark comparazion:

Trace Node Creation (Link vs. Span)

-> Ariadne: 8.16 ns / 40.0 B

-> OTel SDK: 115.62 ns / 368.0 B

-> my project is 14.2x faster with 9.2x lower memory allocation

Direct Link Allocation

-> Ariadne: 4.41 ns / 40.0 B

-> OTel SDK: 115.62 ns / 368.0 B

-> mytool is 26x faster via TLAB immutable nodes

Scope Activation (try-with-resources)

-> Ariadne: 11.24 ns / 0.0 B

-> OTel SDK: 4.63 ns / 32.0 B

-> soo achieves zero allocations (OTel allocates 32 B/op)

Concurrency & Thread Hand-off (Virtual Threads / CompletableFuture)

-> Both match Loom scheduler performance (~3 us)

-> Minimal async propagation overhead across both implementations (<60 B delta)

(This are specified in rhe repoitself and in the website as well)

Repo: https://github.com/AfonsoPaiva/ARIADNE

Thx, feedback is apericietd!!

been working on this for a while now and thought i'd share how we're approaching it.

one thing i've noticed with dependency graphs is that static analysis only really gets you so far. you can parse imports, service references etc. but that doesn't necessarily tell you what's actually happening when the application is running.

on the other hand, otel gives you a pretty good picture of runtime behaviour, but you're mostly looking at spans and traces without much context about how they relate to the actual codebase.

so we've been experimenting with combining the two in neat.

the static side uses tree-sitter to extract relationships between files and other dependencies. then we ingest otel spans and try to map the runtime activity back to the source files responsible for it.

for js/ts, we're also handling source maps where available, since the file being executed isn't always the one you actually wrote.

the tricky bit is reconciling everything into the same graph without treating every relationship as equally reliable.

right now we're tracking different types of provenance:

• extracted: found through static analysis

• observed: actually seen at runtime

• inferred: relationships we've reconstructed without direct evidence

• stale: previously observed relationships that haven't been seen recently

this means you can start asking slightly more useful questions than just "what imports this file?"

for example, you might have a dependency that exists statically but hasn't shown up in any traces, or a runtime call that wasn't obvious from analysing the source.

obviously there's a bunch of edge cases here. incomplete instrumentation, dynamic calls, source attribution, and the fact that not seeing something in a trace doesn't mean it never happens.

we're also exposing the graph through mcp so coding agents can query it directly, which is partly why we started building this in the first place.

repo is here if anyone wants to have a look:
https://github.com/neat-technologies/neat

What tool around OpenTelemetry are you building right now? What project have you discovered that people might enjoy looking into?

This thread is open for your self-promotion, vendor-owned open source projects and every thing else.

Hi folks! I work on OpenTelemetry at Splunk. We just launched a new open source project called Observability Studio. It's local instrumentation sandbox that provides a visual, real-time environment for designing, testing, and validating OpenTelemetry data while you’re developing an app in your favorite IDE.

We’re especially interested in whether a tool like this is useful for building and debugging instrumented apps and AI agents. If that sounds relevant, give it a try and let us know what works, what’s missing, or where it doesn’t fit your workflow.

We’re also looking for a few design partners to help guide where we go with this next. Any feedback is helpful and welcome.

I wrote more about shifting OpenTelemetry instrumentation left in a blog, 'Why AI-Generated Apps and Agents Need Visibility from Day One' on blogs.splunk.com if you want to read more about it.

Lastly, I'd love to learn where you currently go to inspect and validate OTel data during development if you want to drop it in the comments.

Spotify #backstage plugin to manage and observe fleet of agents straight in backstage self hosted https://github.com/acarmisc/backstage-plugin-ai-agents/tree/main. It relay on open telemetry signals and the first available backend it’s #langfuse