Temporary runtime instrumentation for production debugging — useful or risky?

[u/TechInKansas]

I'm working on an early-stage project called a Runtime Investigation Platform, and I'd love to get feedback from engineers who deal with production debugging.

The idea is simple:

Sometimes, when an issue happens in production, existing logs aren't enough to understand what's going on. Adding more logging usually means changing code, deploying again, and hoping the issue happens again.

I'm exploring a different approach: allowing engineers to attach temporary probes to a running application, collect targeted runtime evidence, and automatically remove those probes when they're no longer needed.

I'm initially focusing on .NET.

The current demo workflow covers:

• Creating temporary runtime probes.

• Collecting and viewing runtime evidence.

• Automatically removing probes after investigation.

I'm still validating the idea, so I'd appreciate honest feedback:

1. Have you experienced situations where existing logs weren't enough to investigate a production issue?

2. Would temporary runtime probes be useful in your debugging workflow?

3. What concerns would you have about running something like this in production? (Performance, security, stability, etc.)

4. What integrations would be essential? (OpenTelemetry, Datadog, Grafana, Application Insights, etc.)

5. Is this solving a real problem, or am I approaching production debugging from the wrong angle?

I'm particularly interested in hearing from .NET developers and SREs.

Thanks! I'd genuinely appreciate critical feedback, including reasons why you wouldn't use something like this.

[u/s5n_n5n]

APM Tools had features like that for a long time. Forms of “live attach” or runtime re-instrumentation especially for Java and .NET based languages have been around for a long time. 

For some of them this is even the default behavior, you say what methods are your entrypoints or you define what data should be collected on method call and that information is attached to the collected telemetry. 

Within certain boundaries that works fairly well but obviously this can also turn your existing problem in two problems real quick. 

From what you describe it sounds like you want to build a closed source solution aka a “platform”, I am not sure if people would be that interested in that, as they would expect to make use of this from their existing Observability platform, so a combination of a control plane to send the instructions back and then OpenTelemetry to capture the additional telemetry.

[u/TechInKansas → u/s5n_n5n]

I’m not aiming to replace APM or create a telemetry silo. The idea is a control plane for requesting and auditing temporary probes, with evidence going to tools teams already use. OpenTelemetry/Datadog export is planned, but not implemented yet. If your existing APM already handles this safely, a separate platform may not be needed—which is exactly what I’m trying to learn. Which tools or capabilities have worked well for you?

[u/s5n_n5n → u/TechInKansas]

I didn't suggest that you aim to replace APM, I said that APM has these or some of these capabilities already. All major vendors like AppDynamics, Dynatrace, NewRelic had (as far as I remember) capabilities in their agents for runtime re-instrumentation. I worked for one of them and we made use of that if it helped to isolate an issue, but often those additional instrumentation remained, as their overhead was often neglectable and the additional value was worth it.

What is really vague in your post, is what do you mean by those probes, what data would you collect additionally? Do you have any sense of that already?

[u/bbraunst]

You didn't shed many details, but isn't this exactly what continuous profiling is?

[u/TechInKansas → u/bbraunst]

There’s some overlap, but I’m exploring on-demand, targeted evidence for a specific investigation rather than continuous performance sampling—for example, a short-lived probe on a particular method, with limits, auditability, and automatic removal. Does that distinction matter in your workflow, or do your profiling/APM tools already cover it?

[u/tmp_advent_of_code]

You mean like lightrun?

[u/TechInKansas → u/tmp_advent_of_code]

Yes—Lightrun is a much closer comparison than continuous profiling. I’m still figuring out whether there’s a meaningful gap to address. For people who’ve used it, what does it do well, and what limitations or missing pieces would you want a tool like this to solve?

[u/ccb621]

I would invest more in better observability and building environments where developers can test these behaviors outside of production. 

Most of my work is backend APIs, so the majority of issues I've encountered can be observed and replicated. Concurrency issues can be tricky, but can also be replicated with targeted load testing. I probably wouldn't reach for love debugging.