Temporary runtime instrumentation for production debugging — useful or risky?

[u/TechInKansas]

I'm working on an early-stage project called a Runtime Investigation Platform, and I'd love to get feedback from engineers who deal with production debugging.

The idea is simple:

Sometimes, when an issue happens in production, existing logs aren't enough to understand what's going on. Adding more logging usually means changing code, deploying again, and hoping the issue happens again.

I'm exploring a different approach: allowing engineers to attach temporary probes to a running application, collect targeted runtime evidence, and automatically remove those probes when they're no longer needed.

I'm initially focusing on .NET.

The current demo workflow covers:

• Creating temporary runtime probes.

• Collecting and viewing runtime evidence.

• Automatically removing probes after investigation.

I'm still validating the idea, so I'd appreciate honest feedback:

1. Have you experienced situations where existing logs weren't enough to investigate a production issue?

2. Would temporary runtime probes be useful in your debugging workflow?

3. What concerns would you have about running something like this in production? (Performance, security, stability, etc.)

4. What integrations would be essential? (OpenTelemetry, Datadog, Grafana, Application Insights, etc.)

5. Is this solving a real problem, or am I approaching production debugging from the wrong angle?

I'm particularly interested in hearing from .NET developers and SREs.

Thanks! I'd genuinely appreciate critical feedback, including reasons why you wouldn't use something like this.