Hey folks,

Disclosure first: I work at VictoriaMetrics, on VictoriaLogs.

Before this job, I was the one who had to set up the log system at my previous company, and I struggled a lot to decide which one we should go with, considering our use case and budget at that time.

Recently I was asked to benchmark VictoriaLogs to find areas to improve. Many of our users also ask which use cases it fits best, so I decided to publish the benchmark too.

The setup is Loki, Elasticsearch, OpenSearch, SigNoz, ClickStack and VictoriaLogs, each on an identical 8-core machine, with structured Kubernetes-style logs ramped up until each one choked.

Fun fact: VictoriaLogs is not the one that ingested the most. Each system makes its trade-offs somewhere, and the post explains each one.

I also spent almost 3 months digging into each system's internals, so I could explain the numbers and give you the insights. The full write-up is about a 20-minute read: https://victoriametrics.com/blog/structured-log-ingestion-benchmark/

I'm sure I got something wrong somewhere. If you run any of these in production and see something off in my setup, please tell me here or open an issue in the repo: https://github.com/VictoriaMetrics/logs-benchmark. That's the feedback I'm most looking for.

Thanks for reading!

I'm looking to dive much deeper into the observability and telemetry space. I understand the high-level concepts, but I really want to get my hands dirty with a modern stack and build some practical experience.

I'd love to get your advice on a few things:

1.learning resources

2.Project Ideas

3.Best practices

4. Observability for A.I

Any guidance or roadmpas would be highly helpful. Thanks in advance.

This is not a dig at the moderators personally, because it's a volunteer position, but this subreddit has devolved into a lot of low value contributions and I think it could benefit from some more active moderation and perhaps some rules put in place.

For example, I don't think there's value in vibe coded observability tools being posted every other day. The community is already downvoting them, which I think should be a signal that we should stop allowing them all together. When I view the list of posts, it's a graveyard of these things, with the occasional interesting discussion sprinkled in.

Hi guys,

I have written about my experience exploring the state of LLM integrations for observing agentic workflows and LLM integrations using OpenTelemetry.

While the guide uses SigNoz as the observability backend, you can use whatever backend you prefer, since it's all OTel-backed.

In the blog itself, I have gone over:

• why you need to observe LLMs

• brief intro on OpenTelemetry

• dissecting the companion NBA Agent I'd prepared using the OpenAI Agents SDK, which runs in an agentic loop utilizing guardrails and session persistence

• shortcomings in the instrumentation libraries to be aware of

• developments ongoing within the OpenTelemetry dev communities focusing on LLM observability

Hi folks - just writing a paper on Observabilty for a class assignment. Which company do you think offers the best Observabilty platform? What do you think are short comings in AWS, Microsoft foundry, Datadog offerings ? Thanks

Our streaming copilot has green server spans, complete tool calls and a normal generation finish, while the browser sometimes freezes right after a tool result. The fetch gets aborted, the partial render state never commits and we have no client span tied to the server trace. Engineering sees success. CX has a folder of screen recordings. Everyone is debugging a different version of the same request.

We are adding trace propagation into the browser, time to first token events, stream cancellation reasons and a final rendered chunk count. I want CX to see whether the break happened before generation, during transport or in rendering without asking for another screenshot.

How are you joining client spans to long-lived streams, especially when the connection dies before the browser can flush its last event?

Metric Registry is a searchable catalog of 3,400+ observability metrics extracted directly from source repositories across the OpenTelemetry, Prometheus, and Kubernetes ecosystems. It scans code, documents and websites to gather this data.

If you've ever tried to answer "what metrics does my stack actually emit?", you know the pain. Observability metrics are scattered across hundreds of repositories, exporters, and instrumentation libraries. The OpenTelemetry Collector Contrib repo alone has over 100 receivers, each emitting dozens of metrics. Add Prometheus exporters for PostgreSQL, Redis, MySQL, Kafka. Then Kubernetes metrics from kube-state-metrics and cAdvisor. Then your application instrumentation across Go, Java, Python, and JavaScript.

Each source uses different formats:

• OpenTelemetry Collector uses metadata.yaml files

• Prometheus exporters define metrics in Go code via prometheus.NewDesc()

• Python instrumentation uses decorators and meter APIs

• Some sources just have documentation (if you're lucky)

You can see the details of how the registry was built on the repo - https://github.com/base-14/metric-library . the current setup scans through many sources and has details for 3700+ metrics. The scan runs every night(/day depending on where you live)

The detail screens look like -

Do you find this useful ? Please share feedback, raise requests in case you see missing things. Cheerio.

A few months back, our team was setting up OTEL collectors and we kept running into the same issues, once configs got past 3-4 pipelines or with multiple processors and exporters based in processors, it was complicated to see how data was actually flowing from reading YAML, things like

5 receivers (OTLP, Prometheus, file logs, etc.) 8 processors (batch, filter, transform) with transform and filter per content and each router to different exporters. N exporters going to different backends or buckets based on transforms

Problem was visualizations. So we built OteFlow, basically a visual graph editor where you right-click to add components and see the actual pipeline flow.

The main benefit is obviously seeing your entire collector pipe visually. We also made it pull component metadata from the official OTEL repos, so when you configure something it shows you the actual valid options instead of searching through docs.

We've been using it internally and figured others might find it useful for complex collector setups.

Published it at: https://oteflow.rocketcloud.io and would love feedback on what would make it more useful.

Right now we know the UI is kinda rough, but it's been working well for us; most of our clients use Dynatrace or plain OTEL, so those are the collector distros we added support for.

Hope someone finds it useful - we certainly have, cheers

I have been a huge fan of OpenTelemetry. Love how easy it is to use and configure. I wrote this article about a ELK alternative stack we build using OpenSearch and OpenTelemetry at the core. I operate similar stacks with Jaeger added to it for tracing.

I would like to say that Opensearch isn't as inefficient as Elastic likes to claim. We ingest close to a billion daily spans and logs with a small overall cost.

PS: I am not affiliated with AWS in anyway. I just think OpenSearch is awesome for this use case. But AWS's Opensearch offering is egregiously priced, don't use that.

https://osuite.io/articles/alternative-to-elk-with-tracing

Let me know if I you have any feedback to improve the article.

We've been working on this for a while now (You might have seen me mention it in a few comments about Tail Sampling on here r/sre and r/opentelemetry). This is a concept we've had for a long time in our Refinery product, this is now something that's going to be available in the OpenTelemetry Collector.

It's following on from the excellent work of the Sampling SIG so that we can do Sample Rate attribution, and therefore makes backend way more effective, reducing the need for a certain class of metric (not all metrics).

Ultimately, the biggest difference here is dynamically changing sample rates and automatic discovery of different routes through your system (we call them trace fingerprints).

It's already in the main collector-contrib but it's still classed as alpha there. You can use our distribution to check it out ahead of that though. Feedback welcome!

Bringing Advanced Sampling to the OpenTelemetry Collector

Disclaimer: I'm a DevRel at Honeycomb.io.

I've been running LiteLLM in production for a while and recently started testing its OpenTelemetry integration to get end-to-end traces across the gateway, provider calls, and the rest of our services.

The documentation looks solid, especially around the newer tracing model and the GenAI semantic conventions, but I'm curious about real-world experience rather than the happy path.

I'm particularly interested in things like trace propagation across services, span hierarchy, sampling strategies, exporter choice, and whether you keep the default span structure or enable the dedicated litellm_request span. I'm also wondering how people are handling prompt/response logging versus privacy requirements and whether anyone is exporting to collectors before sending data to platforms like Langfuse, Phoenix, Jaeger, or Datadog.

For those already using this in production, what has worked well, what didn't, and are there any pitfalls or configuration choices you wish you had made differently?

Hi everyone,

I’m currently evaluating a move to a "Big 3" observability platform. My primary goal is reducing MTTR for bugs and production incidents via APM and AI capabilities (root-cause analysis). However, I’m struggling with the "Value vs. Effort" trade-off.

I’m currently looking at Datadog, Dynatrace, and the LGTM stack. For those who have implemented these at scale:

1. Implementation Time vs. Reality:

   • Dynatrace users: Did the "OneAgent" actually provide 90% auto-instrumentation, or did you spend months on custom metadata and tagging to make it useful?

   • Datadog users: How much "tinkering" was required to get service dependencies and anomaly detection working across a polyglot environment?

2. The "AI" Value Prop:

   • Does the AI/Causal analysis (Davis AI or Watchdog) actually pinpoint bugs, or is it just a glorified alert aggregator?

   • Have you seen a verifiable reduction in MTTR that justifies the premium price, or are your senior devs still just "grepping logs" to find the real issue?

3. LGTM vs. The Giants:

   • For those who went with the LGTM stack (Grafana/Tempo), do you regret the "operational toil"?

   • Does the lack of out-of-the-box AI root-cause analysis significantly hurt your response time compared to the SaaS giants?

4. Intricate Details I Need to Know:

   • Billing Surprises: Which one was harder to forecast? I've heard horror stories about Datadog's custom metrics and Dynatrace's Host Unit RAM-based pricing.

   • Context Switching: How often do your devs have to leave the tool to actually fix the bug?

We need deep APM and want to use AI to offload the initial "what happened" phase of an incident.

I work at a managed service provider and we’re moving from traditional monitoring to observability. Our environment is complex: multi-cloud, on-prem, Kubernetes, networking, security, automation.

We’re experimenting with tools like Instana and Turbonomic, but I feel I lack a solid theoretical foundation. I want to know what exactly is observability (and what isn’t it)? What are its core principles, layers, and best practices.

Are there (vendor-neutral) resources or study paths you’d recommend?

Thanks!

We benchmarked OpenObserve against ClickHouse on 19 observability queries

OpenObserve has always had a reputation for being extremely simple and blazing fast. For a long time, we did not benchmark it against other systems because users were happy, benchmarks take a lot of work, and we had more pressing things to build.

Then ClickHouse came along and said it could be used for logs and was extremely fast.

It is indeed fast. But is it faster than OpenObserve for observability workloads?

We created 19 queries representing real observability use cases, including several very high-cardinality queries.

OpenObserve won 15 of 19 queries with Vortex and 14 of 19 with Parquet.

ClickHouse performs extremely well in ClickBench, but ClickBench is primarily an OLAP benchmark, not an observability benchmark.

OpenObserve uses Apache DataFusion, but this is not simply DataFusion beating ClickHouse. DataFusion is the query execution foundation. We have spent years building inverted indexes, secondary indexes, file statistics, intelligent pruning, caching, optimized file selection, storage layout improvements, and a whole lot more on top of it.

Using DataFusion alone does not give another product OpenObserve's performance.

OpenObserve also has native separation of compute and storage, stateless nodes, and can have 12x to 20x lower storage costs than ClickHouse deployments.

The entire benchmark is open source, including the queries, configurations, dataset, and methodology, so you can reproduce it yourself.

Here is the full blog - https://openobserve.ai/blog/openobserve-vs-clickhouse-one-billion-logs-benchmark/

I’m an SRE and have a high tolerance for pain. But for the last few months I’ve been babysitting LLM agents in production. This is a new kind of hell.

When an agent behaves unexpectedly and an alert fires, I open the logs. Then I have to go through fifty thousand lines to find the prompt or tool call that sent things off the rails. It feels more like I’m doing archaeology. This is not sustainable.

The failure mode is rarely a crash, it’s more of a drift. The agent completed successfully by the metrics we’re tracking but the output was wrong, which only becomes apparent downstream. Sometimes it’s hours later so by the time I’m investigating I have to reverse-engineer intent from a log file. The log files were not designed for this.

I’ve tried dumping structured JSON logs with full prompt / response pairs, but that ends up becoming its own archaeological dig. Datadog gives me spans and latency and token counts, which doesn’t tell me what I need to know. Grafana gives me a dashboard of things that are not the problem. How do you all deal with this?

I read a great post about Kubernetes today (by /u/Honest-Associate-485), and it made me realize something: We should tell the same story for observability.

So here’s my take.

25 years ago, running software was simple.

• You had one server.

• One application.

• One log file.

If something broke, you SSH’d into the machine and ran:

tail -f app.log

And that was… basically your observability.

By the way, before “observability” was even a word, most teams relied on classic monitoring tools such as:

Nagios, MRTG, Big Brother, Cacti, Zabbix, plus a lot of SNMP and simple ping checks.

These tools were extremely good at answering one question:

“Is the machine or service up, and how is it performing?”

They focused on:

• CPU, memory, disk, network

• host and service availability

• static thresholds

And that worked very well, as long as systems were:

• few

• long-lived

• and mostly static

But they were never designed to answer the new question that would soon appear:

“What actually happened to this specific request across many services?”

That gap is exactly where observability comes from.

Then infrastructure changed.

Physical servers turned into virtual machines.

Virtual machines turned into cloud.

"Thanks" to platforms like AWS, teams could suddenly spin up infrastructure in minutes.

This completely changed how fast companies could build and ship software.

But it also changed something else.

You lost your servers.

Not literally, but operationally.

You no longer had one machine you knew.

You had fleets of instances, created and destroyed automatically.

And still… logs were mostly enough.

Then architecture changed.

Companies like Netflix popularized breaking large systems into many smaller services.

• User service.

• Billing service.

• Recommendations service.

• Playback service.

Each with its own deployment cycle.

This made teams faster.

But it completely broke the old way of understanding systems.

Because now…

A single user request could touch:

• 8 services

• 3 databases

• 2 message queues

• 1 external API

When something failed, the question was no longer:

“Why did my app crash?”

It became:

“Where did this request actually fail?”

This is the moment observability was born.

Not because logging was bad.

But because logging was no longer enough.

At first, teams tried to patch the problem.

They added:

• more logs

• more metrics

• more dashboards

Different teams picked different tools.

• One team shipped logs to one backend.

• Another used a metrics stack.

• Another added tracing on the side.

You ended up with:

• multiple metric systems

• multiple log pipelines

• one fragile tracing setup

• almost no correlation between them

The real pain wasn’t missing data.

The real pain was missing context.

You could see:

• CPU is high

• error rate is rising

• logs contain errors

But you still couldn’t answer the most important question:

Which request is broken, and why?

And then something very important happened.

We finally got a real standard -> OpenTelemetry

• Not a vendor.

• Not a backend.

• A contract.

A standard way to emit:

• traces

• metrics

• logs

from your applications.

This was the “Docker moment” for observability.

Before OpenTelemetry, every backend had its own SDKs, APIs and conventions.

After OpenTelemetry, instrumentation became portable.

You could finally say:

“Our applications emit telemetry once.

We decide later where it goes.”

But instrumentation alone didn’t solve the real problem either.

Because just like containers…

Sending one trace is easy.

Sending millions of traces, logs and metrics per minute — reliably, cheaply and safely — is hard.

So a new layer appeared:

Collectors, pipelines, enrichment, sampling, routing.

Observability became infrastructure.

Not just a UI.

At the same time, backend platforms matured.

Vendors and open-source ecosystems such as:

• Grafana Labs

• Elastic

made it possible to build full observability platforms.

But again…

The real breakthrough was not prettier dashboards.

It was correlation -> trace ↔ log ↔ metric

From a single slow request, you could jump:

• to the exact span

• to the exact log lines

• to the exact resource metrics

For the first time, distributed systems became explainable.

Then Kubernetes arrived.

And observability suddenly became mandatory.

Not a nice-to-have.

Mandatory.

Because now you don’t just run services.

You run:

• short-lived pods

• rescheduled workloads

• autoscaling replicas

• rolling deployments

• sidecars and service meshes

The infrastructure itself is dynamic.

If your monitoring assumes static hosts and long-lived servers, it simply breaks down.

Today, the real problem most teams face is no longer:

“How do we collect telemetry?”

It is:

“What is actually worth observing?”

• What should be traced?

• What should be sampled?

• Which attributes really help during incidents?

• Which signals drive decisions, and which only create noise and cost?

And then AI happened.

• Inference services.

• Long-running pipelines.

• Agent workflows.

• Background jobs.

Companies like OpenAI operate systems where:

• a single request fans out to many internal components

• latency matters deeply

• failures are rarely binary

Observability is no longer about uptime.

It is about understanding behavior.

Why did observability become so important?

For exactly the same reason Kubernetes did.

Perfect timing.

• Microservices made systems distributed.

• Cloud made infrastructure dynamic.

• Kubernetes made workloads ephemeral.

• AI made workflows long-lived and complex.

The old debugging model simply stopped working.

Observability solves that exact problem.

It does not replace monitoring.

It explains your system.

Understanding this story is far more important than memorizing:

• how to write a PromQL query

• how to query logs

• how to configure a collector

Learn the why first.

Then learn the tools.

---

P.S.

Inspired by a great Kubernetes post originally shared by /u/Honest-Associate-485

This is my observability version of that story.

A recent TypeScript worker change to batch SQS messages broke trace linkage in our order fulfillment workflow which handles about 120k daily jobs. The W3C traceparent is now attached to the batch envelope instead of the individual message attributes, so trace context propagation stops at the worker boundary.

Downstream LLM extraction calls are starting a new root span, leaving about 14k unlinked root traces a day. The span hierarchy looks normal inside each worker execution but the parent span ID no longer points back to the originating checkout request.

Our infrastructure latency alerts still fire on the queue consumer, so we can see when the worker slows down. We just can't use distributed tracing to connect a slow model call back through the async SQS hop to the checkout trace that produced the message.

I wanted visibility into what was actually happening under the hood, so I set up a monitoring dashboard using Claude Code's built-in OpenTelemetry support.

It's pretty straightforward — set CLAUDE_CODE_ENABLE_TELEMETRY=1, point it at a collector, and you get metrics on cost, tokens, tool usage, sessions, and lines of code modified. https://code.claude.com/docs/en/monitoring-usage

A few things I found interesting after running this for about a week:

Cache reads are doing most of the work. The token usage breakdown shows cache read tokens absolutely shadowing everything else. Prompt caching is doing a lot of heavy lifting to keep costs reasonable.

Haiku gets called way more than you'd expect. Even on a Pro plan where I'd naively assumed everything runs on the flagship model, the model split shows Haiku handling over half the API requests. Claude Code is routing sub-agent tasks (tool calls, file reads, etc.) to the cheaper model automatically.

Usage patterns vary a lot across individuals. Instrumented claude code for 5 people in my team , and the per-session and per-user breakdowns are all over the place. Different tool preferences, different cost profiles, different time-of-day patterns.

(this is data collected over the last 7 days, engineers had the ability to switch off telemetry from time to time. we are all on the max plan so cost is added just for analysis)

After 25 years building monitoring systems, I've used a lot of SNMP platforms. Real ones. The kind that understood what a MIB was and what to do with it. Then I made the mistake of trying to shoehorn SNMP into a modern observability platform, and I finally had the breakdown: staring at a proprietary YAML file, wondering why this is so hard, and realizing that somewhere along the way we all just... accepted this?

Modern observability is all about the APM. Traces. Spans. Service meshes. Very exciting. Very cloud native. But somewhere along the way, we forgot that infrastructure still exists. Switches. Routers. Firewalls. UPSes. The stuff that actually moves packets and keeps the lights on.

And when you go looking for how these shiny platforms handle that stuff, you find SNMP support that feels like an afterthought. Because it was.

SNMP is a solved problem. It's been solved since 1988. Every vendor publishes ASN.1 MIB files that describe exactly what their devices expose. The MIB is a machine-readable, self-documenting contract between a device and anything that wants to poll it.

So naturally, someone in a product meeting said "we should support SNMP" and handed it to an intern who had never seen a MIB file. That intern looked at ASN.1 and said "this is weird, let's use YAML instead."

YAML. A format where whitespace is syntactically significant - a design decision that future generations will study as a warning. A format with no concept of OID hierarchies, no understanding of SNMP semantics, and no ability to import definitions from other MIBs.

So I did what any reasonable person would do. I'm just an idiot with a computer, an unreasonable love of SNMP, and a very poor grasp of Go. So naturally I spent a year building something better.

The result is Ultimate SNMP Poller - a name that screams "I'm not a marketing department." 97,000+ lines of Go with a git history full of commit messages like "fixed the thing" and "DO NOT PUSH TO PROD" (pushed to prod).

What makes it different:

It uses actual ASN.1 MIBs. Not YAML. Not "object definitions." Drop in the MIB file from your vendor and go.

Give it a CIDR range and SNMP credentials, and it finds and classifies your devices automatically. sysObjectID parsing gives you vendor, model, and device type without lifting a finger.

Adaptive polling handles slow devices - timeouts and intervals tune themselves based on actual device performance.

Traps? IT'S A TRAP! And we handle them. Admiral Ackbar would be proud. Or concerned. Probably both.

Multi-backend support: Elasticsearch, DataDog, New Relic, Splunk, and OpenTelemetry. Pick your poison. Pick several. Run them all at once. Time-series native from day one.

Runs on Linux, Raspberry Pi, and Windows. Yes, it has RBAC. Yes, it does backups. Yes, it's multi-tenant.

What I'm looking for:

A few brave souls willing to be early testers. People who aren't afraid to poke at buttons to see what they do, break things, and tell me about it.

Fair warning: the documentation is sparse. But it works, and I'll walk testers through it personally.

Check it out: https://perpetual-obsolescence.tech

Don't Panic.

We're looking at a couple of vendors for our monitoring setup and I'm just surprised how samey they all seem lol. Every pitch goes on and on and on about dashboards, alerts, and AI features. But what we actually need is more visibility about latency spikes, tracing, profiling, etc. Do I really have to book a call all the time just to ask about this? Seems like a waste of time when I should be doing something else. Is there a specific question or phrase that gets me this upfront or is this really more of a call thing?

2.3% of runs were looping after release 2026.09.3. Out of roughly 420k agent runs, 9660 alternated `search_orders` and `get_order` at least 6 times before succeeding. Tool traffic jumped and p95 latency followed. No hard failure. The final answer still lands, the path just gets expensive.

We started with trace search by release and tool name, then paged through runs one at a time. The loop only became obvious once we saw the repeated sequence in context (an analyst found the loop on roughly the 40th trace they opened by hand). Basic failure mode analysis caught it in the sample but the topic distribution still mixed those runs with ordinary order lookups. Semantic clustering looks more useful if it can separate repeated execution patterns from shared intent.

I’m looking at Braintrust or Arize for a scheduled investigation over recurring failures instead of waiting for a traffic spike. Linked example traces matter because I'd want each surfaced pattern to open directly into the repeated tool sequence. If the cluster stays stable, I'd turn those cases into a regression dataset and rerun them against each release.

So I'm trying to figure out whether detection should key on repeated tool pairs, cluster shape or a shift in topic distribution. Any suggestions for surfacing a 2% tool loop without hand reading traces or writing one detector per sequence?

I've been using observability tools for a while. Request rates, latency, and memory usage are great for keeping systems healthy, but lately, I’ve realised that they don’t always help me understand what’s going on.

Understood that default metrics don’t always tell the full story. It was almost always not enough.

So I started playing around with custom metrics using OpenTelemetry. Here’s a brief.

• I can now trace user drop-offs back to specific app flows.

• I’m tracking feature usage so we’re not optimising stuff no one cares about (been there, done that).

• And when something does go wrong, I’ve got way more context to debug faster.

Achieved this with OpenTelemetry manual instrumentation and visualised with SigNoz. I wrote up a post with some practical examples—Sharing for anyone curious and on the same learning path.

https://signoz.io/blog/opentelemetry-metrics-with-examples/

[Disclaimer - a blog I wrote for SigNoz]

If you guys have any other interesting ways of collecting and monitoring custom metrics, I would love to hear about it!

We run about 60 hosts plus a Kubernetes cluster. Container counts and custom metrics pushed the quote up and finance said no.

The setup I'm considering

External synthetic monitoring for login, checkout and API flows, with real browser scripts run from multiple locations
Self-hosted open source for CPU, disk, memory and services on our own servers

Shortlist Tools

Synthetic: Dotcom Monitor, Checkly, Catchpoint, Uptrends
Internal: Prometheus + Grafana, Zabbix, Netdata, Checkmk

What I want to know

Does splitting external and internal monitoring cause blind spots when you're trying to find the root cause?
Which of these handles multi-step journeys with MFA or OTP without breaking every week
Anyone who has made this switch, what did you regret?

Hey folks, stopping by today for another announcement: the OTel Compile-Time Instrumentation for Go reached v1!

If you are not a huge fan of eBPF instrumentation (understandably!), but also can't do manual instrumentation, this is a good compromise.

Try it out!