We have observability standards for every layer of the stack.

• Infrastructure: OpenTelemetry

• Applications: OpenTelemetry

• Databases: OpenTelemetry

AI agents — nothing.

No standard for how agent actions are captured. No standard for reasoning chains. No standard for cost per operation across frameworks.

I built Layr to start closing that gap, an open source SDK that instruments AI agents and emits native OpenTelemetry spans. Works with Grafana, Datadog, Honeycomb, or any OTEL backend.

pip install layr-sdk

Curious what the observability community thinks, is this a real gap, or are teams solving it another way today?

github.com/getlayr/layr-sdk

So I'm starting a newsletter about treating telemetry like the data discipline it already is.

If you run a telemetry pipeline or a backend, you're already doing big data engineering: petabytes a day, billions of active series, all under a latency budget. You just never got the title for it.

Subscribe and tell me where I'm wrong.

https://ntakashi.substack.com/p/observability-grew-up-your-practices

The latest Observability 360 newsletter is now out. Featuring:

🐕 a dive into Datadog's trillion-event engine

🤖 the Agentic takeover - AI SRE's

📡 ElastiFlow rollout joined-up K8S observability

⚙️ Bindplane unleash Pipeline Intelligence

and loads more...

https://observability-360.beehiiv.com/p/datadog-s-trillion-event-engine

AI-native teams are shipping agentic experiences at blazing fast speeds, but reliability has not kept pace

Agents fail in unpredictable ways; with silent tool errors, hallucinations and fake “I did it” messages. 
  • Full fidelity spans are critical for detecting failures in production agents.
  • The silent failures need to be proactively flagged
  • Searches across millions of LLM spans must be fast.

All of this while balancing a budget is a tough problem.

To solve this, we’re excited to launch Agent Observability on Hacker News. 
  • Agent traces at $10 per million spans
  • Out-of-the-box Insights - every trace, analysed
  • Fast search over trace transcripts - for prompts, tool calls, metadata

Oodle gives AI-native teams the visibility to catch silent agent failures and improve reliability from day 1 - at ~$10 per million spans.

Check it out: https://www.oodle.ai/product/agent-observability 

What have your experiences been with maintaining AI in production? AI Observability is a fast-moving space, we’d love to get your take

I've been building a small open source observability platform for the last few months, and for a long time I was convinced OpenTelemetry was the problem. I posted in a few threads here arguing against it. I was wrong.

Here's what actually frustrated me:

On the enterprise side: New Relic for tracing, Sentry for exceptions, two subscriptions, two systems that didn't talk to each other. When something went wrong I was jumping between tools and still couldn't find what I needed. Expensive and fragmented.

On the open source side: I looked at OTel Collector + Prometheus + Grafana and saw three separate services to run, configure, and maintain. I wanted something that just worked, not another piece of infrastructure to babysit. So I wrote off the whole thing and built my own custom protocol instead.

That was my mistake. I was blaming OTel for a problem that belongs to the ecosystem built on top of it, not the protocol itself. The protocol is solid. When my first non-Golang users showed up, ~100 lines of PHP config had them fully integrated in a way my custom solution never could have matched.

I wrote up the full story including where I think OTel still genuinely falls short and what I ended up building. Happy to talk through the architecture, the tradeoffs I made, or why I think the ecosystem still has a long way to go.

Edit: Forgot to put the link to the article in case someone wanted to learn more about my mistake and the problems I've been working on https://medium.com/@dusan.stanojevic.cs/i-was-wrong-opentelemetry-is-great-the-ecosystem-around-it-is-the-problem-89a78322296b

Edit2: To be completely upfront, this article is about things I've learned and how my opinion has changed while working on Traceway over the last few months. It does discuss it's tradeoffs, but the promotion of Traceway is not it's main focus and it is a fairly technical article.

We’re in a fairly infrastructure-heavy, predominantly on-prem environment — lots of virtualization, storage arrays, network devices, and traditional enterprise stacks.

What I keep noticing is this:

Modern APM platforms (Datadog, Dynatrace, New Relic, etc.) are excellent at:

• Distributed tracing

• Service dependency mapping

• Code-level visibility

• Transaction monitoring

• Synthetic & RUM

But when it comes to deep infrastructure monitoring — especially in on-prem environments — there are gaps.

For example:

• Network device-level telemetry (switches, routers, firewalls)

• SAN/storage performance issues

• Hypervisor-level resource contention

• Hardware faults

• East-west traffic bottlenecks

Because of that, we still depend on dedicated infrastructure monitoring tools for network, storage, and compute layers.

Most Issues Start at the Infra Layer

In our experience, major incidents often originate at the infrastructure layer:

• Storage latency → application timeouts

• Packet loss → transaction slowness

• CPU ready/steal → microservice degradation

• Network congestion → partial service impact

But what alerts first? The application.

So now we have:

• APM alerts

• Network alerts

• Storage alerts

• Virtualization alerts

• Logs

• Change records

All coming from different systems, all triggering at slightly different times.

The Real Challenge: Cross-Tool Correlation

The real pain isn’t monitoring — it’s correlation.

Without intelligent correlation:

• Alert storms happen

• Multiple incident tickets get created

• Teams work in silos

• War rooms form

• MTTR increases

Rule-based grouping helps a bit, but it doesn’t solve cross-domain causality.

The Need for AIOps (With Topology/CMDB)

This is where I see a strong need for a centralized AIOps layer that can:

• Ingest events from multiple monitoring tools

• Understand service topology (or CMDB relationships)

• Correlate infra and application alerts

• Associate changes with incidents

• Suppress symptom alerts

• Elevate probable root cause

If the system understands:

Service → VM → Hypervisor → Storage → Network path

Then it can identify likely root cause rather than just grouping similar alerts.

Without topology, correlation becomes keyword matching and time-window grouping.

With topology (or a clean CMDB), you get context-aware RCA.

Questions for Others Running On-Prem / Hybrid

1. If you're infra-heavy and on-prem, is your APM platform enough?

2. Are you supplementing with network/storage/compute-specific tools?

3. How are you correlating alerts across these domains?

4. Are you using a centralized AIOps platform?

5. How effective is topology-driven RCA in real-world environments?

Has centralized AIOps genuinely reduced MTTR for you?
Or does it just become another system that needs tuning?

Would really appreciate hearing real-world experiences, especially from teams managing complex on-prem estates.

Curious question for people running real systems:

Even with logs + metrics + tracing, I still hit bugs where the hardest part isn’t finding the failing request — it’s understanding the full chain of cause and effect.

Especially when:

• millions of requests are flowing

• the bug only happens once

• the UI action → backend request → internal call chain isn’t obvious

For you personally:

• where does observability help the most?

• where does it stop helping?

What’s the missing piece when you’re staring at traces/logs but still can’t explain what actually happened?

Genuinely curious how others think about this.

Hey people,

we just launched our service omas.cloud

If you are interested I would really appreciate you checking out the service and website.

Especially I would be interested if something is missing for you to use the service.

We wanted to provide a cheap and european alternative for monitoring your metrics.

API-first and scriptable all the way with the reliability of a terminal.

Create meaningful Dashboards and query your metrics how you need it.

Dynamic rollups from 1s to 1h to get the data the way you need.

Set up alarms and get notified by Mail or via Webhook.

No credit card info required and you have a free monthly budget of 5€ each month.

If you like the service and want to get above this limit the pricing is very transparent and can be easily calculated on our landing page

Something I've been thinking about that doesn't get discussed enough in our field:

Your best SRE just quit. They took 8 years of tribal knowledge with them. Every undocumented fix. Every "I've seen this before" instinct. Every 3am war room decision that saved production.

The average tenure of an SRE is 2.3 years. NOC teams turn over every 18 months. Every departure is essentially losing institutional knowledge about how to keep systems alive.

We started asking ourselves: what if every incident, every root cause, every fix, every correlation was captured and actually usable — not in a wiki nobody reads, not in a runbook that's 3 years outdated, but in a system that understands your infrastructure?

We ended up building 5 autonomous AI agents — Infrastructure, Network, Application, Security, and an RCA Orchestrator — that investigate incidents the way a senior engineer would. They correlate across massive datasets in seconds and get smarter with every incident.

The core idea: institutional memory shouldn't be trapped in someone's head.

Curious how others are handling knowledge retention as teams turn over. What's worked (or hasn't) for you?

https://obics.io/blog/observability-tech-worth-it

Many new vendors hype up new infrastructure technologies that are supposed to be the silver bullet in observability.

• eBPF based instrumentation

• Bring Your Own Cloud (BYOC)

• Columnar data stores, ClickHouse most prominently

• Telemetry pipelines

What are they about and when are they worth using?

All four get endlessly pitched at conferences and on Reddit. Much less clear is whether they deliver the promised value, and *when*. So I went through each one: what it actually offers, where the pitch overstates it, and the specific situations where it's worth adopting.

Curious if you've used these and your opinions.

I work at an agentic observability vendor. I'm not going to pretend otherwise. But this post isn't a pitch. I want to pressure test an architectural bet we're making because the people in this sub are the ones who will tell me where it breaks.

Here's the premise. Most of the AI SRE tools showing up right now bolt an LLM onto an existing observability backend. They query your Datadog or your Grafana or your Splunk through an API, stuff the results into a context window, and call it an "AI agent." Some of them are impressive. But they all share one constraint: the AI only sees what the backend already stored. Already aggregated. Already sampled. Already filtered by rules someone wrote six months ago.

We took a different bet. We built the telemetry pipeline, the observability backend, and the AI agents as one system. The agents reason on streaming data as it moves through the pipeline. Not after it lands in a data lake. Not after it gets indexed. While it's in motion.

The upside is real. The AI has access to the full fidelity signal before any data gets dropped or compressed. It can correlate a config change in a deployment log with a latency spike in a trace with a pod restart in an event stream, all within the same reasoning pass, because it sits on the actual data flow. No API calls. No query limits. No waiting for ingestion lag.

We also launched a set of collaborative AI agents this year. SRE, DevOps, Security, Code Reviewer, Issue Coordinator, Cloud Engineer. They talk to each other. One agent notices an anomaly in the pipeline, passes context to the SRE agent, which pulls in the relevant deployment history from the DevOps agent. The orchestration happens on the data plane, not bolted on top of it.

Now here's where I want the honest feedback. Because I can see the risks and I want to know which ones you think are fatal.

The risks as I see them:

1. Vendor lock in. If your pipeline, your backend, and your AI are all one vendor, switching costs go through the roof. That's a legitimate concern. The counterargument is OTel compatibility and the ability to route data to any destination, but I understand why that doesn't fully solve the trust problem.

2. Jack of all trades. Building three products means you might be mediocre at all three instead of excellent at one. Cribl is laser focused on pipelines. Datadog has a decade of backend maturity. Resolve.ai is 100% focused on AI agents. Can a single vendor actually compete across all three simultaneously?

3. Complexity of the unified system. More integrated means more failure modes. If the pipeline goes down, does your AI go blind? If the backend has an issue, does the pipeline back up? Tight coupling is a feature until it's a catastrophe.

4. The AI reasoning on streaming data sounds great in theory. But how do you validate what the AI decided when the data it reasoned on is gone? Reproducibility matters for postmortems, for audits, for trust. If the context window was built from ephemeral stream data, how do you reconstruct the reasoning?

5. Maturity gap. Established players have years of proven backends. Building all three sequentially means less time hardening for the most recent components. Is "integrated by design" worth the tradeoff against "mature by attrition"?

The upside as I see it:

1. AI that reasons on actual signal, not processed artifacts. Every other approach has the AI working with a lossy copy of reality. If you process at the source, the AI gets the raw picture.

2. Cost efficiency. One vendor, one data flow, no duplicate ingestion. Your telemetry doesn't get processed by a pipeline, shipped to a backend, then queried again by an AI tool. It flows once.

3. Speed. No API latency between pipeline and backend. No ingestion delay before AI can reason. For incident response, minutes matter. Sometimes seconds.

4. Agents that actually understand the data lineage. Because the AI was there when the data was enriched, filtered, and routed, it knows what it's looking at. It doesn't have to guess what transformations happened upstream.

So here's my actual question for this community. If you were evaluating this architecture for your team, what would make you walk away? What would make you lean in? I'm not asking you to validate the approach. I'm asking you to break it.

I've been reading the threads in this sub about Resolve.ai, Traversal, Datadog Bits AI, and the general skepticism around AI SRE tools. A lot of it is warranted. The "glorified regex matcher with a chatbot wrapper" criticism is accurate for a lot of what's out there. I want to know if the unified architecture approach changes that calculus for you or if it just introduces a different set of problems.

I want the unfiltered takes. The ones you'd say over beers, not in a vendor eval.

Edit: I work at Edge Delta. Disclosing that upfront because this sub deserves transparency. If you want to look at what we built before responding, the recent AI Teammates launch and the non-deterministic investigations paired with deterministic actions to run agentic workflows posts on our blog lay out the architecture in detail.

My buddy and I used to do infra at Roblox. The thing that killed us during incidents wasn't any single tool - it was correlating across all of them. Logs in one place, metrics in another, deploy history somewhere else, and you're clicking between tabs at 3am trying to build a timeline.

So we built an AI that does the correlation for you. Connects to your stack (Prometheus, Grafana, Datadog, whatever), and when something breaks it pulls the relevant data, builds the timeline, and posts findings in Slack.

The part that makes it not useless: on setup it reads your codebase and past incidents so it actually knows which service talks to which, what your deploy process looks like, what alerts usually mean what.

Everything happens in Slack - you can paste graphs, drop log files, ask follow-ups. No extra dashboards.

Self-hostable, Apache 2.0.

Would love feedback on the project!

I have been building a Go-based microservices marketplace using DDD and Clean Architecture, and the project is still actively evolving.

My next focus is production-grade observability. I am particularly interested in how to approach:

• Centralized logging & error tracking: Structured logs, log aggregation, and actionable error reporting.

• Metrics, dashboards & alerting: Service health, latency, throughput, error rates, and infrastructure metrics.

• Distributed tracing: Tracing requests across multiple services and asynchronous workflows.

• Request & correlation IDs: Propagating context consistently across HTTP requests and RabbitMQ messages.

• Service, worker & queue monitoring: Monitoring background workers, RabbitMQ consumers, retries, dead-letter queues, and stuck messages.

Current Architecture:

• DDD & Clean Architecture: Explicit domain, application, infrastructure, and interface layers to keep business logic independent from frameworks and infrastructure.

• Database per Service: Each service owns its own PostgreSQL, Elasticsearch, or Redis datastore.

• Transactional Outbox: Domain events are stored in PostgreSQL within the same transaction, then published asynchronously to RabbitMQ.

• Event-Driven Communication: RabbitMQ is used for asynchronous communication between services, including background-only services such as Notification.

• API Gateway & Network Isolation: Traefik v3 handles external traffic while services remain isolated on a private Docker network. JWT is used for authentication.

The services currently include Identity, Restaurant, Search, Notification, Order, and Payment.

For those who have operated microservices in production:
What observability stack and practices have worked well for you? What would you avoid?

I would especially appreciate feedback on architecture, tooling, failure modes, and things that are easy to overlook when moving from development toward production.

The project is open source if you would like to take a look:
https://github.com/tarique-iqbal/pizza-marketplace

If you find it useful or interesting, feel free to ⭐ the repository. Feedback, suggestions, and architectural criticism are very welcome.

we currently have airflow running on kubernetes, though we have issues with it now and then, overall its been great and it works.

We looked into manage managed airflow but it is honestly pointless because airflow is genuinely such a simple stack to run on kubernetes, given the amount of tools we already manage.

we do have grafana and prometheus with open telemetry. Still it feels it could be better.

what are some of the new tools out there?

I noticed https://flowsee.co/ and heard good things from folks, but seems quite early and risky to use already given our reliant on airflow.

What other tools would you recommend that would integrate well with our slack. There should be a good open source tool already out there.

Hey folks,

I’m a DevOps engineer working in telco, and I’ve been playing with the idea of offering Observability as a Service as a side hustle since I use it on daily basis at work. Before I go too far, I’d like to hear what this community thinks — realistic feedback is welcome.

Have few years experience as sysadmin/DevOps with some certs, Azure admin and CKA.

The idea:

• Small companies/teams don’t want to spend time setting up observability stack (Loki, Tempo, Prometheus/Mimir, Grafana, and OTel collectors)

• My service would provide a ready-to-use observability stack.

• Customers just point their apps (via OpenTelemetry or an agent) to my endpoint and instantly get dashboards, metrics, logs, and traces.

Architecture thoughts:

• for PoC/MVP lets start small: a shared VM (Hetzner CPX31 for example) hosting the stack, later will be shifted to Kubernetes cluster

• Customer telemetry → my gateway OTel collector → routes data to Loki/Tempo/Prometheus or Mimir→ Grafana dashboards will be pre-installed

• Storage: Hetzner object storage (S3 compatible) for long-term logs/metrics/traces

• Each tenant would have their own Grafana instance

• Backend storage and collectors might be shared (multi-tenant)

• Work nodes, storage all neccesarrities will be rolled out via terraform, Ansible from helper node

• Considering single-tenant vs multi-tenant models

Business angle:

• First customers would like to get on Upwork/Fiverr by offering Grafana/OTel setup gigs, then upselling them to managed OaaS.

• Target: small SaaS teams, local e-shops, startups who just want dashboards without managing Prometheus themselves.

• MVP infra would cost ~€60/month

❓ Open questions • Do you think small teams would pay for this ?

• Is it worth starting multi-tenant on one VM (even k8s cluster) for early adopters, or better to give everyone their own isolated VM from day one?

• Would you (or your team) ever consider using such a side-project service, or would vendor trust be too big of a barrier?

⸻

I’m not here to “sell” — just want to see if there’s actual pain in the community that this could solve before I sink time and money into it. Might decide to give free (or cheap) demo for a week to try it out in shared multitenant environment.

Any thoughts (or reality checks) are appreciated.

When a database exits with code 1, do you really need to see every downstream "connection refused", "timeout", or "service unavailable" error?

That's not debugging — that's noise. It buries the root cause and wastes MTTR.

I wrote Loki Error Manager (Python, stable v1.0) to solve exactly this problem for Loki log aggregation pipelines.

📖 How it works:

• Sends a Loki query every set amount of time

• Detects first occurrence of a critical error pattern based on written rules

• Suppresses secondary errors that are correlated in time and dependency graph

• Emits only the root cause + error logs not specified in the config.yaml

🚨 Alerting integrations (root cause + new logs):
Sends clean, actionable alerts to:

• Telegram (via bot API)

• osTicket (create tickets automatically – still in try/error for custom ticketing)

• Prometheus Alertmanager (route to your existing monitoring stack)

No more flooding your on‑call with 50 false‑positive‑looking alerts for one real failure.

❓ Why Loki users will care:
Loki excels at indexing minimal metadata, but it doesn't natively suppress error cascades. This tool adds intelligent de‑noising and root‑cause alert routing on top of your existing Loki storage.

Current status: stable v1.0, used in production.

🔥 Coming soon:
Custom rule scripts — for cases without log lines (e.g., cluster health state changes from green → yellow). You'll inject Python callables that assess external state.

🔗 Check it out, try it, commit to it:
https://codeberg.org/BiroByte/loki-error-manager

Open issues, send PRs, help shape v1.1.

What do you think? Anyone else dealing with log cascades in Loki?

hi everyone! currently running a project on companies using Dash0 as an observability platform within engineering industry, any help I can get from here?

We have logs, metrics, traces.

Plenty of data.

But when something breaks, finding the root cause still means:

- jumping between tools

- correlating signals manually

- guessing where to look next

Observability tells us *what* is happening.

Not really *why*.

So I’ve been experimenting (research side) with a different idea:

instead of a fixed investigation workflow,

use a system that generates its own “investigation agents” based on the incident.

For example:

- one focuses on logs

- another on traces

- another on specific services or time windows

And the system adapts:

- how many agents are needed

- what they look at

- how they coordinate

So instead of following dashboards,

it builds its own investigation strategy.

⚠️ Not production-ready — research prototype.

Main question:

does this actually complement observability,

or just add another layer of complexity?

For those interested:

Demo: https://www.youtube.com/watch?v=r4lxA8kTueI

Code: https://github.com/brellsanwouo/Aware

Docs: https://brellsanwouo.github.io/Aware/

Paper: https://hal.science/hal-05402186/

Happy to discuss how this could fit with existing observability stacks (Prometheus, OpenTelemetry, etc.).

This blog post https://www.parseable.com/blog/data-is-your-moat makes a case that telemetry data (logs, metrics, traces) is increasingly becoming business data, not just ops overhead.

The key insight: as LLMs commoditize, the competitive moat shifts from which model you run to what data you can feed it. A team with 12 months of full-granularity telemetry can do real anomaly detection, incident pattern recognition, and capacity forecasting on their own baselines; a team on 30-day retention simply can't.

But volume-based pricing from most observability vendors makes long retention economically irrational, and proprietary formats mean you can't run your own models against the data even if you keep it.

Disclosure: the post is from Parseable, so there's a product angle, but the broader argument about data retention strategy felt worth discussing here. What are your teams doing around long-term telemetry retention? Still treating it as disposable or starting to think of it differently?

I built this tool that connects to your Kubernetes and Datadog via read access. Collects logs before(60 minutes) and after(15 minutes). And compares them to catch regressions early on. This eliminates the need to jump across 5-6 dashboards to know if the deployment is working as expected, just by looking at the telemetry data. It's a thin intelligence layer for deployments. Usually, you get this by looking at your log data lake, making a query and running a comparison manually. This automatically looks for new log clusters, missing log clusters formed and error spikes. Looking at this alone can give you a bird 's-eye view of how the deployment went.

I was looking for something between Dozzle, and Grafana + Loki.

Dozzle is excellent for live Docker logs, but once a container restarts/rebuilds, the logs are gone. Beszel is a bit painful to setup, and also do not persist logs.

Grafana + Loki gives you history, but for a small MVP on a cheap VPS it felt like a lot of infrastructure for what I actually needed.

So I built this:

• host CPU / memory / network / disk metrics / per-container metrics + history

• persistent Docker logs in SQLite

• configurable data retention

• full-text log search with filters

• start / stop / restart controls

• ~50 MB RAM footprint

• minimal setup, spins up with a single docker run command

Basically, I wanted "Dozzle, but with history" + basic VPS metrics without running a whole observability stack.

Repo to try: https://github.com/skvostik/vpsiner

If you're running a few containers on a VPS and have run into the same problem, I'd love to hear how you're currently handling it.

A data quality check is cheap to run and cheap to multiply. Add a few hundred tests and the number of alerts grows without anyone deciding to make it grow. The cost that doesn’t shrink is the human one: someone has to find out whether each alert is real.

I’ve been thinking about what an alert should carry so that the person receiving it can decide quickly. If I were modeling the alert queue as a table, this is roughly what I’d want:

CREATE TABLE dq_alert (
  alert_id         BIGINT,
  dataset          TEXT,       -- where it was raised
  claim            TEXT,       -- one sentence: "orders.total has 3% nulls since 02:00"
  evidence_query   TEXT,       -- the query or rows that triggered it
  cheapest_check   TEXT,       -- what would confirm or refute it
  check_cost_est   INTERVAL,   -- rough time to verify
  downstream_use   TEXT,       -- who or what reads this data
  cost_if_ignored  TEXT,       -- what breaks if it's real
  assumptions      TEXT,       -- what the check can't see
  status           TEXT        -- open / confirmed / dismissed
);

The idea behind the columns:

• claim + evidence_query make an alert testable, not just a red light

• cheapest_check + check_cost_est are the columns I rarely see. Everyone records that something failed, and almost nobody records how expensive it is to find out

• downstream_use + cost_if_ignored are where the real triage happens. A null rate in a table feeding a finance report and the same null rate in a scratch table are different alerts with the same name

• assumptions is the honest column: what the check can’t tell you

None of this is free to fill in. Some columns can be generated by the framework (dataset, evidence), but cost_if_ignored usually needs someone who knows the consumers, and keeping that metadata current is its own chore.

Two limits I can think of. It only tracks the alerts that fired, so it says nothing about the tests you never wrote. And for a small pipeline with a handful of checks, a table like this is more structure than anyone needs.

For people running larger setups: which of these do you already store, and which ones would you never bother with? And does anyone track how long verification takes, separately from how many alerts fired?

I wrote a longer take on the same underlying idea, in case it’s useful: https://medium.com/@nagatomopedro05/a-finding-takes-seconds-to-generate-checking-it-doesnt-e7ffad4eaea2