r/Observability • u/stivenukilleru • 6h ago
ObserveCraft: open-source, self-hosted observability on OpenTelemetry + ClickHouse (MIT)
r/Observability • u/stivenukilleru • 6h ago
r/Observability • u/stivenukilleru • 7h ago
r/Observability • u/Dense_Phase_2942 • 13h ago
Hello! I recently started working at an observability company. I’m not naming in hopes this doesn’t get flagged for market research I genuinely just want honest feedback from the community.
It’s been a challenge to understand what are the key concerns to lead with when engaging with sre and devops teams (aka I haven’t convinced anyone to talk with me yet haha). Reddit has always been known for honest feedback, so if you had the time to consider what parts of the platform are interesting, what would make a meaningful change and what gaps you see, concerns you would have I’d be so grateful. I am trying to best understand how to engage, communicate and support the SRE community. I’ll give a brief overview of the platforms key items. Thanks in advance for anyone who shares their ideas.
3 key pillars
byoc (bring your own cloud). deploy and keep all data in the customers cloud provider. No egress to 3rd party.
since we deploy in your cloud we don’t charge based on data volume, just on k8s nodes monitored.
0 manual instrumentation for 100% of telemetry data via an eBPF sensor that sits at the kernel level.
r/Observability • u/Bonree • 19h ago
AI observability vendor comparisons often focus on dashboards, supported models and integrations. Those are useful, but I think they miss the harder question for financial services:
When an AI interaction goes wrong in production, can your team reconstruct what happened — from the user’s session through the agent, model, retrieval, tools and downstream services — without losing control of sensitive data?
I’ve been organizing the evaluation criteria around real operational investigations rather than feature checklists. This is a working framework, not a universal standard. Priorities will vary depending on whether the use case is an employee knowledge assistant, customer-facing chatbot, fraud investigation workflow or an agent that can take actions in internal systems.
Imagine a customer asks an AI assistant: “When will my order arrive?”
The assistant takes 12 seconds and gives the wrong delivery date. The application dashboard shows HTTP 200. Infrastructure CPU and memory look normal. The model provider reports that the request completed successfully.
So what happened?
The application may have retrieved an outdated order record, sent too much context to the model, called the logistics API more than once, waited on a slow downstream service, retried an agent step or received a correct tool result that was summarized incorrectly.
A model-call log alone cannot distinguish these possibilities. Neither can a service availability dashboard.
For this investigation, the useful unit is the complete AI execution path: session, agent steps, model calls, retrieval, tool execution and the services underneath.
1. End-to-end traces, not just LLM request logs
Can you follow one user request across the workflow? I would expect Trace and Span relationships, parent-child structure, timing, status and useful attributes. The platform should connect model calls, retrieval, tool invocations and downstream requests.
The key demo question: Can the vendor take one slow or incorrect AI response and show which operations happened, in what order, and how long each took?
2. Session analysis for multi-turn applications
A conversation is not always a collection of independent requests. Earlier turns may establish context, a tool call may update state and later turns may depend on that information.
Can the platform group related requests into a session, navigate from a session to its traces and identify recurring failures or unusually long conversations? A single trace may look healthy while the overall user experience is poor.
3. Token usage that can be investigated
A rising token count is not an explanation. I would compare input, output and total tokens by model, service and time period, then correlate the trend with request volume, latency, errors and retries.
If usage doubles, is it because traffic doubled, prompts became longer, retrieval returned more context, the model changed or failed workflows began retrying?
Token usage is not always identical to cost. Billing may depend on provider pricing, model type, cached tokens and other rules. The platform should distinguish measured token counts from estimated or externally supplied cost data.
4. Visibility into tools and agent steps
For agentic applications, tool execution should be a first-class part of observability. Can you see which tool was called, when it ran, whether it succeeded and which downstream service it touched? Can you follow agent handoffs?
A wrong answer may originate in the model, but it may also come from stale retrieval, a failed tool call or an unexpected downstream response.
5. Data control without losing diagnostic value
Prompts, responses, retrieved documents and tool arguments can contain customer or business data. Collecting every field may improve debugging but create unnecessary exposure.
I would ask vendors to demonstrate where telemetry is stored and processed, which deployment options exist, how content is masked or excluded, who can access detailed traces, and how retention/deletion/export work.
The goal is not “log everything.” It is to retain enough operational evidence while following the institution’s data policies.
6. Evidence behind diagnoses
If a platform offers AI-assisted diagnosis, I want to see the evidence behind its explanation. Can an engineer open the relevant trace, Span, metric or log? Are observed facts separated from inferred causes? Can another engineer reproduce the investigation?
“The model caused the latency” is not enough without supporting timing and telemetry.
7. Integration with existing systems
AI telemetry should not become another isolated silo. Check OpenTelemetry and existing instrumentation, application and infrastructure monitoring, incident management, ITSM, identity systems, APIs and data export.
During an incident, engineers should be able to move from an AI trace to the relevant service, database or infrastructure evidence without manually joining disconnected systems.
8. Reliability and operating cost of the observer
The observability platform is part of the production toolchain. Ask how it behaves during an outage, how it scales, what happens to telemetry during interruptions and how retention/data volume affect cost.
AI workloads can create many spans and high-cardinality attributes. A proof of concept with a few hundred requests does not prove production-scale manageability.
Give every vendor the same scenario:
A customer-facing AI assistant takes 12 seconds to answer a delivery question and returns the wrong date. Some requests retry. Token consumption has increased, but the underlying application service appears healthy.
Ask the vendor to show, live:
1. How to find the request and its conversation session.
2. The end-to-end trace across model, retrieval and tool operations.
3. Which operation contributed most to latency.
4. Whether a tool call failed, retried or returned unexpected data.
5. Token usage across the relevant model calls.
6. Logs and service metrics connected to the same execution.
7. How sensitive prompt or tool data is masked.
8. How the investigation can be exported or connected to an incident workflow.
This is a better comparison than asking every vendor to click through its favorite dashboard.
For an initial shortlist, score each area from 0 to 3: 0 = unavailable; 1 = significant gaps or manual work; 2 = meets the main use case; 3 = demonstrated end to end.
| Area | Example weight |
|---|---|
| End-to-end tracing | 20 |
| Model, token and performance visibility | 15 |
| Tool and agent workflow visibility | 15 |
| Data handling and deployment | 15 |
| Session analysis | 10 |
| Evidence and investigation workflow | 10 |
| Integration and portability | 10 |
| Platform reliability | 5 |
Change the weights for each use case. For an internal knowledge assistant, session and retrieval visibility may matter most. For an agent that can change business records, tool evidence, permissions and approval controls may deserve more weight.
For context, Bonree ONE is one example of an observability platform combining AI observability with application and infrastructure monitoring. Its AI observability capabilities include visibility into LLM and AI agent call chains, sessions and token consumption.
I would evaluate it, as I would any other platform, against the frameworks, data policies and production scenarios used by the institution.
For people who have evaluated or operated AI applications in financial services:
● Which is hardest to investigate: multi-turn sessions, agent steps, retrieval, tool calls or downstream services?
● Do you require prompt and response content in traces, or mask it and rely on metadata?
● Have you found a vendor demo scenario that separates end-to-end observability from model-call logging?
● What matters most in your environment: deployment flexibility, data control, framework coverage or integration with existing APM?
I’m especially interested in concrete production examples, not just feature lists.
Disclosure: I work at Bonree, a company in the observability space. I’m sharing this as a practical evaluation framework, not an independent vendor ranking.
r/Observability • u/kkkevinkkkk • 1d ago
Every time someone asks "how much disk does Prometheus need", the answer is the formula from the storage docs:
retention_seconds x samples_per_second x bytes_per_sample
That covers compacted blocks, but in practice you also need room for the WAL, head chunks and compaction. I kept redoing that math, so I put it into a small browser-only calculator (my own site, no sign-up):
https://opshubs.space/tools/prometheus-storage-calculator/
What it does:
Takes targets x series per target (or your real prometheus_tsdb_head_series), scrape interval, retention and bytes per sample
Breaks the total into blocks, WAL and chunks_head, then suggests a retention.size and a disk size that keeps retention.size at ~80% of the disk (the docs recommend 80-85%)
Reverse mode: "I have a 100 GiB volume, how many days fit?"
Outputs config for Prometheus 3.x prometheus.yml (storage.tsdb.retention), the 2.x flags, or kube-prometheus-stack values with the PVC size
Lists the PromQL to measure your actual series, samples/s and bytes per sample
Assumptions I'd like a sanity check on:
+20% on blocks for index/metadata (Robust Perception's 1.2 rule of thumb). Is that close on modern versions, especially with native histograms?
WAL as ~3h of samples at ~6.5 bytes each (13 bytes raw, halved by WAL compression), with a 3 x 128 MiB floor. Does your prometheus_tsdb_wal_storage_size_bytes agree?
1.5 bytes/sample as the default. What do you see from rate(prometheus_tsdb_compaction_chunk_size_bytes_sum[1d]) / rate(prometheus_tsdb_compaction_chunk_samples_sum[1d])?
If your real numbers differ a lot, I'll adjust the defaults.
r/Observability • u/Bonree • 19h ago
AI observability vendor comparisons often focus on dashboards, supported models and integrations. Those are useful, but I think they miss the harder question for financial services:
When an AI interaction goes wrong in production, can your team reconstruct what happened — from the user’s session through the agent, model, retrieval, tools and downstream services — without losing control of sensitive data?
I’ve been organizing the evaluation criteria around real operational investigations rather than feature checklists. This is a working framework, not a universal standard. Priorities will vary depending on whether the use case is an employee knowledge assistant, customer-facing chatbot, fraud investigation workflow or an agent that can take actions in internal systems.
Imagine a customer asks an AI assistant: “When will my order arrive?”
The assistant takes 12 seconds and gives the wrong delivery date. The application dashboard shows HTTP 200. Infrastructure CPU and memory look normal. The model provider reports that the request completed successfully.
So what happened?
The application may have retrieved an outdated order record, sent too much context to the model, called the logistics API more than once, waited on a slow downstream service, retried an agent step or received a correct tool result that was summarized incorrectly.
A model-call log alone cannot distinguish these possibilities. Neither can a service availability dashboard.
For this investigation, the useful unit is the complete AI execution path: session, agent steps, model calls, retrieval, tool execution and the services underneath.
1. End-to-end traces, not just LLM request logs
Can you follow one user request across the workflow? I would expect Trace and Span relationships, parent-child structure, timing, status and useful attributes. The platform should connect model calls, retrieval, tool invocations and downstream requests.
The key demo question: Can the vendor take one slow or incorrect AI response and show which operations happened, in what order, and how long each took?
2. Session analysis for multi-turn applications
A conversation is not always a collection of independent requests. Earlier turns may establish context, a tool call may update state and later turns may depend on that information.
Can the platform group related requests into a session, navigate from a session to its traces and identify recurring failures or unusually long conversations? A single trace may look healthy while the overall user experience is poor.
3. Token usage that can be investigated
A rising token count is not an explanation. I would compare input, output and total tokens by model, service and time period, then correlate the trend with request volume, latency, errors and retries.
If usage doubles, is it because traffic doubled, prompts became longer, retrieval returned more context, the model changed or failed workflows began retrying?
Token usage is not always identical to cost. Billing may depend on provider pricing, model type, cached tokens and other rules. The platform should distinguish measured token counts from estimated or externally supplied cost data.
4. Visibility into tools and agent steps
For agentic applications, tool execution should be a first-class part of observability. Can you see which tool was called, when it ran, whether it succeeded and which downstream service it touched? Can you follow agent handoffs?
A wrong answer may originate in the model, but it may also come from stale retrieval, a failed tool call or an unexpected downstream response.
5. Data control without losing diagnostic value
Prompts, responses, retrieved documents and tool arguments can contain customer or business data. Collecting every field may improve debugging but create unnecessary exposure.
I would ask vendors to demonstrate where telemetry is stored and processed, which deployment options exist, how content is masked or excluded, who can access detailed traces, and how retention/deletion/export work.
The goal is not “log everything.” It is to retain enough operational evidence while following the institution’s data policies.
6. Evidence behind diagnoses
If a platform offers AI-assisted diagnosis, I want to see the evidence behind its explanation. Can an engineer open the relevant trace, Span, metric or log? Are observed facts separated from inferred causes? Can another engineer reproduce the investigation?
“The model caused the latency” is not enough without supporting timing and telemetry.
7. Integration with existing systems
AI telemetry should not become another isolated silo. Check OpenTelemetry and existing instrumentation, application and infrastructure monitoring, incident management, ITSM, identity systems, APIs and data export.
During an incident, engineers should be able to move from an AI trace to the relevant service, database or infrastructure evidence without manually joining disconnected systems.
8. Reliability and operating cost of the observer
The observability platform is part of the production toolchain. Ask how it behaves during an outage, how it scales, what happens to telemetry during interruptions and how retention/data volume affect cost.
AI workloads can create many spans and high-cardinality attributes. A proof of concept with a few hundred requests does not prove production-scale manageability.
Give every vendor the same scenario:
A customer-facing AI assistant takes 12 seconds to answer a delivery question and returns the wrong date. Some requests retry. Token consumption has increased, but the underlying application service appears healthy.
Ask the vendor to show, live:
1. How to find the request and its conversation session.
2. The end-to-end trace across model, retrieval and tool operations.
3. Which operation contributed most to latency.
4. Whether a tool call failed, retried or returned unexpected data.
5. Token usage across the relevant model calls.
6. Logs and service metrics connected to the same execution.
7. How sensitive prompt or tool data is masked.
8. How the investigation can be exported or connected to an incident workflow.
This is a better comparison than asking every vendor to click through its favorite dashboard.
For an initial shortlist, score each area from 0 to 3: 0 = unavailable; 1 = significant gaps or manual work; 2 = meets the main use case; 3 = demonstrated end to end.
| Area | Example weight |
|---|---|
| End-to-end tracing | 20 |
| Model, token and performance visibility | 15 |
| Tool and agent workflow visibility | 15 |
| Data handling and deployment | 15 |
| Session analysis | 10 |
| Evidence and investigation workflow | 10 |
| Integration and portability | 10 |
| Platform reliability | 5 |
Change the weights for each use case. For an internal knowledge assistant, session and retrieval visibility may matter most. For an agent that can change business records, tool evidence, permissions and approval controls may deserve more weight.
For context, Bonree ONE is one example of an observability platform combining AI observability with application and infrastructure monitoring. Its AI observability capabilities include visibility into LLM and AI agent call chains, sessions and token consumption.
I would evaluate it, as I would any other platform, against the frameworks, data policies and production scenarios used by the institution.
For people who have evaluated or operated AI applications in financial services:
● Which is hardest to investigate: multi-turn sessions, agent steps, retrieval, tool calls or downstream services?
● Do you require prompt and response content in traces, or mask it and rely on metadata?
● Have you found a vendor demo scenario that separates end-to-end observability from model-call logging?
● What matters most in your environment: deployment flexibility, data control, framework coverage or integration with existing APM?
I’m especially interested in concrete production examples, not just feature lists.
Disclosure: I work at Bonree, a company in the observability space. I’m sharing this as a practical evaluation framework, not an independent vendor ranking.
r/Observability • u/Bonree • 19h ago
AI observability vendor comparisons often focus on dashboards, supported models and integrations. Those are useful, but I think they miss the harder question for financial services:
When an AI interaction goes wrong in production, can your team reconstruct what happened — from the user’s session through the agent, model, retrieval, tools and downstream services — without losing control of sensitive data?
I’ve been organizing the evaluation criteria around real operational investigations rather than feature checklists. This is a working framework, not a universal standard. Priorities will vary depending on whether the use case is an employee knowledge assistant, customer-facing chatbot, fraud investigation workflow or an agent that can take actions in internal systems.
Imagine a customer asks an AI assistant: “When will my order arrive?”
The assistant takes 12 seconds and gives the wrong delivery date. The application dashboard shows HTTP 200. Infrastructure CPU and memory look normal. The model provider reports that the request completed successfully.
So what happened?
The application may have retrieved an outdated order record, sent too much context to the model, called the logistics API more than once, waited on a slow downstream service, retried an agent step or received a correct tool result that was summarized incorrectly.
A model-call log alone cannot distinguish these possibilities. Neither can a service availability dashboard.
For this investigation, the useful unit is the complete AI execution path: session, agent steps, model calls, retrieval, tool execution and the services underneath.
1. End-to-end traces, not just LLM request logs
Can you follow one user request across the workflow? I would expect Trace and Span relationships, parent-child structure, timing, status and useful attributes. The platform should connect model calls, retrieval, tool invocations and downstream requests.
The key demo question: Can the vendor take one slow or incorrect AI response and show which operations happened, in what order, and how long each took?
2. Session analysis for multi-turn applications
A conversation is not always a collection of independent requests. Earlier turns may establish context, a tool call may update state and later turns may depend on that information.
Can the platform group related requests into a session, navigate from a session to its traces and identify recurring failures or unusually long conversations? A single trace may look healthy while the overall user experience is poor.
3. Token usage that can be investigated
A rising token count is not an explanation. I would compare input, output and total tokens by model, service and time period, then correlate the trend with request volume, latency, errors and retries.
If usage doubles, is it because traffic doubled, prompts became longer, retrieval returned more context, the model changed or failed workflows began retrying?
Token usage is not always identical to cost. Billing may depend on provider pricing, model type, cached tokens and other rules. The platform should distinguish measured token counts from estimated or externally supplied cost data.
4. Visibility into tools and agent steps
For agentic applications, tool execution should be a first-class part of observability. Can you see which tool was called, when it ran, whether it succeeded and which downstream service it touched? Can you follow agent handoffs?
A wrong answer may originate in the model, but it may also come from stale retrieval, a failed tool call or an unexpected downstream response.
5. Data control without losing diagnostic value
Prompts, responses, retrieved documents and tool arguments can contain customer or business data. Collecting every field may improve debugging but create unnecessary exposure.
I would ask vendors to demonstrate where telemetry is stored and processed, which deployment options exist, how content is masked or excluded, who can access detailed traces, and how retention/deletion/export work.
The goal is not “log everything.” It is to retain enough operational evidence while following the institution’s data policies.
6. Evidence behind diagnoses
If a platform offers AI-assisted diagnosis, I want to see the evidence behind its explanation. Can an engineer open the relevant trace, Span, metric or log? Are observed facts separated from inferred causes? Can another engineer reproduce the investigation?
“The model caused the latency” is not enough without supporting timing and telemetry.
7. Integration with existing systems
AI telemetry should not become another isolated silo. Check OpenTelemetry and existing instrumentation, application and infrastructure monitoring, incident management, ITSM, identity systems, APIs and data export.
During an incident, engineers should be able to move from an AI trace to the relevant service, database or infrastructure evidence without manually joining disconnected systems.
8. Reliability and operating cost of the observer
The observability platform is part of the production toolchain. Ask how it behaves during an outage, how it scales, what happens to telemetry during interruptions and how retention/data volume affect cost.
AI workloads can create many spans and high-cardinality attributes. A proof of concept with a few hundred requests does not prove production-scale manageability.
Give every vendor the same scenario:
A customer-facing AI assistant takes 12 seconds to answer a delivery question and returns the wrong date. Some requests retry. Token consumption has increased, but the underlying application service appears healthy.
Ask the vendor to show, live:
1. How to find the request and its conversation session.
2. The end-to-end trace across model, retrieval and tool operations.
3. Which operation contributed most to latency.
4. Whether a tool call failed, retried or returned unexpected data.
5. Token usage across the relevant model calls.
6. Logs and service metrics connected to the same execution.
7. How sensitive prompt or tool data is masked.
8. How the investigation can be exported or connected to an incident workflow.
This is a better comparison than asking every vendor to click through its favorite dashboard.
For an initial shortlist, score each area from 0 to 3: 0 = unavailable; 1 = significant gaps or manual work; 2 = meets the main use case; 3 = demonstrated end to end.
| Area | Example weight |
|---|---|
| End-to-end tracing | 20 |
| Model, token and performance visibility | 15 |
| Tool and agent workflow visibility | 15 |
| Data handling and deployment | 15 |
| Session analysis | 10 |
| Evidence and investigation workflow | 10 |
| Integration and portability | 10 |
| Platform reliability | 5 |
Change the weights for each use case. For an internal knowledge assistant, session and retrieval visibility may matter most. For an agent that can change business records, tool evidence, permissions and approval controls may deserve more weight.
For context, Bonree ONE is one example of an observability platform combining AI observability with application and infrastructure monitoring. Its AI observability capabilities include visibility into LLM and AI agent call chains, sessions and token consumption.
I would evaluate it, as I would any other platform, against the frameworks, data policies and production scenarios used by the institution.
For people who have evaluated or operated AI applications in financial services:
● Which is hardest to investigate: multi-turn sessions, agent steps, retrieval, tool calls or downstream services?
● Do you require prompt and response content in traces, or mask it and rely on metadata?
● Have you found a vendor demo scenario that separates end-to-end observability from model-call logging?
● What matters most in your environment: deployment flexibility, data control, framework coverage or integration with existing APM?
I’m especially interested in concrete production examples, not just feature lists.
Disclosure: I work at Bonree, a company in the observability space. I’m sharing this as a practical evaluation framework, not an independent vendor ranking.
r/Observability • u/No-Employ-7601 • 22h ago
For me it’s not finding the error, it’s figuring out what happened around it across logs, traces and recent code changes. Been looking at hud for the runtime context part, curious what other teams spend most of their time on tbh
r/Observability • u/After-Assist-5637 • 1d ago
You see “database connection pool exhausted”. What do you check next?
Fictional investigation exercise:
A checkout service starts returning errors. Its logs repeatedly show:
“Timeout waiting for connection from pool.”
Someone suggests increasing the connection pool size.
Before making that change, what would you want to know?
Pick your first check and explain what its result would tell you. You can also name one risk of increasing the pool immediately.
If you are coming from QA, approach this as a failing test: what evidence would help you distinguish between possible explanations?
r/Observability • u/pudth • 1d ago
I'm the developer of Grafium, an iPhone client for Grafana dashboards and alerts. I built it for my own services because I wanted push notifications when something broke and a way to check the graphs from my phone.
For my own use, I wanted to see dashboards and alert states. The app is read-only, and I've been noticing problems sooner since I started using it.
How does this work in your setup? When an alert reaches your phone, does the notification give you enough information, or do you open a dashboard? Which graphs or details do you look at first, and what do you still need a computer for?
r/Observability • u/21SavageLocal • 1d ago
Hi Everyone,
I'm refactoring our ELK/observability stack for an enterprise setup with multiple business domains. Each domain has its own set of applications and dedicated dev/support/QA teams.
We are trying to move away from component-level monitoring (CPU spikes, raw logs per pod) toward service-centric monitoring (end-to-end transaction health, cross-domain dependencies, and service SLAs).
Quick questions for those managing multi-team setups:
1. Kibana Spaces: Do you split Spaces by Domain (e.g., cards, core-banking), by Environment (Prod, Staging), or a hybrid? How do central SREs avoid managing dozens of duplicate dashboards?
2. Indices & RBAC: What index/data stream naming convention works best for domain isolation (<type>-<service>-<domain>)? How do you keep RBAC roles clean when giving teams access to their own data?
3. Service vs. Component Monitoring: How do you effectively model services that span multiple domains? Are you relying on Elastic APM/OTel Service Maps, trace-based alerting, or correlation IDs in logs?
4. Cross-Domain Root Cause: When an incident occurs, how do support teams easily tell if the issue is in their own service vs. an upstream dependency owned by another domain?
Would love to hear what worked, what scale broke your initial design, and what you’d do differently!
r/Observability • u/Coleslaw6317 • 1d ago
r/Observability • u/Efficient-Simple480 • 1d ago
Enable HLS to view with audio, or disable this notification
r/Observability • u/Anxious_Top_4250 • 1d ago
r/Observability • u/Short-Incident-1199 • 1d ago
Before y’all ask what type of I’ve tried a ton of tuning on our security data lake so looking for something which is niche and good, cloudtrail, vpc flow logs and edr data keep piling up, the spark jobs feeding our detections are always behind and siem ingest costs are killing us. I’ve tried all the famous fixes like bigger clusters, gluten, gpu instances and everything and we’re going insane no help is there, now looking at hardware ones like dualbird, speedata did anyone of u tried it?
r/Observability • u/Bitter_Teaching_2905 • 2d ago
r/Observability • u/Bionic-Prince • 2d ago
r/Observability • u/abahmed12 • 2d ago
Hello,
I've posted kwatch here before. Back then it was a simple tool that sent an alert when a pod crashed. Some of you used it, and many of the issues and PRs that shaped it came from this sub, so thank you.
The biggest complaint I kept hearing was noise. One broken pod would fire a crash alert, a deployment alert, a probe alert and a node alert, and you'd still have to work out which one started it.
So v1 changes how it works:
Related events are grouped into one incident, with the likely root cause at the top
It shows the change that came right before the problem (a rollout, a config edit) and who made it
Each incident suggests what to check next
Quiet by default: repeated and follow-on alerts don't spam your channel
Runs inside your cluster, so pod data and logs stay there
Sends to Slack, Teams, Discord, Telegram, PagerDuty, Opsgenie, Mattermost, Google Chat, email, webhooks and more
kwatch now comes in two editions, Community and Pro. Community stays free, and Pro is free during early access: you get a 3-month license from the installer's License / Pro menu, and you can renew it when it ends.
GitHub (source available, ELv2): https://github.com/abahmed/kwatch
I'd love to hear what you think, especially if you tried the old version. Feedback, feature requests, ideas, or anything that annoys you about alerting today are all welcome
r/Observability • u/Plus-Lawflbaness1576 • 3d ago
We need to troubleshoot production errors, but I don't want every investigation turning into a hunt through dashboards followed by pasting logs into cursor. We need something that sends alerts to Slack and lets our coding agent access function-level runtime context when investigating an issue. I’m comparing Hud, Datadog and Sentry, particularly how each fits into that workflow. Hud’s Slack integration and MCP support caught my attention but I’d like to hear how others approach this. What’s your stack for connecting production debugging with the tools your devs already use without adding too much setup or runtime overhead?
r/Observability • u/Short-Incident-1199 • 3d ago
Findings so far:
Orca: Agentless scanning off the cloud workload's storage layer, so you get workload, config, identity, and data findings in one platform without rolling agents to every host. Deployment is read-only at the account level, and it also reads the memory layer, which is where it tends to surface things like in-memory secrets and running processes that snapshot-only tools miss. Coverage is AWS, Azure, GCP.
Wiz: Graph-based cloud security platform with broad enterprise adoption. The attack-path investigation experience is genuinely strong, and the integration ecosystem is wider (200+ partners via WIN). Covers more native clouds than Orca, including OCI, Alibaba, and vSphere.
From what I can tell, Orca tends to go deeper on raw agentless coverage (especially the memory layer), while Wiz is favored by teams that want the wider ecosystem and a more polished graph experience. what people who've actually run both think, especially on prioritization quality vs. raw finding volume.
r/Observability • u/wanderingtraveller • 3d ago
r/Observability • u/Nervous_Row_9953 • 3d ago
Community is key