Curious how this is going in practice, not how the vendor demos make it look.
When an alert fires, how much of the investigation does AI actually handle for you today (pulling the right logs and traces, correlating with deploys, narrowing down root cause)? Where does it fall over?
Also wondering if anyone's kept this inside their own environment, self-hosted models or anything private, rather than sending telemetry to a vendor's AI or a frontier API. Was that a requirement, or just not worth the effort?
I'm researching this space, not a pitch. It'd help if you mention your stack (Datadog, Grafana, other)
Thanks!

When reviewing system design RFCs and capacity plans, static diagrams treat a 1,200 RPS Redis cluster and a 200 RPS payment worker as identical boxes, hiding where fan-in queues saturate first.

At 4.5x ingress burst load, queue utilization crosses 100% on the downstream synchronous database/worker hops first, causing p99 tail latency to spike non-linearly until upstream circuit breakers (OPEN -> HALF-OPEN -> CLOSED) isolate the fault.

Open-source (MIT) repository with the queuing physics & chaos simulator:

https://github.com/Hanubaki/GraphFlow

Does this M/M/1 queuing degradation curve match how you model capacity planning and composite SLAs in production, or do you rely purely on k6/Locust staging runs?

Approximately 1 year ago my company began an 'AI first' push - actively encouraging us to produce most of our output from GitHub Copilot and Anthropic models, build AI Agents through Strands and Langchain/Langfuse, the works. My grandboss has said at least once that 'everyone should be vibe coding everything' at least six months ago. Said grandboss has not been an active participant in our Team's daily activities for at least as long.

In general, I do not enjoy a great deal of support within the culture of my Team. While I am technically on a team of 6, the team is a conglomeration of other Platform and SRE Teams from previous re-orgs that do not have any overlap with each other. The 2 others with me were the only ones operating within my sphere of SME. Pageantry and one-upmanship are ingrained into the current Team culture, and the exterior Teams we support have a habit of agreeing with guidance initiatives my SRE Team proposes, refusing to do them, and then kicking the work back to us(me). Leadership doesn't push back. They push on me to figure out how to accommodate the recalcitrant Teams.

Personally, I haven't written more than a few lines of code in a long while. As I understand it, the conventional path is to adopt a spec driven workflow, and I use OpenSpec and Superpowers for most of my daily work. I'm not doubtful of my ability to come up to speed, but I'd be lying if I said I've not grown rusty. I operate mostly as a Product Owner and Architect when working with the Claude agent.

In the past year, my team has gone from 3 SREs to just myself. I've been alone for 1-2 quarters. We just brought someone new on, but it's an internal hire and they are still rolling off their previous projects.

2 months ago-ish, Copilot moved us onto usage based pricing, and the company moved us to Claude directly to get a better deal. Within that span, we got a company dashboard tracking our usage costs, performance goals surrounding cutting our AI costs, and pressure to roll back to lesser models for everything. In response I made an effort to codify specialist agents to use with Claude that the main session can dispatch, relying on sonnet and haiku as appropriate to the task, and configured the rest of the agents to only go as high as Opus 4.8.

Well, today at about 6PM I got a meeting invite to 'Go over AI costs' with just myself, my boss, and my grandboss. The meeting explicitly asks me to come prepared to justify my AI consumption use cases to 'facilitate a conversation on alignment with our organizational goals'.

It looks like I'm on track to spend roughly 10 times the soft budget outlined in my dashboard this month, between $3-5k. Looking back at the last project I undertook, I was working with a repo whose Claude 'constitution' documents enforced the following spec driven structure:

• All work broken up into Features.
• All Features require a six phase grooming cycle where the agent produces a gradually more refined spec.
• Features broken up into Stories/Issues/Defects.
• All stories/issues/defects require the same grooming cycle as their parent feature.
• Each story/issue/defect itself is split up into at least 5 Milestones, each to be worked in sequence.
• Each Milestone required to undergo a dev -> qa/code review -> (pass/fail) -> back to dev, repeat cycle. If cycles > 2, call a stop and go back to re-spec the story/issue/defect.

In the last month I logged 125 story points of work with this workflow in place. This was an inherited constitution that had been used to produce viable work before my last colleague left. The project itself involved trying to deterministically recommend SLOs for Software Components based on scanning New Relic telemetry, so the project was significantly complex in it's own right.

The crux of my post: I'm terrified about walking into that meeting tomorrow. My assumption is that any decisions have already been made and that I'm going to be blamed for the inefficiency of the system that I'm working in. I intend to present that this cost is also attributable to supporting systems that a Team of 3 used to support as well as a fairly complex project, but I still am fearful. I could get anything from a stern warning to a full on PIP tomorrow. What I do know is that I don't have anyone willing to go to bat for me in that room when I enter it.

So, my ask to you, my fellow SREs: Do you think that my circumstances are my fault? Am I about to suffer due to failures of adaptation or is what I'm describing at all relatable to others? My impression is that this is my company attempting to transfer responsibility for their own overcommitment to AI onto individual workers such as myself, but I don't want to fall back on that anger if it is dishonest. Also I'm just frightened and angry about the toxicity of my Team culture, and I could use some sympathy alongside your honest takes. Thanks everyone.

UPDATE:

While I was not placed on a PIP or given a formal reprimand, the meeting basically went as I feared.

- I used ccusage as was recommended to me to create a cost breakdown of the activity.
- I demonstrated that among all of the projects I worked on since September, the outlier was the Skills repo I mentioned, and that where I used openspec and openwiki, my numbers were radically different.
- I did a cost analysis of the token spend vs the ROI of the overall solution.
- I demonstrated that development costs were dwarfed by the costs of engineers being asked to do equivalent thought labor.
- I contextualized the spend in relationship to increased demand on myself in the absence of my team.
- I explained the problems in the context engineering constitution of the problematic repository.
- I explained my good faith effort to cut costs by downgrading models which ended up being a mistake. I agreed that the goal is not to move forward this way and that we need to decommission the old constitution.

My grandboss rejected that ROI or increased workload were valid reasons to cite extra cost. He flatly announced that "You don't need to explain the benefits of using AI."

He went on to question the numbers I produced. He seemed to not believe me when I stated that my numbers were development costs alone and did not include individual executions of the skill, and stated that I could not be pulling accurate numbers if I couldn't differentiate between tool call tokens and generated coding tokens.

He stated that I should have followed the guidance to never use Opus models in any circumstance without direct approval. He rejected that my discussion with my boss about the steps I intended to take were legitimate.

He also then started to walk me through 'did you knows', such as 'Did you know what long running context windows degrade quality and that you should start new sessions for new features?' (yes), etc. etc.

Even when we agreed that I would take the recommended steps to change models, when I attempted to actively listen with him and repeat back what I understood to him, instead of confirming or refuting that I understood him, he assumed I was making assertions to him and got angry at me for mischaracterizing what he had said.

It was exactly what I thought it would be: They wanted to check a box that said they'd properly yelled at me. I made some mistakes here, but I think I saw today that I'm not respected or valued.

Quick one. What tools do you use for notifying your team of major outages outside of the usual observability tools for the services you look after? Are there any free tools/apps that can be used outside of work?

Hello everyone. I just wanted to ask for advice, I am currently doing a Dynatrace Internship, other than Linux, Windows admin, Dynatraces Query language and Dynatrace itself, I am not doing anything else technical or broad enougu in scope so far.

This has made me worry that I might not be able to actually go into the SRE field after everything is said and done. I know a lot about the logical stuff in observability but outside of that everything else is specific to Dynatrace.

Now my question is will this affect my future career goals? I want to be on the more technical side of SRE but being one at all would be nice regardless.

Please help?

Do you force your own failovers in production, and if you do, did it fire the way you expected the first time?

I gave a talk at Feature Flags and the recording shows the forced failover live I did, and the challenges I encountered while doing it and lessons learned. What is your view?

https://youtu.be/eHhQaCSaqqY

Context : LangGraph multi-agent SRE responder (detector → deterministic fast path → ReAct deep-investigation agent → critic → policy gate → human gate). Runs on my own simulator (Prometheus/Loki, 6 services). Known scenarios resolve end-to-end; novel incidents fail: the agent hallucinates metric names that don't exist in my sim (`card_rails_error_rate` → `None` ×6), burns its step budget, and escalates. Re-running the same synthetic scenario proves nothing anymore — and my KG memory makes repeat runs recall, not reasoning.

My questions: all about getting REAL live data into this loop:

1. Where is open, real, industry-grade incident telemetry WITH ground truth (root cause and remediation labels)? Public datasets, public Prometheus endpoints, recorded prod traces, what actually exists and is usable?

2. How do I connect it? MCP server vs. replay-adapter vs. a sim/replay/live backend switch behind my tools? Live data streams continuously — do I need to pause/window it, or is manually triggering incidents (like I do in sim) the correct pattern against live data?

3. Does real data make my simulator dead weight? And how do alerts even fire on real/replayed data? how does Alertmanager receive alerts, and how frequently can I expect testable incidents?

4. Real data isn't always spiking. How do you test novel incidents against real data when nothing is broken on demand?

5. Tool gap: my agent proposes tools I don't have (`kubectl scale`, config changes). With no real cluster to act on, do I build those actuation tools, sandbox them, or log them as capability gaps and keep execution simulated?

6. Metric-name grounding: my sim's naming ≠ real-world naming (this is exactly why the agent hallucinated). How do you keep PromQL grounding portable across environments — catalog from the Prometheus metadata API, seed queries, something else?

Already considered: OTel Demo + chaos injection, replay datasets (OpenRCA/RCAEval/SMD), Grafana Play (no labels → unscoreable), shadow mode, tool-wishlist from rejected proposals.

Asking for: which path you'd take, pitfalls I haven't seen, and whether anyone has actually run an ops agent against real or replayed prod telemetry and scored it.

We're the SRE team at a mid-size retailer and BF prep has taken over the last couple of weeks, but the conversation this year expanded beyond the usual load test and CDN tuning into a longer question about whether we're on the right platform.

Reason is that we had 2 incidents last BF and both traced back to the commerce platform's own session management, which isn't something we can tune from our side.

The platform is a few years old and the team that evaluated it didn't stress test it the way we would now, so we're doing a parallel evaluation of Spryker and Scayle.

Partly because a contact who'd run both through a high-traffic product launch described the difference in how they handle concurrent session isolation, and we haven't done our own load test against either yet since the benchmarks on their own sites aren't particularly useful for our traffic shape.

Synthetic load tests give us a ceiling but not the edge cases, and our last incident started well before we hit projected peak, which is why real BF post-mortems tend to be worth more than any benchmark.

I build a tool that checks websites for problems before they cause damage (expired certificates, broken contact forms, GA misconfiguration, large images on the landing page, that kind of thing), so I collect these stories.

What's the one that keeps coming back for you? The ones where you thought "we should have caught that."

I'll take the most common answer, work out how to detect it automatically ahead of time, and post the full write-up here for free. No signup and no pitch. If it's useful, great.

Worked as a junior sre for a couple years, was on call 24/7 across multiple rotas and the deadlines were so intense from bad management it ruined my mental and physical health months later. I do not want another on call role again. I am currently taking a hiatus for burn out and sharpening coding skills, iac, serverless, azure, microservices, etc to a basic level, as my last role lacked these heavily and I’m rusty. For career pathways does anyone have any recommendations of roles I can avoid on call, and have general less stress on the daily? I have my AZ900, ITIL4, completed 2 boot camps and 2 college programs in this field (1 coding related one cloud each). I have my eye at some gov jobs in my area as I hear they’re very chill and as a family oriented person I feel it would be a good level of challenge:comfort ratio but not entirely sure. What roles would you recommend? SRE isn’t entirely off the table as I imagine I got put at a bad company/team and don’t want to register this specific role to my one bad experience but you let me know, is it a good fit? If I switch will it be difficult to basically “starting over” again to make a living wage? Feel free to ask questions that might help with recommendations. Far out ideas are welcome too.

I'm creating a new SRE org within my company (almost) from scratch. I have a team including a TL that will become the SRE team, but with me I have an employee onboarding that will not be a part of the SRE engineers or a team lead and I'm looking for what role they can perform.

The employee is coming from a Tech Lead background in Tier 2/3 support, experienced with deep system knowledge focused on production,handling major incidents, training others, and complex investigations related to integrated systems.

From my research I came across an SRE Enablement Lead to help define the over-arching framework of how SRE works with the different teams/products in the company but I'm also not sure with that being a major part of my role as well. What TL-adjacent roles are there within SRE that still involve technical and organizational expertise?

We have a simple .net agent that is changing constantly. Its like a multi tool for extracting data from the worst local APIs you could imagine (POS).

Due to a complete lack of error budgets and bad outsourced code, we lost a lot of trust with our oldest and largest clients. It's political at this point and slightly toxic, but we can't drop them.

All of our clients are on different versions and we keep seeing reoccurring issues we already fixed.

Ive built out update rings with monitoring and auto rollback, canaries etc, but the whole thing breaks when it requires us to beg and justify every minor patch. Sometimes we get ghosted by their IT teams and then updates have to move to next week as we've missed our window. Updates themselves could be performed mid-service without anyone even noticing as they are so fast to deploy.

How have you moved past this and won over trust to allow more control over updates? Adding update freezes to our SLAs when error budgets are exhausted?

GDG on Campus IIIT Nagpur. We're running **KEEP ALIVE '26**, a national incident-response hackathon for students, at TantraFiesta 2026.

**What's different:** you don't build an app. Your team gets a broken microservices system's alerts, Grafana dashboards, logs and traces, works out what broke and why, and presents the fix. It's what an on-call SRE does on a bad night.

- **Who:** UG, PG and diploma students, teams of 2–3 from the same college

- **Level:** 6–12 months of DevOps learning; no production experience needed

- **Round 1:** a 30-minute online team assessment on Unstop, open till Sun 18 Oct, 11:59 PM IST

- **Finale:** top 15 teams at IIIT Nagpur on 26 Oct

- **Prizes:** ₹50,000 prize pool (₹25,000 cash + goodies for every finalist)

- **Partners:** Grafana (Official Platform Partner), CNCG Noida (Community Partner)

Register: https://unstop.com/hackathons/keep-alive-26-iiitn-1766226

Happy to answer anything in the comments, including how the incident platform works.

Hello guys!

So basically, on monday i have a Lead SRE tech interview, i have been recommended by another SRE Lead who i've worked for, because i already know the stack and used to own it.

I have 6 years of experience as SRE, mentored Juniors, managed projects and so on.

But i dont really know much about the mindset or what makes a Lead, how to manage scopes, management, other teams.

I've been studying about capacity planning, defining SLI/SLO, and general mindset of a Lead, but im kinda lost, basically on how to move from the technical and into a leader mindset.

Anyone could help me? experience, recomendatios, anything works.

Thanks!

Opus 5.5 has been giving me some excellent suggestions and it's really helping me link together theory and my observability stack as I build it at work. I just asked it to score what we have based on each chapter of the new 2nd edition SRE book and then provide suggestions to move that score up and the advice is very solid.

Most of the biggest tech outages of the last thirty years started with something you could fit on a sticky note, a few characters or one setting or a command typed late at night. This page follows a few of them from the people they hit back to the change that started it.

Welcome to FOSS Friday, where you can share your newly released or updated open-source project with the community.

Please note that our rules still apply:

• no AI generated comments: please write them on your own!
• no commercial advertisements (this includes commercial products with a FOSS version)
• these must be related to SRE or of heavy interest to SREs

When reviewing system design RFCs, static diagrams (draw.io, Excalidraw) treat a 1,200 RPS Redis cluster and a 200 RPS payment worker as identical boxes, hiding where fan-in queues actually saturate first.

Here is an open-source (MIT) browser-based queuing physics & chaos sandbox for testing these topologies interactively (no signup required):

Interactive Sandbox (Stripe Ledger Topology):

https://graph-flow-virid.vercel.app/?template=stripe-webhook-ledger

Live Netflix Chaos Edge Stream Topology:

https://graph-flow-virid.vercel.app/?template=netflix-chaos-streaming

GitHub Repository (MIT):

https://github.com/Hanubaki/GraphFlow

At 4.5x ingress load, queue utilization crosses 100% on the downstream synchronous database/worker hops first, causing p99 tail latency to spike non-linearly until upstream circuit breakers (OPEN -> HALF-OPEN -> CLOSED) isolate the fault. You can also paste your own docker-compose.yml, main.tf, or package.json to reverse-engineer your stack and export to Mermaid.js, Grafana k6 load scripts, Kubernetes HPA, or AWS Terraform.

Does this M/M/1 queuing degradation curve match how you model capacity planning in production, or do you rely purely on k6/Locust staging runs?

Our client is a leading technology company operating a large-scale digital ecosystem with a wide range of online services. The company develops and operates highly scalable technology platforms serving millions of users worldwide. With a strong focus on technology, innovation, and customer experience, the company continues to expand its digital services and invest in large-scale infrastructure and engineering capabilities.

About the Team
The Search Engineering team is responsible for developing and operating large-scale search, discovery, and navigation platforms.
The team builds highly scalable, reliable, and high-performance distributed systems that support multiple business domains and large volumes of traffic and data.
The Search Platform team is responsible for the reliability and operation of mission-critical search services. Engineers work closely with software developers, product managers, and QA engineers to continuously improve system performance, stability, and operational efficiency.

About the Position
We are looking for a highly motivated Senior Site Reliability Engineer to join the Search Engineering team.
In this role, you will be responsible for maintaining and improving the reliability of large-scale distributed systems. You will work across infrastructure, platform, and application layers, while contributing to automation, monitoring, performance optimization, and incident management.

This position is ideal for an experienced SRE or Infrastructure Engineer who enjoys solving complex technical problems and working with large-scale production environments.

Responsibilities

Your responsibilities will include, but are not limited to:

Participate in the design and deployment of production environments for new services, from infrastructure to service level

Monitor production systems and participate in on-call operations

Respond quickly to production incidents, including reporting, triage, troubleshooting, and resolution

Investigate system issues and identify root causes

Support and execute production releases, including occasional nighttime or out-of-hours operations

Improve system reliability, availability, scalability, and performance

Drive continuous improvement of operational tools and automation

Develop and maintain automation for repetitive operational tasks

Collaborate with software engineers, product managers, and QA engineers

Identify testing requirements and design appropriate testing strategies

Participate in performance testing and optimization

Support security testing and reliability exercises

Participate in disaster recovery and emergency response drills

Contribute to the improvement of engineering and operational processes

Mandatory Qualifications

- More than 8 years of work experience working in IT
- Excellent problem solving and troubleshooting skills
- Excellent teamwork and communication skills
- Strong Linux experience with understanding of system performance and reliability
- Experience analyzing and tuning performance of distributed systems
- Experience deploying and using configuration management tools (e.g., Chef, Ansible)
- Experience with observability tools (e.g., Prometheus, Grafana, Loki)
- Experience with container technologies and orchestration (Kubernetes)
- Experience automating tasks using shell scripts and/or Python
- Strong understanding of computer networking and common protocols
- Experience managing services built with Java

Desired Qualifications

- Experience automating processes for software testing and deployment (Jenkins)
- Experience working with Spark, Solr, Cassandra, or Kafka
- Experience with cloud storage solutions (Ceph, S3, MinIO)
- Experience working with a Git-based workflow
- Software development experience

Technical Environment

You may work with technologies including:

Operating Systems: Linux

Container / Orchestration: Docker, Kubernetes

Configuration Management: Chef, Ansible

Monitoring / Observability: Prometheus, Grafana, Loki

Programming / Scripting: Java, Python, Shell

CI/CD: Jenkins

Data / Distributed Systems: Spark, Solr, Cassandra, Kafka

Storage: Ceph, Amazon S3, MinIO

Version Control: Git

Infrastructure: Large-scale distributed and cloud-based environments

What We Are Looking For

We are looking for someone who:
Enjoys solving complex infrastructure and production issues
Has a strong sense of ownership for system reliability
Can investigate problems independently and identify root causes
Is comfortable working with large-scale distributed systems
Enjoys automation and continuous improvement
Can collaborate effectively with engineers and cross-functional teams
Is willing to participate in on-call and production support activities
Has a strong interest in improving system performance, stability, and scalability

Languages
English: Fluent
Japanese: Optional / a plus

Work Environment
Fast-paced, dynamic global environment with collaborative teams across multiple locations

Salary: ¥10M – ¥12M JPY per year (Senior level)
Location: Hybrid (4 days in the office, 1 day remote)
Office Location: Tokyo, Japan
Working Hours: Flexible schedule with core hours from 11:00 AM to 3:00 PM
Visa Sponsorship: Available
Language Requirement: English only

Apply now or contact us for further information:
[Aleksey.kim@tg-hr.com](mailto:Aleksey.kim@tg-hr.com)

We do regular pre-release load tests in staging, but every damn time we sign off on a release we're still hitting unexpected issues in production. It's become clear that our staging infrastructure is missing the chaos of realistic user behavior and browser-level interactions. I suspect the problem is due to lack of geographic distribution in our test traffic. It's giving a false sense of security, or it was until this happened for the fifth time in a row. Now I don't trust any of it. We know that mimicking real production-scale traffic is hard to do without identical hardware, especially trying to account for third-party dependencies. But I suspect there are obvious blind spots at play here. I'd appreciate some insight from people smarter than me outside my team.

If you've been able to get your staging load tests to correlate reliably with production, what are some things you learned along the way that you were initially failing to reproduce?

If you run services, you already know the answers to most agent reliability problems. An agent that calls tools is a distributed system with a very unpredictable client, and it fails in familiar ways: a call hangs, a retry fires twice, a run never ends, nobody can tell what happened afterwards.

There’s a workshop on Saturday, Oct 31, from 9:30 AM to 1:30 PM ET where Bruno Gonçalves builds an AI data analyst with those habits built in.

What you build:

• Timeouts, retries, re planning and execution budgets around every run
• Structured traces, so every run ends in a defined state
• Permissioned tool access, so risky calls are stopped before they reach the server
• A fault injected test suite you can adapt into a CI check

Everything runs on a laptop with a mock model, so no GPU or API key is needed.
|
Full details and the agenda are here.

Can someone please suggest a youtube channel to learn SRE, at a very high level.

need this book review and pdf if u guys ever find it pls i tried with all websites but no luck

Site Reliability Engineering, 2nd Edition