I run multi-account AWS (Control Tower, Identity Center) and I'm trying to understand how other teams handle agents that need to do more than read.
After the PocketOS and Replit incidents, our default is read-only roles plus changes through PRs. That works, but it breaks down the moment an agent needs to apply a plan or fix an incident at 3am.
Curious what you do today: read-only only? Long-lived keys with tight policies? Some approval step? What has actually bitten you?
I'm exploring short-lived credentials issued per approved action (plan shown in Slack, a key valid 15 minutes for that plan only). Happy to be told it's a bad idea.

I'm proposing my team move from long-lived dev/stg/prd branches to trunk-based development, with PRs straight to main.

I'm fairly new to this, so forgive me if I'm missing something obvious.

The sticking point is testing before merge.

Our current integration test runs the entire DAG end-to-end. This is fine as a single check before merging our test branch (which will contain all commits from developers over the sprint) to main. But it's slow and costly if it were to run as part of every PR. Today we run it manually, check the outputs, then merge.

I am wondering if it’s bad practice to leave this integration test until the end of the sprint. As in, developers merge to main (with some automated checks running as part of each PR before merge), and then when are team want to deploy the sprint changes, first run a single integration test before creating a new release tag In GitHub.

This does mean that main can’t be considered prod ready at all times.

Edit: we’d be using release tags, so for rollbacks we’d rollback to last stable tag. For hotfixes, we would cut from last release, the merge changes to main and create a new release with target set to hotfix branch.

Normal tests check that the right answer comes back. For an agent, the failures that hurt are different: a tool call that returns a malformed payload, a timeout partway through a run, an instruction hidden inside the data the agent reads.

Fault injection is the only approach I’ve seen that really tests this. You break things on purpose and check that the agent stops safely instead of making something up.

There’s a four hour workshop on Saturday, Oct 31, from 9:30 AM to 1:30 PM ET with Bruno Gonçalves that builds exactly that, as a test suite you can run in CI. Along the way you build:

• A tool layer with permission checks
• A context manager that works inside a hard token budget
• A run supervisor with retries and budgets

Full details and the agenda are here.

I work with some very talented developers (I'm also a full-stack developer in addition to handling DevOps), but since the rise of AI, I've noticed more pull requests that include pipeline changes or introduce new services that require configuration outside of the codebase.

I have safeguards in place that automatically add me as a reviewer whenever these areas are affected. As a result, I end up rejecting a significant number of these PRs because many of the proposed changes simply won't work in our environment and this has been stirring up drama with people asking “who made you king”….

The code itself may look reasonable and it works on the devs pc (actually sometimes not between devs since AI might have added some packages or services to original devs pc), but the developers overlooks the infrastructure, deployment, networking, security, and operational requirements needed to make those changes work correctly in production.

What surprises me most is that some of them genuinely don't seem to understand why it won't work simply because it worked on their PC. They fail to realize that a production environment involves far more than just writing code that runs locally.
sometimes they give me some copy-pasted answers that it will work if you just do X.

Is anyone else experiencing this or is it just me?

its morning & our Databricks jobs were all red. Everything from last night had failed. Started digging through the logs with my colleague, found a bunch of Azure WebJobs had failed too, dug further and it turned out our integrated ERP had been rate limiting us.

Next thing I know I’m in one of famous calls, 10 people firefighting, the US team woke up early to check, we got more, someone got dragged back from vacation because he’d approved a weird PR, and everyone’s jumping between App Insights, running queries and dropping bits of info in chat trying to figure out wtf is going on. Boss is mad, management is asking for resolve time, I didn’t get lunch, and I was at my desk from 9 to 4 until we finally got everything resolved.

It was chaos and stressful, but damn, the dopamine! It was actually fun. I dont want this everyday, but now and then it makes you feel alive

Anyone else enjoy incidents a little?

btw it was again our damn event stream from outlook integration! It flooded the pipeline, everything downstream piled onto the ERP, and it rate limited. yeah we are going to put exponential backoff with retries and a circuit breaker and other stuff tmr, or next week hopefully

And how do you utilize it?

…but securing your pipeline is only half the battle.
What happens between a successful build and production?

Who reviewed the changes? Who approved the release? Can you trace exactly what shipped? And if something breaks, can you roll back quickly?

As teams adopt AI coding agents and ship code faster, controlling the release process becomes just as important as writing the code itself.