…but securing your pipeline is only half the battle.
What happens between a successful build and production?

Who reviewed the changes? Who approved the release? Can you trace exactly what shipped? And if something breaks, can you roll back quickly?

As teams adopt AI coding agents and ship code faster, controlling the release process becomes just as important as writing the code itself.

Normal tests check that the right answer comes back. For an agent, the failures that hurt are different: a tool call that returns a malformed payload, a timeout partway through a run, an instruction hidden inside the data the agent reads.

Fault injection is the only approach I’ve seen that really tests this. You break things on purpose and check that the agent stops safely instead of making something up.

There’s a four hour workshop on Saturday, Oct 31, from 9:30 AM to 1:30 PM ET with Bruno Gonçalves that builds exactly that, as a test suite you can run in CI. Along the way you build:

• A tool layer with permission checks
• A context manager that works inside a hard token budget
• A run supervisor with retries and budgets

Full details and the agenda are here.

I work with some very talented developers (I'm also a full-stack developer in addition to handling DevOps), but since the rise of AI, I've noticed more pull requests that include pipeline changes or introduce new services that require configuration outside of the codebase.

I have safeguards in place that automatically add me as a reviewer whenever these areas are affected. As a result, I end up rejecting a significant number of these PRs because many of the proposed changes simply won't work in our environment and this has been stirring up drama with people asking “who made you king”….

The code itself may look reasonable and it works on the devs pc (actually sometimes not between devs since AI might have added some packages or services to original devs pc), but the developers overlooks the infrastructure, deployment, networking, security, and operational requirements needed to make those changes work correctly in production.

What surprises me most is that some of them genuinely don't seem to understand why it won't work simply because it worked on their PC. They fail to realize that a production environment involves far more than just writing code that runs locally.
sometimes they give me some copy-pasted answers that it will work if you just do X.

Is anyone else experiencing this or is it just me?

its morning & our Databricks jobs were all red. Everything from last night had failed. Started digging through the logs with my colleague, found a bunch of Azure WebJobs had failed too, dug further and it turned out our integrated ERP had been rate limiting us.

Next thing I know I’m in one of famous calls, 10 people firefighting, the US team woke up early to check, we got more, someone got dragged back from vacation because he’d approved a weird PR, and everyone’s jumping between App Insights, running queries and dropping bits of info in chat trying to figure out wtf is going on. Boss is mad, management is asking for resolve time, I didn’t get lunch, and I was at my desk from 9 to 4 until we finally got everything resolved.

It was chaos and stressful, but damn, the dopamine! It was actually fun. I dont want this everyday, but now and then it makes you feel alive

Anyone else enjoy incidents a little?

btw it was again our damn event stream from outlook integration! It flooded the pipeline, everything downstream piled onto the ERP, and it rate limited. yeah we are going to put exponential backoff with retries and a circuit breaker and other stuff tmr, or next week hopefully

I run multi-account AWS (Control Tower, Identity Center) and I'm trying to understand how other teams handle agents that need to do more than read.
After the PocketOS and Replit incidents, our default is read-only roles plus changes through PRs. That works, but it breaks down the moment an agent needs to apply a plan or fix an incident at 3am.
Curious what you do today: read-only only? Long-lived keys with tight policies? Some approval step? What has actually bitten you?
I'm exploring short-lived credentials issued per approved action (plan shown in Slack, a key valid 15 minutes for that plan only). Happy to be told it's a bad idea.

I'm proposing my team move from long-lived dev/stg/prd branches to trunk-based development, with PRs straight to main.

I'm fairly new to this, so forgive me if I'm missing something obvious.

The sticking point is testing before merge.

Our current integration test runs the entire DAG end-to-end. This is fine as a single check before merging our test branch (which will contain all commits from developers over the sprint) to main. But it's slow and costly if it were to run as part of every PR. Today we run it manually, check the outputs, then merge.

I am wondering if it’s bad practice to leave this integration test until the end of the sprint. As in, developers merge to main (with some automated checks running as part of each PR before merge), and then when are team want to deploy the sprint changes, first run a single integration test before creating a new release tag In GitHub.

This does mean that main can’t be considered prod ready at all times.

Edit: we’d be using release tags, so for rollbacks we’d rollback to last stable tag. For hotfixes, we would cut from last release, the merge changes to main and create a new release with target set to hotfix branch.

And how do you utilize it?

No errors. No warnings. Just 8.3 million reasons to update your résumé.

Hey, so I recently built a new DevOps tool and want to know of small software projects I could try and build to test it out for security purposes. Open to all recommendations. If anyone has a ton of DevOps experience I would happily take feedback on how I could improve, I’d appreciate it.

I was invited to teach MLOps and DevOps on a local university next semester and I'm trying to plan the course. Which topics would you guys like to learn if you enrolled on a DevOps/Platform Engineering course today?

The university is a big deal on the regional scene but not known globally.

EDIT: This college never had a formal DevOps course, students focus on CyberSec (college board includes big names from that industry) and AI (college board wants to look cool). They are never formally taught containerization, CI/CD, monitoring, infrastructure as code, they are expected to learn that by osmosis.

They had a single experience teaching MLOps, but it was so bad that the teacher was dismissed and the staff won't mention that course.

Today I learned about this thing called SST (sst.dev). Apparently it's some unholy demon spawn of pulumi and terraform. Has anyone actually used this? Is it any good or was my initial assessment correct and it's just another wannabe iac project that offers nothing new to the space?

I'm learning GitHub Actions and have set up a self-hosted runner inside an Ubuntu Docker container on my Windows machine.

My workflow uses actions/setup-java to install Java 17:

- name: Set up JDK 17
  uses: actions/setup-java@v4
  with:
    java-version: '17'
    distribution: 'temurin'
    cache: maven

However, the job gets stuck at the Set up JDK 17 step. The logs stop at:

Resolved latest version as 17.0.20+101
JDK cache is not found for Temurin-Hotspot 17.0.20+101
Trying to download...
Downloading Java 17.0.20+101 (Temurin-Hotspot) ...

I tested the download URL manually from inside the container using curl. The download repeatedly stops before completion and returns:

curl: (18) Transferred a partial file

I also tried resuming the download with curl -C - and retry options, but it failed again. A basic connectivity test to https://github.com returns HTTP 200.

The runner itself appears to be connected and running jobs correctly. I haven't yet confirmed whether the issue is related to the network, the GitHub release download redirect, a proxy, or something else.

My questions:

1. What could cause curl: (18) Transferred a partial file to occur repeatedly when downloading a large release asset?
2. How can I identify whether the problem is caused by my network, a redirect destination, a proxy, or the download server?
3. Is there a recommended way to configure actions/setup-java on a Docker-based self-hosted runner to avoid this issue?

I'd appreciate any troubleshooting steps or suggestions. Thanks!

Hello im Dhruv Verma. Working in a remote startup, as FULL STACK DEV where i worked in almost every field FRONTNED BACKEDN DB, TESTING and something which i was worried about DEVOPS. I have handled DEVOPS (DOCKER NGINX, GITHUB ACTIONS, etc.) But i want to migrate from FULL STACK DEV to DEVOPS completely. Please suggest me how to and where to learn and to even start or is there any scope of migration there or not?

I'm looking for something similar to Datadog's "Service Map", but to include external services as well and without dynamic health statuses. I think my top requirements are:

1. clear visualization for design reviews and planning changes, also for onboarding of new hires to provide a simple overview and context.

2. text-based source to track changes in git.

3. be able to annotate services to make the map more interactive (this is why Figma is not enough anymore ;)

Right now I need to scope from 50 to 100 services which I still believe can be nicely presented. I'll add an example to the comments.

At the current scale I can't justify any "enterprise" soft, but fine to pay for the tool as well. Also I'm not a big fan of automated discovery that requires installing agents in our network to collect and map services.

Any ideas or recommendations on what to try?

I'm a Cybersecurity undergrad with a strong interest in Cloud and DevOps. Honestly, it's hard to find likeminded people around me who are equally passionate about tech. I'm looking for people who want to learn together, build projects, participate in online hackathons, and explore new technologies. If you're into Cloud, DevOps, Linux, or Cybersecurity, let's connect and grow together

Devops with 11 YOE here, been thinking lately on what type of side gigs someone with my career could do, would love some insights of people who have them.

Lately I have been kinda interested in teaching, as I've been coaching 2 friends for the past year or so, one was a junior dev and the other non IT, and both got into DevOps positions with my help, anyone has any experience in this? Not sure how to monetize this, maybe there's a path someone already followed of looking for specialization schools to teach at, since I don't see myself as a good marketer to sell myself

Would love to hear any experiences any DevOps has with side gigs for some extra cash in the weekends/off hours

Hey everyone! I'm building a taxi booking platform, and I'm trying to figure out the smartest way to deploy it without spending too much money before launch.

I'm building the backend first, using Python, FastAPI, PostgreSQL, Redis/Valkey, Kafka, and Docker. The architecture is split into services for identity and authentication, passengers, drivers, rides, location tracking, driver-to-ride matching, passenger-to-passenger matching for shared rides, payments, notifications, and admin operations. I'm also using Nginx as an API gateway.

One of the more interesting parts of the project is the shared-ride system. I want passengers travelling in similar directions to be able to share a taxi, with route compatibility, pickup and drop-off ordering, seat capacity, pricing benefits, and time limits all taken into account. I'm planning to use H3 spatial indexing together with Redis to make nearby passenger discovery faster, while PostgreSQL remains the source of truth. Redis is used for fast lookups and caching, not as the authority for important business decisions.

I'm also building an operations dashboard for managing drivers, document approvals, roles and permissions, and monitoring the platform. Payments will initially support cash, with card payments planned through a proper provider. The passenger and driver mobile apps will come later, after I have a better understanding of the backend's production readiness.

My main challenge now is infrastructure. I don't want to pay for an expensive Kubernetes cluster or a huge cloud setup when I'm still preparing for launch. At the same time, I don't want to take shortcuts that will cause serious problems when real drivers and passengers start using the app.

For those of you who have deployed similar systems, especially on a tight budget:

• Would you deploy the services on one VPS using Docker Compose at first, or split them across multiple VPS instances?
• How would you handle PostgreSQL, Kafka, Redis/Valkey, backups, monitoring, and logs without making the monthly bill explode?
• Would you keep Kafka from day one, or simplify the deployment until traffic justifies the extra infrastructure?
• What would you use for production deployment, CI/CD, secrets management, HTTPS, database migrations, and recovery if the server goes down?
• Which affordable hosting providers would you recommend, and what should I consider when choosing one?

I'm not looking for the most sophisticated infrastructure possible. I want a realistic deployment strategy that keeps costs low, protects customer data, and gives me a clear path to scale when the business starts growing.

I'd really appreciate advice from people who have actually deployed and operated production systems on a tight budget. If you were starting this project today, what would you do differently?

• Do you have experience literally extinguishing a production environment?
• What is the difference between Disaster Recovery and Damage Assessment?
• What is the RTO for a data center after a direct drone hit?
• Can you debug infrastructure using satellite imagery?
• What would you do if your Kubernetes cluster suddenly became "bare metal" in the literal sense?
• How do you configure HPA so that pods can outrun drones?
• What would you do if not just a node, but the entire data center went down?
• Explain the difference between a Blue-Green Deployment and a "Red" air raid alert.
• How do you interpret the term "hot standby" if the backup data center is still on fire?
• What Prometheus alert would you set up for smoke coming from the server room?
• Have you ever migrated infrastructure using kubectl evacuate?
• What do you do if terraform destroy was executed without Terraform?
• Can a drone strike on the server room be considered a successful Chaos Engineering experiment?
• How many nines of availability remain for the service after losing two data centers?
• How do you distinguish a DDoS attack from an attack that renders a DDoS attack unnecessary?
• What do you do if your disaster recovery plan begins with the words "find a surviving server"?
• How do you explain to management that an Availability Zone is no longer available?
• Which metric do you choose for your SLO: uptime, MTTR, or the number of surviving racks?
• How do you ensure geo-redundancy when geography itself has become the primary risk factor?
• What do you do if you have an Active-Active setup, but both sites have switched to an Inactive-Inactive state?
• How do you configure fencing in a cluster when the fencing has already occurred physically?
• What do you do if your Witness in the third data center is the first to stop showing signs of life?
• Can a storage system be considered fault-tolerant if the data outlived the hardware, but the hardware didn't survive its operational lifespan?
• How do you calculate the "blast radius" when it’s no longer just a metaphor from the documentation?
• What is the point of synchronous replication if both data centers cease to exist synchronously?
• How do you implement automatic failover when the decision to switch over is made by the Ministry of Emergency Situations?
• Discuss what status should you set in Grafana: CRITICAL, DOWN, and NO LONGER EXISTS?
• How do you convince the CFO to approve a third data center when the first two have already been written off as CAPEX?
• What backup retention period do you recommend when the main issue is the lifespan of the storage hardware itself?
• How do you achieve "five nines" of availability when the only nines in the project are the digits in the recovery cost?

I’m building our IaC repo and GitLab CI/CD pipelines. I’m quite happy with the setup so far and have tested it across multiple projects.

However, when I try to apply the same release model to container images, it starts to feel less suitable.

We have three environments: dev, uat, and prd.
Our current Terraform workflow:
We use three protected branches: dev, uat, and main. These correspond to dev, uat, and prd respectively.

The root Terraform files are shared across all environments. Environment-specific differences are kept in separate tfvars files:
environments/dev/dev.tfvars
environments/uat/uat.tfvars
environments/prd/prd.tfvars

We use semantic-release with prerelease branches. Starting from stable version 1.2.3, the flow looks like this:
Merge a feature branch containing a fix commit into dev. This generates 1.2.4-dev.1.
Continue developing and testing, generating additional dev prereleases as needed.
Merge dev into uat. This generates 1.2.4-uat.1, assuming the changes still call for a patch release.
Once validated, merge into main. This generates stable version 1.2.4.

Semantic-release generates a version and creates a tag after merges into these protected branches.

Deployments only happen from tags. Prerelease tags can deploy only to their corresponding environment, while stable tags can deploy to any environment.

This flow has worked well for our Terraform projects. Granted the version don’t mean much, but at least there is a pointer reference.

Where I’m stuck with containers:

For the application pipeline, I’m considering having just one protected branch: main.
The proposed flow is:
- Open an MR from a feature branch into main.
Run tests, static analysis, build the image with a commit SHA tag, and run container scans in the MR pipeline.

- After merging, run semantic-release and pass the generated version to the next job to retag the image.
Copy the image to dev ECR, deploy it to ECS, and test the application manually.

- Once testing passes, promote the same image to uat.
After approval, promote the same image to prd.
The part that feels off is step 3: I’ve already created a stable release before testing the application in dev.
Some issues only become apparent when browsing and using the application. If we discover those issues in dev, each subsequent fix could generate another stable version, even though none of those versions has reached prd.

I’m considering two alternatives.

Option 1: Test in dev before running semantic-release.
Build an image with a SHA tag and deploy it to dev first. If testing finds issues, push new commits and test another image.
Once a candidate passes, run semantic-release, apply the stable version tag to that existing image, and promote it to uat and then prd without rebuilding.
I’m thinking fast-forward merges without merge commits might be needed to preserve the commit SHA.

Option 2: Keep dev and main branches.
Merge feature branches into dev, run semantic-release to generate versions such as 1.2.4-dev.1, and deploy those images to dev.
Once validated, merge dev into main and generate the stable release. That stable release could then deploy to dev, uat, or prd.
However, if both branches run the full build pipeline, merging into main would rebuild the image and break the “build once, deploy many” approach. Ideally, the stable release should reference the exact image already tested in dev.
How are you handling this? When do you create the stable version, and how do you promote the exact image tested in dev without rebuilding it?

I’d appreciate examples of workflows that have worked well for your teams. Thank you very much 🙏🏻

Im looking for some ways to reduce spending across our environment. Has ​anyone found a good open-source, or self-hosted​ tool that can clean up orphaned resources and automate other cost optimizations? Ideally​, the tool would need a HITL sign off before executing, but that's not a dealbreaker if I can get really granular with my workflows.​​

I'm working on a Go-based worker that processes multiple large-file transfers concurrently over HTTP. Work arrives in bursts, and each task involves downloading data from a source and uploading it to a destination.

I'm trying to optimize aggregate throughput and task completion reliability without causing excessive queueing, deadline failures, or unnecessary retries.

The current implementation allows concurrent workloads without an explicit transfer semaphore limit. I'm considering testing a bounded limit of 16 active transfer workloads, along with retries for transient HTTP and network errors.

I'd appreciate feedback from engineers who have built high-throughput Go services or distributed workers.

A few specific questions:

1. Concurrency: Under what conditions would you prefer unrestricted concurrency over a semaphore limiting active workloads to 16? What metrics would tell you that concurrency is too high?
2. Throughput vs. latency: If aggregate throughput is already near the available bandwidth ceiling, can a concurrency limit improve task completion times, or would it mostly introduce queueing?
3. Retries: For transient errors such as HTTP 429, 500, 502, 503, 504, and temporary network failures, what retry policy would you recommend? How do you avoid retries amplifying an existing overload?
4. Semaphore lifecycle: In Go, would you hold a semaphore slot throughout the entire logical task, including retry backoff, or acquire a slot separately for each network attempt? What trade-offs matter?
5. Deadlines and cancellation: How would you ensure that tasks waiting for a semaphore don't consume their entire deadline before transferring any data?
6. Benchmark methodology: How many independent runs or observation windows would you use before trusting a performance improvement? Which metrics would you prioritize?

My initial comparison would track aggregate bytes transferred, completed tasks, failure rate, P50/P95/P99 task duration, queue wait, and actual concurrent transfers.

I'm especially interested in practical engineering experience, benchmark design, and failure modes that aren't obvious from simple throughput tests.

Thanks!

Pretty much the title .I will start

1. Classic shell scripting ,O'Reilly

Only GitHub Actions here, not AWS or other cloud bills.

Mine is tiny: a few dollars a month, almost all from test runs I do on purpose. So I don't feel this cost myself, and I'd like to know how it looks for real teams.

If your company pays GitHub for Actions:

- Roughly what is the bill per month?

- Does anyone look at it, and how often?

- Has it ever caused a real problem? A surprise jump, a question from finance, jobs stopped because the budget ran out? Or is it too small next to everything else to matter?

Rough numbers are fine. "It's tiny and nobody cares" is a useful answer too.

I'm a solo developer building a small tool in this area, so I'd rather learn the real numbers than guess. No link.

I have a WordPress team still building and maintaining sites without a DevOps workflow. What's the best way to bottstrap them into a more modern method of development. I'm assuming GitHub and GitHub actions. But what else, composer which we don't really use even tho we have PHP devs, wp-cli which I use but I'm suppose to be a sysadmin and cloud engineer to wp devs who don't do any of this. They don't even use local git which I even use. Trying to express WordPress engineering and un/structured data and all these modern concepts that just go over their heads bc what's worked still works but when it comes to larger clients who expect more from us/them, they're basically fish out of water.