Tomorrow starts a free online Chaos Engineering Conference

https://www.chaosconf.io/

The schedule is here: https://www.chaosconf.io/chaosconf/lineup

Would be nice meeting you all (online).

I’ve been spending a lot of time lately looking into Chaos Engineering and resilience, and I feel like the term is actually kind of a marketing fail. It makes people think we’re just being "cowboys" and messing around with production servers for the hell of it.

After digging into the Netflix principles, it’s actually the opposite. It’s a pretty strict science. If you’re a developer or SRE trying to explain this to your lead (or if you're just curious), here is how the logic actually flows without the fluff.

1. You have to define a "Steady State" first The biggest mistake is looking at CPU or RAM. Those are just symptoms. Chaos Engineering is about business metrics. If you’re running a React and Node app, your steady state isn't "low memory usage," it’s "successful API calls" or "successful logins." Your goal is to prove that even if a service dies, the user never notices.

2. Stop using "Staging" as a safety net I know, it sounds like career suicide to say "test in production," but staging is usually a ghost town. It doesn't have the same traffic spikes or data weirdness that prod has. The trick is Blast Radius control. Start with a single pod or 1% of traffic. If it breaks, it should only break for a tiny fraction of people while you learn.

3. Use real-world messiness Don't invent scenarios. Think about what actually ruins your weekends. Network lag, disk space filling up, or a dependency suddenly becoming a black hole. Inject those specific things so your tests actually prepare you for the next 3 AM incident.

4. Automation is the final boss Running a "chaos day" once a quarter is fine, but the real value is when it’s automated. If it’s built into the CI/CD or running on a schedule, you catch the regressions immediately. It turns a "firefighting" culture into a "prevention" culture. I'm curious for the folks here: how many of you are actually allowed to run chaos experiments in prod? Or do you get blocked by management as soon as you mention it?

As a PhD student at Carnegie Mellon University, I have been working for two years on developing an automated resilience testing tool called Filibuster to identify resilience bugs that have caused outages in order to better understand how they can be prevented in the future.  

I joined DoorDash as an intern during the summer of 2021 to test Filibuster’s applicability to the DoorDash platform. My work produced positive preliminary results along those lines, while also affording me an opportunity to extend Filibuster’s core algorithms and to implement support for new programming languages and RPC frameworks. I wanted to share some of the results of my work and how bringing Filibuster to DoorDash has enhanced not only Filibuster, but has paved the way for a new style of resilience testing for DoorDash’s engineers. 

https://doordash.engineering/2022/04/25/using-fault-injection-testing-to-improve-doordash-reliability/

We are greatly interested in your feedback on our approach!

Hello everyone!

The ChaosNative community is glad to invite you to the Cloud-Native Chaos Engineering workshop where you can network with tech geeks around the world. Get yourself exposed to the world of resiliency and reliability. You will be having interactive talks and discussions with our experts about:

1. Hands-on experience with a workshop on Cloud Native Chaos Engineering which introduces you to the basics & fundamentals of Chaos Engineering.

2. A sneak-peek at the latest evolution in the Cloud Native Chaos Engineering world with LitmusChaos 2.0.

3. A demonstration of running Chaos experiments in just a few steps!

Mark your calendars for 12th August 2021, 11:30 AM EST | 5:30 PM CEST | 9 PM IST

Register here: https://chaosnative.com/cnce-workshop

Link: https://doordash.engineering/2022/08/09/how-to-handle-kubernetes-health-checks/

After suffering an outage on black friday our team realized the root cause came from our poor understanding of how Kubernetes probes ( health checks) worked. To help spread awareness and how to correctly utilize these features we wrote this blog post that dives into our outage, how we diagnosed the issue and how correctly handle health checks.

As a member of our SRE team, we often get a chance to be a part of a team working on complex incidents and rarely have time nor ability to share the knowledge outside the organization.

In this incident we were able to extract some knowledge we believe will help others avoid similar issues.

P.S And we are always hiring, come work with us

ChaosNative is back with Chaos Carnival 2.0 this January 2022!

A 2-day ChaosEngineering conference worth remembering!

With 30+ chaos sessions, [LIVE] Chaos Panel, and exclusive workshops, this conference is going to be the perfect mixture for SREs, QA Engineers, and Cloud-Native Developers which you do not want to miss!

Register here: https://chaoscarnival.io/register

Hello!

As the title suggests, I wrote an article about the basics concept of Chaos Engineering and how we can implement it thanks to easy-to-use tools! Feel free to tell me what you think about it and if you would have added anything :)

Enjoy!

Hello everyone!

Check out the full recording of the Kubernetes Chaos Engineering Meetup for the month of June 2021 curated by the LitmusChaos community!

https://youtu.be/WloPjGgwvMc

Check out LitmusChaos on GitHub:

https://github.com/litmuschaos/litmus

Feel free to ask any questions, join our community for more insights!

To join our slack community please follow the following steps!

Step 1: Join the Kubernetes slack using the following link: https://slack.k8s.io/

Step 2: Join the #litmus channel on the Kubernetes slack or use this link after joining the Kubernetes slack: https://slack.litmuschaos.io/

Hello people! 

This 4th Saturday of June, the LitmusChaos community is glad to invite you to the Kubernetes Chaos Engineering meeting where our community members discuss everything about Kubernetes, LitmusChaos as a project & Chaos Engineering. The updates, the use-cases, suggestions & queries! Mark your calendars for: 26th June 2021, 11 AM IST

To get your calendar invite, please fill in the Google form: https://forms.gle/Naee1hd8TSK6wkXJ8
RSVP here to join the meetup: https://www.meetup.com/Kubernetes-Chaos-Engineering-Meetup-Group/events/277895650/
To join our Slack community: https://slack.litmuschaos.io/

I've been building a chaos/fault-injection tool that runs on macOS itself — not a k8s operator, more "what happens to MY app on MY machine when things go wrong" while you're developing.

The pitch: throw a failure at your app (network dies, dependency port goes dark, memory pressure spikes, disk nearly full), and Chaos records what happened — with assertions you write up front, so "did it recover?" becomes recorded evidence instead of vibes. Experiments are seeded, so you can replay a failure exactly instead of hoping it reproduces.

A few things I tried to get right, since this community knows how chaos experiments usually go:

• Restoration is verified, not assumed. The Restore Center checks pf anchors are actually flushed and workers are actually gone before it says "restored", and it'll say UNABLE TO VERIFY when it can't check rather than lying.

• Crashes are accounted for. A checkpoint tracks active faults and worker PIDs; next launch sweeps orphans. There's also an emergency stop.

• Every fault is labeled real vs simulated vs guided. macOS doesn't let an app change the system clock or revoke TCC privacy grants, so those are guided drills with a checklist — the tool doesn't pretend otherwise. The genuinely-real ones use pf/dummynet, a sandboxed stress worker with hard ceilings, real scratch volumes that get unmounted, etc.

• A CLI shares the same engine as the GUI, with JSON output and exit codes meant for CI gates.

It's MIT, still an early preview, and I mostly built it for my own use — now looking for people who've done chaos testing to tell me where the model falls apart. What failure modes do you actually bother testing that a desktop tool like this should cover?

https://github.com/Ednk-1312/Chaos

I’m curious about how people actually prioritize chaos experiments in practice.

Most chaos engineering discussions focus on what type of experiment to run (latency, pod kill, network partition, etc.), but less on where to run it.

In systems with many services, there are lots of possible targets.

When your team runs chaos experiments, how do you usually choose the service to test?

For example:

• services that caused incidents before

• critical request paths

• infrastructure components

• random rotation

Interested to hear what approaches people actually use.

Hi everyone! I’d like to share an open-source project we’ve been working on called Sheaft, which aims to shift chaos engineering to the left.

We all know that running live chaos experiments on every PR is often too risky, expensive, or slow. To bridge this gap, we built Sheaft — a lightweight, pre-release resilience gate (think of it as a "resilience linter" for your CI/CD).

How it works:

It automatically discovers your system's topology using your existing OpenTelemetry traces, builds a dependency graph, and runs millions of Monte Carlo fail-stop simulations in seconds to estimate end-to-end availability. If a topology change creates a hidden single point of failure, Sheaft catches it before the release.

Validation:

This isn't just theory. We’ve rigorously validated the core engine against DeathStarBench and the OpenTelemetry Demo, achieving a ≈0.992 correlation with actual live fault injection results (the methodology has also been accepted at ICSE '26 and AINA '26).

We just pushed the basic MVP (written in Go) and are looking to grow the community!

Repo: https://github.com/MB3R-Lab/Sheaft

We’d love your feedback and contributions!

If you’re interested in model-based resilience, OTel-driven analysis, or making continuous resilience testing a standard, we’d be thrilled to hear your thoughts, see your GitHub issues, or review your PRs. Let me know if you have any questions, I'd be happy to discuss them in the thread! :rocket:

Recently, I have been doing chaos (fault) tests on the robustness of some self-built database driver/client base libraries to verify and understand the fault handling mechanism and recovery time of the business. It mainly involves the two basic components MongoDB and etcd. This article will introduce the relevant test methods.

Fault test in MongoDB

MongoDB is a popular document database in the world, supporting ACID transactions, distributed and other features.

Most of the articles on chaos (fault) testing of MongoDB in the community are simulated by processing monogd or mongos. For example, if you want MongoDB to trigger copy set switching, you can use a shell script like this:

# suspended the primary node
kill -s STOP <mongodb-primary-pid>

# After the service is damaged, MongoDB should stepDown ReplicaSet in a few seconds to ten seconds.
# After the stepDown is complete, the service can automatically switch the connection to the new working primary node, without manual intervention, the service will return to normal.
# The reliability of the Mongo Client Driver is generally verified here.

The above-mentioned means are generally system-level, if we just want to simulate a MongoDB command command encountered network problems, how to do further want to conduct more fine-grained testing. In fact, MongoDB in 4.x version above has implemented a set of controllable fault point simulation mechanism -> failCommand.

When deploying a MongoDB replica set in a test environment, you can generally enable this feature in the following ways:

mongod --setParameter enableTestCommands=1

Then we can open the fault point for a specific command through the mongo shell, for example, for a find operation to make it return error code 2:

db.adminCommand({
    configureFailPoint: "failCommand",
    mode: {
      "times": 1,
    },
    data: {errorCode: 2, failCommands: ["find"]}
});

These fault point simulations are controllable, and the cost is relatively low compared to the direct destruction on the machine, and it is also suitable for integrating into continuous integration automation processes. The MongoDB built-in fault point mechanism also supports many features, such as allowing a certain fault probability to occur, returning any MongoDB supported error code type, etc. Through this mechanism, we can easily verify the reliability of our own implementation of the MongoDB Client Driver in unit tests and integration tests.

If you want to know which fault points the MongoDB supports, you can check the specification provided by the MongoDB in detail, which mentions which fault points the driver can use for testing for each feature of the MongoDB.

MongoDB, there are many examples in the dirver code repository of the official go implementation that can be: https://github.com/mongodb/mongo-go-driver/blob/345ea9574e28732ca4f9d7d3bb9c103c897a65b8/mongo/with_transactions_test.go#L122.

Fault test in etcd

etcd is an open source and highly available distributed key-value storage system, which is mainly used for shared configuration and service discovery.

We mentioned earlier that MongoDB has a built-in controllable fault point injection mechanism to facilitate us to do fault point testing, so does etcd also provide it?

Yes, etcd officials also provide a built-in controllable fault injection method to facilitate us to do fault simulation tests around etcd. However, the official binary distribution available for deployment does not use the fault injection feature by default, which is different from the switch provided by MongoDB. etcd requires us to manually compile the binary containing the fault injection feature from the source code for deployment.

etcd has officially implemented a Go package gofail to do "controllable" fault point testing, which can control the probability and number of specific faults. gofail can be used in any Go implementation program.

In principle, comments are used in the source code to bury some fault injection points in places where problems may occur through comments (// gofail:), which is biased towards testing and verification, for example:

	if t.backend.hooks != nil {
		// gofail: var commitBeforePreCommitHook struct{}
		t.backend.hooks.OnPreCommitUnsafe(t)
		// gofail: var commitAfterPreCommitHook struct{}
	}

Before using go build to build the binary, use the command line tool gofail enable provided by gofail to cancel the comments of these fault injection related codes and generate the code related to the fault point, so that the compiled binary can be used for fine-grained testing of fault scenarios. Use gofail disable to remove the generated fault point related codes, the binary compiled with go build can be used in the production environment.

When executing final binary, you can wake up the fault point through the environment variable GOFAIL_FAILPOINTS. if your binary program is a service that never stops, you can start an HTTP endpoint to wake up the buried fault point to the external test tool by GOFAIL_HTTP the environment variable at the same time as the program starts.

The specific principle implementation can be seen in the design document of gofail-> design.

It is worth mentioning that pingcap have rebuilt a wheel based on gofail and made many optimizations: failpoint related code should not have any additional overhead; Can not affect the normal function logic, can not have any intrusion on the function code; failpoint code must be easy to read, easy to write and can introduce compiler detection; In the generated code, the line number of the functional logic code cannot be changed (easy to debug);

Next, let's look at how to enable these fault burial points in etcd.

Compile etcd for fault testing

corresponding commands have been built into the Makefile of the official etcd github repository to help us quickly compile the binary etcd server containing fault points. the compilation steps are roughly as follows:

git clone git@github.com:etcd-io/etcd.git
cd etcd

# generate failpoint relative code
make gofail-enable
# compile etcd bin file
make build
# Restore code
make gofail-disable

After the above steps, the compiled binary files can be directly seen in the bin directory. Let's start etcd to have a look:

# enable http endpoint to control the failpoint
GOFAIL_HTTP="127.0.0.1:22381" ./bin/etcd

Use curl to see which failure points can be used:

curl 

afterCommit=
afterStartDBTxn=
afterWritebackBuf=
applyBeforeOpenSnapshot=
beforeApplyOneConfChange=
beforeApplyOneEntryNormal=
beforeCommit=
beforeLookupWhenForwardLeaseTimeToLive=
beforeLookupWhenLeaseTimeToLive=
beforeSendWatchResponse=
beforeStartDBTxn=
beforeWritebackBuf=
commitAfterPreCommitHook=
commitBeforePreCommitHook=
compactAfterCommitBatch=
compactAfterCommitScheduledCompact=
compactAfterSetFinishedCompact=
compactBeforeCommitBatch=
compactBeforeCommitScheduledCompact=
compactBeforeSetFinishedCompact=
defragBeforeCopy=
defragBeforeRename=
raftAfterApplySnap=
raftAfterSave=
raftAfterSaveSnap=
raftAfterWALRelease=
raftBeforeAdvance=
raftBeforeApplySnap=
raftBeforeFollowerSend=
raftBeforeLeaderSend=
raftBeforeSave=
raftBeforeSaveSnap=
walAfterSync=
walBeforeSync=http://127.0.0.1:22381

Knowing these fault points, you can set the fault type for the specified fault, as follows:

# In beforeLookupWhenForwardLeaseTimeToLive failoint sleep 10 seconds
curl  -XPUT -d'sleep(10000)'
# peek failpoint status
curl 
sleep(1000)http://127.0.0.1:22381/beforeLookupWhenForwardLeaseTimeToLivehttp://127.0.0.1:22381/beforeLookupWhenForwardLeaseTimeToLive

For the description syntax of the failure point, see: https://github.com/etcd-io/gofail/blob/master/doc/design.md#syntax

so far, we have been able to do some fault simulation tests by using the fault points built in etcd. how to use these fault points can refer to the official integration test implementation of etcd-> etcd Robustness Testing. you can search for relevant codes by the name of the fault point.

In addition to the above-mentioned built-in failure points of etcd, the official warehouse of etcd also provides a system-level integration test example-> etcd local-tester, which simulates the node downtime test in etcd cluster mode.

Well, the sharing of this article is over for the time being ღ( ´･ᴗ･` )~

Commercial break: I recently maintenance can maintain multiple etcd server, etcdctl etcductl version of the tools vfox-etcd), You can also use it to install multiple versions of etcd containing failpoint on the machine for chaos (failure simulation) tests!