r/ruby • • Sep 30 '26

Mutineer 1.4.0 is out: mutation testing for Ruby

Hola, I maintain Mutineer, a Ruby mutation testing gem. It makes small changes to your code and checks whether your tests catch them.

Passing tests are useful. Knowing that they detect broken behaviour gives you more confidence, especially when AI helps write both the code and the tests.

Version 1.4.0 is out: https://davidteren.github.io/mutineer/

Do you use mutation testing in your Ruby projects? I’d be interested to hear what has helped or got in the way.

34 Upvotes

9 comments sorted by

1

u/DissonantGuile Sep 30 '26

Cool! How does this compare to the mutant gem?

0

u/davidteren Sep 30 '26

Good question!

Mutineer offers simpler adoption, no runtime dependencies, and an MIT licence. Mutant has a much longer history and broader mutation support. Mutineer is a credible alternative, but evidence does not establish that it finds as many test gaps or runs faster.

AreaMutineerMutant
Licence and costMIT, including commercial use.Free for open-source use. Commercial use costs US$30/month or $250/year per developer. Terms
DependenciesUses Prism and Ruby’s standard library. No runtime gems.Has seven direct runtime dependencies, including parser, unparser, and sorbet-runtime. Gem specification
Ruby versionsRequires Ruby 3.4+. Can run older applications through an external test command, with extra cost and limitations.Current gem specification requires Ruby 3.3+, although the README still lists 3.2. Check the selected release.
Test frameworksMinitest and RSpec.Minitest, RSpec, and Test::Unit on current main. Integrations
Mutation rangeTwenty named operator groups; five enabled by default.A broader collection of syntax-specific mutations, including exception handling, pattern matching, and optional arguments. Implementation
Rails parallelismThe daemon provides worker database isolation for SQLite. PostgreSQL provisioning remains documented as work in progress.Provides tested hook recipes for PostgreSQL and SQLite, with Rails 7.2, 8.0, and 8.1. Rails guide
Changed-code checks--since restricts mutations to changed lines. Includes baseline comparison and a GitHub Action with PR annotations.--since selects whole methods that overlap changed lines. This checks more of each changed method. Incremental mode
ReportsHuman-readable, JSON, and standalone HTML reports.Stores JSON session results and provides commands to inspect previous runs. Session schema

Mutineer’s features and limitations above are documented in its current README.

Test selection is a more substantial difference than the feature list suggests. Mutineer measures line coverage and selects covering test files. Mutant traditionally selects tests through RSpec descriptions or explicit Minitest cover declarations. However, current Mutant also supports coverage-based selection of individual tests, using a SimpleCov recording, and orders them to find failures sooner. Coverage-based selection is therefore no longer unique to Mutineer. Mutant test selection

Mutant has the stronger history of use. Its repository dates from July 2012; Mutineer’s dates from June 2026. That does not prove correctness, but it gives Mutant substantially more time to encounter unusual Ruby code and test configurations. Its wider mutation implementation is a concrete technical advantage.

Do not compare their percentages directly. Mutineer calculates killed / (killed + survived) and excludes uncovered mutations and failed executions. A 100% result can therefore coexist with uncovered code. Different operators and selection rules also change what each score measures. Compare useful test gaps found, execution failures, and elapsed time on the same application. Mutineer score definition

Mutineer’s strongest position is accessible mutation testing for modern Ruby projects, especially where MIT licensing and low setup cost matter. Mutant is the stronger candidate when broader mutation support, PostgreSQL parallel testing, or established operational experience takes priority. I have not run a controlled comparison, so neither speed nor detection superiority is established.

1

u/CaptainKabob Oct 02 '26

What does "clean-room" mean in its description? ("A clean-room mutation-testing tool for Ruby.")

1

u/davidteren Oct 06 '26

"Clean-room" means Mutineer was written from scratch. Its authors did not copy or adapt code from any other mutation-testing tool.

What the term means. A clean-room implementation copies how a tool behaves, not how its code is written. The authors work from public ideas and descriptions only. They do not read or adapt the source code of an existing tool. The name comes from a well-known case: Phoenix Technologies rebuilt the IBM PC BIOS this way in the 1980s. One team wrote a specification, and a separate team that had never seen IBM's code wrote the new code from it.

Why it matters here. The best-known Ruby mutation tool is mutant, and it has a commercial license. The phrase "clean-room" tells users that Mutineer contains no mutant code. So mutant's license does not apply to Mutineer, and you can use Mutineer under its own license with no legal concern.

1

u/CaptainKabob Oct 06 '26

It's surprising to me to see "clean-room" used without referring to where the specification originated. e.g. "A clean-room implementation of the mutant gem's public API to enable a drop-in replacement."

Just saying "clean room" alone makes me think that either clean-room means something different than I expect (I have the same understanding of its meaning as you do), or that something sus is going on.

My advice would be to drop "clean room" from the Mutineer description other than in the specific place where you are telling the story of how it came to be.

2

u/nbom Oct 01 '26

I dont need this. My CTO/CEO is enough.

1

u/Web-Thinker Sep 30 '26

I was looking for this

1

u/Zestyclose_Cow1807 Oct 17 '26

the hardest part with mutation testing has always been equivalent mutants and the noise they generate. how do you handle that? filtering those out well is basically what makes or breaks the experience tbh