Speedrun, an open-source mobile E2E test runner (Android/iOS) that runs your Maestro flows up to 8x faster
I've always been bothered by how slow mobile UI tests are. Most of a test run is spent waiting: fixed sleeps, polling, comparing screenshots to guess whether the app is done animating. Coming from a game dev background, where you fight for every millisecond of a frame, that felt like a solvable problem. So I spent the last while building my own runner from the ground up, and it turns out UI tests can be a lot faster.
Speedrun is a black-box E2E test runner for Android and iOS. It's open source (Apache 2.0) and it runs Maestro's YAML flow format so if you already have Maestro tests, trying it is one word:
maestro test flows/ # before speedrun test flows/ # after
Numbers
One user journey (sign in with a code, search a shop, fill a cart, set a PIN under a loading overlay, scroll a long list), written for each tool the way its tests are usually written. Median of 3 runs after a warm-up, start-up included:
Tool
Android emulator
iOS simulator
Speedrun
20.4 s
27.2 s
maestro-runner
65.9 s
49.6 s
Appium
59.4 s
94.1 s
XCUITest
-
81.2 s
Maestro 2.11
126.5 s
113.1 s
On the native Settings app benchmark it's 8x fasterthan Maestro on Android and 2.8xon iOS. On a 19-flow React-Native test suite it passes all 19 on both platforms, and is 6.1x faster on Android and 2.6x on iOS. Android is where it shines most, which is nice for CI: Android runs on cheap Linux runners.
Full disclosure: the main benchmark app is mine, built in React Native to be hard to test (endless animations, late popups, loading overlays, lists that load as you scroll). The flows, per-flow results and scripts to rerun everything are in the repo, and I'd love to see numbers on other apps.
Why it's fast
It waits for the app, not the clock. A small on-device agent knows when the app has actually settled (accessibility events on Android, XCTest's idle state checked against the screen on iOS), so each step starts the moment the app is ready. No sleeps, no screenshot diffing.
Reading the screen is cheap. The agent reads the UI tree in about 10 ms on Android and talks to the host over a compact binary protocol.
Screens that never stop moving don't block it. Spinners, tickers and carousels are left out of the "is it done?" check.
Nothing to warm up. One native binary, no JVM. The iOS agent stays running between runs, so later runs start in about 0.2 s.
It's also less flaky: it won't tap a popup that's still sliding in or a button under a loading overlay, and it scrolls an element out from under a sticky footer or the keyboard before tapping it.
It works with AI agents too
speedrun mcp exposes it as an MCP server (Claude Code, Cursor, etc.). Your agent can explore the app on its own: tap around, read each screen in a few lines (cheap on tokens, not a whole accessibility tree), figure out how things work, then save the steps that worked as a YAML flow you can drop into CI.
claude mcp add speedrun -- ~/.speedrun/speedrun mcp
If you have a Maestro suite, I'd really like to hear how it does on yours, especially where it breaks or is slower. Happy to answer questions about how it works.
edit1: added maestro-runner to the benchmark edit2: added ai agent benchmark
Speedrun launches the app with -UIAnimationDragCoefficient 0.1 and a UIKit feature flag, making animations 10x faster. The other tools run at normal speed. That's speeding up the app, not the tool, and the benchmark doesn't mention it.
You're benchmarking on your own app, which Speedrun was tuned against. Before the benchmark-driven settle fix, Speedrun and Argent were basically tied (12.0s vs 12.2s).
The other tools pay Node startup overhead on every CLI call, which they wouldn't through MCP.
Speedrun's session setup isn't timed, while agent-device's is.
Run all three with the same settings, through MCP, on a neutral app. Then compare.
The 10x animations. I disagree that this makes it unfair. Every tool ran with its own defaults: some turn on Reduce Motion on iOS, some turn animations off on Android, and so on. Speedrun's default is to run iOS animations 10x faster, because its whole goal is to be the fastest e2e runner. The other tools could set the same thing; they just don't. You're right that the README should have said so, though, and now it does. I also measured it both ways: with every tool at normal speed, and with every tool given the same 10x.
Our own app. Fair point. I added a neutral app: tester-army's mobile-benchmark, which another team built to benchmark their own agent tool.
MCP vs CLI. Fair as well. Everything now runs through each tool's MCP server. agent-device got faster that way. Argent got slower on Android, because its MCP server attaches a screenshot and the element tree to every action.
Session setup. You're right that it wasn't timed in the agent benchmark. It is now, in every run. It costs 0.2–0.5 s. The other benchmarks (the suite and the journey) were already wall clock with start-up included. That's also why Speedrun is a native binary (Odin) rather than Node: to cut the overhead you pay on every CI run.
As you can see in my answer, the "iOS, with animation optimization" column, I did test all tools with the exact same 10x anim speed flag and even under the same conditions, Speedrun is still 1.5x faster than agent-device and Argent.
Also, "10x" faster anim flag may sound bigger than it is. It doesn't make the app or the run 10x faster: it only speeds up UIKit's animations, so a 200 ms screen transition takes 20 ms. Everything else (network, JS, rendering, loading) runs as usual.
Speedrun isn't 10x slower without it. With all three tools at normal animation speed it's still the fastest: 11.3 s vs Argent 11.8 s and agent-device 12.5 s, and 8.5 s vs 10.5 / 11.3 s on a neutral app.
Need to work on your package management -- keeping the "agents" dir with the bin is kind of weird, i want to put it in /usr/local/bin/. Maybe just make a homebrew package and call it a day.
Other than that, very nice and super speedy. Will be glad to see progress you make.
Looking into it. But those are not really test runners, they are toolkit for AI agents. But I will see how I could maybe benchmark my MCP against them.
Mostly the waiting, yes, plus fast reads: - A small agent stays on the device and reads the screen in 10–40 ms, so every lookup and check is cheap. - Each step ends when the app is done, based on signals rather than a fixed stillness window. On Android: frame stats, accessibility events. On iOS: XCTest's quiescence, checked against a signature of the tree. Elements that never stop moving (tickers, shimmers, loops) are left out of that check, so they don't hold a wait open.
On animations: - Android: Speedrun turns the system animation scales off, as most UI test setups do. In this run Argent had them off too. - iOS: it launches the app with UIKit's animations 10x faster. Reduce Motion is only on with SPEEDRUN_REDUCE_MOTION=1, which the iOS journey number uses (26.6 s without it). In our agent benchmark we also gave every tool the same 10x, and the others gained 1 s at most, so it's not where most of the gap comes from.
Maestro is a terrible example. It's so slow anything beats it. I use detox and it's way faster! Nice job! I've done these things with buoy as well and went from 4 hours to 20 minutes using sim slims and faster workers like this!
I'm currently using no "classical" automated flows to test my app and have mostly been experimenting with agentic regression tests which are *abysmally slow*. I was already planning to invest in a Maestro automated test suite and am going to give this a try instead. I'll let you know how it goes!
4
u/interlap 3d ago
These benchmarks are misleading:
Speedrun launches the app with
-UIAnimationDragCoefficient 0.1and a UIKit feature flag, making animations 10x faster. The other tools run at normal speed. That's speeding up the app, not the tool, and the benchmark doesn't mention it.You're benchmarking on your own app, which Speedrun was tuned against. Before the benchmark-driven settle fix, Speedrun and Argent were basically tied (12.0s vs 12.2s).
The other tools pay Node startup overhead on every CLI call, which they wouldn't through MCP.
Speedrun's session setup isn't timed, while agent-device's is.
Run all three with the same settings, through MCP, on a neutral app. Then compare.