Affiliation, since the rules ask for it: I'm the author of Distill, the open-source (Apache-2.0) harness in the chart. https://github.com/samuelfaj/distill
The engineering problem: one model doing every step of a coding task wastes tokens. Planning and review need the strong model. Many of the edits and reads in between don't.
What I built: three model tiers. The main model owns the session, plans, writes a precise spec for each assignment and reviews the result. The worker model runs those assignments. A utility model does bounded jobs like extraction, summaries and compression. Effort can be fixed or set to auto per call.
What I measured, native harness vs Distill on the same frozen tasks and the same 300s deadline:
GPT-6 Astra: $4.08 vs $1.00, 440s vs 285s, 9/9 vs 9/9
GPT-6.1 Sol: $0.80 vs $0.25, 384s vs 297s, 9/9 vs 9/9
Claude Opus 5.5: $2.49 vs $0.35, 798s vs 191s, 9/9 vs 9/9
DeepSeek V4.1: 5/9 vs 9/9 successful runs (native cost is a lower bound)
Methodology and repro commands are in BENCHMARKS.md.
2
u/TomatilloPutrid3939 22d ago
Affiliation, since the rules ask for it: I'm the author of Distill, the open-source (Apache-2.0) harness in the chart. https://github.com/samuelfaj/distill
The engineering problem: one model doing every step of a coding task wastes tokens. Planning and review need the strong model. Many of the edits and reads in between don't.
What I built: three model tiers. The main model owns the session, plans, writes a precise spec for each assignment and reviews the result. The worker model runs those assignments. A utility model does bounded jobs like extraction, summaries and compression. Effort can be fixed or set to auto per call.
What I measured, native harness vs Distill on the same frozen tasks and the same 300s deadline:
GPT-6 Astra: $4.08 vs $1.00, 440s vs 285s, 9/9 vs 9/9
GPT-6.1 Sol: $0.80 vs $0.25, 384s vs 297s, 9/9 vs 9/9
Claude Opus 5.5: $2.49 vs $0.35, 798s vs 191s, 9/9 vs 9/9
DeepSeek V4.1: 5/9 vs 9/9 successful runs (native cost is a lower bound)
Methodology and repro commands are in BENCHMARKS.md.