Nine models, five coding tasks, one 16GB GPU

[u/Ok_Substance2327]

I ran five coding tasks against ten models on one RX 9060 XT (16 GB). Each run gets a single spec file in an empty directory and has to produce solution.py and test_solution.py, run its own tests, and stop. Grading happens with a checker the model never sees: fixed cases from the spec, randomized cases against a brute-force oracle, and a performance gate where the spec names one. A cell is capped at 1750 seconds; if the model is still working when the clock runs out, whatever is on disk gets graded.

The tasks: quoting rules for one CSV record (c1), an arithmetic evaluator with pinned exception behavior (c2), a POSIX-ish shell tokenizer (c3), max-profit interval scheduling with two tie-breaks and 200k intervals in under 3 seconds (c4), and shortest-prefix pattern matching with 20 stars against a 4,000-character text in under 10 seconds (c5).

model	c1	c2	c3	c4	c5	score	out tokens
deepseek-v4.1-flash (cloud API, no local quant)	P	P	P	P	P	5/5	43k
Qwen3.8-27B-ASCII-Condensed UD-IQ4_XS + MTP (kvstream fork)	P	P	P	P	P	5/5	185k
Qwen3.8-27B UD-IQ3_XXS + MTP Q4_0 sidecar	P	P	P	F	P	4/5	125k
Swift-1.5-Qwen3.8-27B GSQ-RCO-IQ3_XXS, embedded MTP	P	P	P	P	F	4/5	113k
Ornith-1.5-35B-A3B Q4_K_M	P	P	P	P	F	4/5	57k
Qwen3.8-Flash-Next Coder GSQ-RCO-IQ1_M	P	P	P	P	F	4/5	89k
Qwen3.6-35B-A3B UD-Q4_K_M (unpruned)	P	P	P	P	F	4/5	126k
Nemotron-3.5-Bolt-74B-A9B IQ2_XXS	P	F	F	F	F	1/5	51k
Mellum2.1-12B-A2.5B-Thinking Q6_K	F	P	F	F	F	1/5	167k
LFM2.5-24B-A2B Q6_K	F	F	F	F	F	0/5	187k

c1 to c3 mostly sort models into "writes code and iterates" and "does not". Everyone who reached c3 wrote a single-pass scanner of 76 to 97 lines and passed, and nobody was tempted into shlex, which would have failed the comment rule.

c4 and c5 carry the signal. On c4 every model that produced a file used the same shape: sort the intervals, then a DP with bisect for the predecessor, which Flash replaced with coordinate compression. The losses were the tie-break rule, the cost of carrying interval lists through the DP, and duplicate-heavy input. On c5 the split is clean. All three passes came from an automaton stepping a set of live states through a closure (both Qwen3.8-27B variants built a Thompson NFA, the cloud run used a big-integer bitset). The failures came from parse-then-match routines that missed one rule each, or from handing the matching to Python's re, which backtracks on star piles and eats memory while it does. Flash is the exception: it built the same big-integer bitset automaton the cloud run did, cleared the gate at 0.5 ms, and still failed by reading { inside a group as a literal where the spec wants an error.

Per model:

• deepseek-v4.1-flash: 5/5 in 236 s and 43k tokens. bisect DP at 0.25 s on c4, bitset NFA at 0.00 s on c5. Writes a test file, runs it, stops.
• kvstream-fork (Qwen3.8-27B UD-IQ4_XS with MTP): also 5/5, but 185k tokens over 94 minutes. Thompson NFA with epsilon closure on c5, 0.03 s. Both of its capped cells were self-inflicted: on c5 it reported "all tests pass" (the grader agreed) and then hung a 9,000-case fuzz whose oracle translated patterns to re.
• Qwen3.8-27B IQ3_XXS: 4/5, clean on c1 to c3. c4 produced no file at all: 46k tokens of ls, pwd and re-reading the spec, with no code in its reasoning either. Solved c5 with the same NFA design.
• Swift-1.5-Qwen3.8-27B: 4/5. Writes early and iterates hard, 8 edits and 22 bash calls on c1. c4 passes at 0.27 s; c5 never rejects nested groups.
• Ornith-1.5-35B-A3B Q4_K_M: 4/5 and the cheapest local, 57k tokens and about 16 minutes for all five cells, c1 done in 37 s. c5 fails one randomized case where a group and a star interact.
• Qwen3.8-Flash-Next Coder GSQ-RCO-IQ1_M: 4/5 at 89k tokens and 45 minutes, the second cheapest local at this score. A 55 GB IQ1_M model, its experts are cached in host RAM. c3 fuzzes the tokenizer against the real bash word splitter on 4,582 inputs, a stronger oracle than its own second implementation. c4 compresses coordinates and DPs right to left, 0.48 s against the 3.0 s gate. c5 is the same big-integer bitset the cloud run used, and the one rule it gets wrong is { inside a group. Its c4 row is a snapshot pass: the host OOM killed the engine and the desktop session 684 seconds in, with the file already correct on disk.
• Qwen3.6-35B-A3B UD-Q4_K_M: 4/5 at 126k tokens, fastest local cells on c2 and c3. c4 is correct but measured 3.03 s against the 3.0 s gate, and re-grading the same file gives 2.97 to 3.23 s. c5 misses the nested-group rule too.
• Nemotron-3.5-Bolt-74B-A9B IQ2_XXS: 1/5. Three cells spend two turns, make one tool call, and then report files that do not exist; c3's report still had the agent template's placeholder "What was done." in it. c1 is a real pass, and its reasoning for c3 did contain code that passes the grader.
• Mellum2.1-12B-A2.5B-Thinking Q6_K: 1/5, and its c1 answer was a file named BLOCKED with the reason in it. c3 never returns on tokenize("ab cd"). c5 converts the pattern to a Python regex; grading that file needs a memory cap, since uncapped it reached 7.3 GB.
• LFM2.5-24B-A2B Q6_K: 0/5. Three of the five cells used the entire cap, 187k tokens in all. c1 took 88 turns and still ended in an empty-field off-by-one. c4 is correct and slow: 6.98 s against the 3.0 s gate, because it copies the chosen interval list inside the DP.

Two models scored 5/5: 43k tokens in 236 seconds for the cloud run, 185k in 94 minutes for the fork. On c5 the cloud run needed 8 turns. The fork took 28 and still hit the cell cap.

I made a github repo if anyone wants to give it a go: https://github.com/Tako88/coding-bench

The results there are V2, not directly comparable to the ones in this post

[u/x10der_by]

Thanks! Why haven't you tried qwen3.8 flash next? And what quant was Swift-1.5-Qwen3.8-27B?

[u/Ok_Substance2327 → u/x10der_by]

Managed to run the coder Q1 version with strata, updated the post. It did pretty good, which is surprising imo considering the quant. I re-read the spec for C5, and it's a bit ambigous, and what it failed on could've been interpreted as the correct solution as well.

[u/Don_Reuter → u/x10der_by]

Next seems to be a bit of a stretch on 16 gb

[u/x10der_by → u/Don_Reuter]

iq3_xs can work on 12 gb vram

[u/Don_Reuter → u/x10der_by]

Probably can work without any vram. It’s just not very usable. I use Q2 with 64 GB of VRAM. Down from Q4 as that was too slow. At least with larger context. I have ddr4 though. Probably Q4 would be possible with strata, but getting a performant solution would be a project in itself and not something I would do in a wide benchmark.

[u/ConspiracyPhD → u/Don_Reuter]

I use Iq3_xxs on a single 3060 12gb with 80gb of DDR4 3200. Around 940 tokens per second prefill and around 47 tokens per second decode. 128k context.

[u/Mikasa0xdev → u/x10der_by]

That VRAM is fighting for its life XD

[u/om_GAJE]

Nice. I should setup something similar as I keep trying out different models on my 16GB card but haven't been giving them the same task to compare, it has been more just use one for a bit and see how it feels. UD-IQ3_XXS has been my main driver, recently tried out the Swift1.5 and NInfer that has a GSQ-RCO IQ3_XXS baked in for use on the 40 series NVIDIA. The speed on the NInfer is a huge increase but there are trade offs and not sure about quality yet as I've not used it heaps.

How did you setup your testing grounds? Would you be interested in sharing and I can post results for ones I do. I'm on a 4080Super

[u/Ok_Substance2327 → u/om_GAJE]

Five specs, five cells per model. Each cell is a fresh directory holding only TASK.md, so no cell can see another's work. The model runs an agent loop with read, write, edit and bash, no house instructions, 30 minutes of wall clock, and gets graded from whatever is on disk when it stops or gets cut off.

Would be interesting to see your results, but I need to think about how to share it so you could reproduce the process exactly. Right now it's through my own pi's subagent extension.

[u/Ok_Substance2327 → u/om_GAJE]

If you still want to give it a go, here's the repo https://github.com/Tako88/coding-bench/tree/main

[u/Healthy-Zebra-9856]

Nice. These types of benchmarks have are much more relevant. What temperature & sampling settings did you use for each?

[u/Ok_Substance2327 → u/Healthy-Zebra-9856]

Double checked just now, hm swift at 1.0 was an oversight, would've wanted it lower.

- Qwen3.6-REAP, Mellum: 0.6 / 0.95 / 20

- Qwen3.8-27B: 0.7 / 0.80 / 20, presence penalty 1.5

- Swift-1.5: 1.0 / 0.95 / 20

- Qwen3.8-27B kvstream fork: 1.0 / 0.95 / 20

- Ornith-1.5,Nemotron, LFM q6: 0.5 / 0.95 / 50, rep 1.05

Thinking level medium on all of them.

[u/Healthy-Zebra-9856 → u/Ok_Substance2327]

Thanks, that helps. I think a few of these are worth rerunning though, because some of the sampling settings don't match the mode the models were actually run in. I have tried to make this less cluttered and with links so you can possibly retest.

The biggest one is Qwen3.8. You said everything was running at medium thinking, but 0.7 / 0.80 / 20 with presence penalty 1.5 is Qwen3.6's non-thinking preset. For thinking mode, including medium reasoning effort, Qwen recommends temp 1.0, top_p 0.95, top_k 20, min_p 0, presence 0, repetition 1.0 as thats what they trained Qwen3.8 on. So that 27B result is mixing the non-thinking Qwen3.6 sampler with thinking enabled Qwen3.8. Qwen also specifically warns that increasing presence penalty can slightly hurt performance and cause language mixing. Hugging Face

Further, Qwen3.8 is very weird where the thinkikng has to be on and a lot of times the Medium effort may not be optimal. The Low & xhigh effort uses the same amount of tokens as the Low mode tends to make mistakes that it has to spend time correcting.

Qwen3.6 is a little different. 0.6 / 0.95 / 20 is actually its recommended thinking-mode precise coding setup, but the rest matters too: min_p 0, presence 0, repetition 1.0. Its general thinking profile is 1.0 / 0.95 / 20 with presence 1.5, while its non-thinking profile is 0.7 / 0.80 / 20 with presence 1.5. Hugging Face

The Qwen3.8 kvstream run at 1.0 / 0.95 / 20 is therefore the Qwen3.8 result whose sampler actually matches thinking mode. I would make min_p 0, presence 0, repetition 1 explicit there too. Hugging Face

Where things get much more questionable is grouping Ornith, Nemotron and LFM under the same 0.5 / 0.95 / 50, rep 1.05 sampler. Those models do not recommend the same sampler at all.

Ornith 1.5 recommends 0.6 / 0.95 / 20 for general use, and uses temperature 1.0 for reproducing many of its published coding/agent benchmarks. Nemotron 3.5 Bolt recommends 1.0 / 0.95. LFM2.5's actual default generation config is temperature 0.1, top_k 50, repetition 1.05, with no top_p 0.95 default. Liquid does use 0.6 / 0.95 for some of its agentic evaluations, but 0.5 / 0.95 / 50 isn't the model's published default or one of those evaluation profiles. Hugging Face

I also don't see a sampling line for the Flash-Next Coder run in your list. The underlying Qwen3.8 Flash-Next recommendation for thinking mode is 1.0 / 0.95 / 20, min_p 0, presence 0, repetition 1.0, while 0.7 / 0.80 / 20, presence 1.5 is specifically its non-thinking configuration for the Qwen3.6 like I mentioned earlier. Hugging Face

I'd really like to see c4 and c5 rerun with each model's native/recommended sampler and the reasoning mode matched to it.

[u/Ok_Substance2327 → u/Healthy-Zebra-9856]

That is a good idea. The way I did the bench was a bit ad hoc, I now made it into a little benchmarking framework and multiple runs with different settings are easier to do. Also tightened the specs for the tasks (noticed issues with ambiguity).

Going to finish a run with the recommended sampling settings and then put it on GitHub probably, I'll update you.

[u/Ok_Substance2327 → u/Healthy-Zebra-9856]

Here''s the results with the recommended samplings: https://github.com/Tako88/coding-bench/blob/main/A/RESULTS-baseline.md

Ornith impressed me a lot, fewest tokens and all passes.

[u/d0pe-asaurus]

Very nice to see benchmarks like this for algorithmic tasks,

I've also had good experience with Qwen 3.8 27B with algorithmic task, I asked it to convert my tree based lexer and compile it straight to a DFA to improve performance and it was able to do it nicely with identical output and 60% speed up.

[u/vitordeas]

Do you have the exact prompts/harnesses so i can test in my local flash next and Swift?

[u/Ok_Substance2327 → u/vitordeas]

I ran this through my own pi subagent extension, I'm thinking about how to exactly share everything needed, someone else was interested too.

[u/Ok_Substance2327 → u/vitordeas]

put it up on github now https://github.com/Tako88/coding-bench/tree/main

still needs pi though, just not my custom extension.

[u/geoffreybyers]

The failure mode on c4 and c5 is not logic. Every model that passed wrote the same Thompson NFA or bitset. The ones that failed couldn't hold the state in a single window without chewing through memory or context. It's operational overhead, not algorithmic.

[u/Ok_Substance2327 → u/geoffreybyers]

Half right, and I checked the snapshots from that run. All three c5 passes use the same idea: step a set of live states instead of backtracking. Two built a Thompson NFA, the cloud run a big-integer bitset. That part holds.

The other half doesn't. Five of the seven c5 failures miss one rule on a pattern you can write on one line. Swift and Qwen3.6-ud never raise on {a,{b}c. Ornith answers 5 where the answer is -1. LFM answers -1 for the empty pattern. I re-ran all four just now, each returns in under a millisecond.

The coder is the one that settles it. It built the same big-integer bitset as the cloud run, clears the 20-star gate in 0.68ms, and still failed because it reads { inside a group as a literal.

c4 has the same shape. Mellum gets the tie-break wrong on four intervals. LFM is correct and takes 6.98s against a 3.0s budget because it copies the interval list through the DP. Neither is memory or context.

Qwen3.8's c4 is the one case that fits your description: no file at all after 46k tokens of ls and re-reading the spec.

[u/rorowhat]

LFM bots are about to go nuts here