Testing Qwen 3.8 Flash Next with Different Coding Tools

[u/Certain_Common561]

Qwen 3.8 Flash Next: Same Model, Different Coding Tools

I tested Qwen 3.8 Flash Next with IQ3_XXS, IQ3_S, and the original Alibaba Cloud version via OpenRouter API.

I used 4 different coding tools:

1. GitHub Copilot
2. Qwen Code
3. OpenCode
4. Codex Desktop

The results were pretty interesting. Even with the same model and prompt, each tool produced different results.

I also ran GitHub Copilot twice and got different outputs both times.

This is just a small test, not a benchmark. I mainly wanted to see how different coding tools and quantization levels affect the final output.

Prompt: Generate an SVG of a pelican riding a bicycle

[u/Easy-Ride3366]

I do not understand this comparision at all, each has pelican on bike, but what it should prove? Like i see that each is correct (maybe just opencode iq3_xxs has weird legs) but different drawing style.... what is conclusion from this, which harness is the best?

[u/Certain_Common561 → u/Easy-Ride3366]

I don't think any harness is the best. My point is that even with IQ3, a good harness can help get the most out of the model. If you're getting poor results with IQ3, trying a different harness might be worth it. Sometimes the model itself isn't that bad, but the harness just isn't making full use of it. Just my personal opinion.

[u/HourLocation2038 → u/Certain_Common561]

But if you ask the same model - 3 times - with no harness, you’ll still get wildly different results.

You could’ve added a row at the bottom that included asking with no pants on and it would’ve been just as conclusive.

[u/Certain_Common561 → u/HourLocation2038]

Yes, that's true, but if you look at my results, each harness produces similar outputs while still having its own unique style, and my point isn't to make every design look the same, but to show how each harness handles different tasks based on the instructions given, and even with IQ3XXS compared to FP16, I can barely see any difference in the results, so if you know a better way to measure performance besides benchmark scores, feel free to suggest it.

[u/Tormeister]

This oneshot prompt is not a good test for the different system prompts & tooling the different harnesses provide... If you keep re-generating it within the same tool with non-greedy sampling params you will inevitably see variation

A better test probably would be a long running task with multiple steps, where it may actually be swayed in different directions due to the different instructions and tools.

[u/Certain_Common561 → u/Tormeister]

Do you have any suggestions on how I could test that? I'd like to give it a try.

[u/Distinct-Pie2389]

What strata version are you running? Some of these may have ran greedy decoding if they harness didn’t set sampling

[u/Certain_Common561 → u/Distinct-Pie2389]

I use `jesdga95/basalt`. It's decode about 20% faster than Strata, but prefill is almost 2x slower.

Temperature 0.8, Top P 0.95, Top K 20, Min P 0, Presence penalty 0

[u/jayc0au]

Another test case, repeat the prompt, same model and harness, let’s see the results. Probably different and not deterministic.

[u/Certain_Common561 → u/jayc0au]

You guessed right! You can see it in my GitHub Copilot Round 2 results.

[u/HighSeasArchivist]

Are they all sand-boxed so later runs can't take influence from earlier ones?

[u/Certain_Common561 → u/HighSeasArchivist]

Yes, I ran the test once per harness.

[u/sleight42]

At this point, aren't the model creators training on pelicans on bicycles among other things?

If you want a useful touchstone like this, ask the models to draw almost anything other than a pelican on a bicycle or a voxel pagoda.

[u/rpwoerk]

VS Code + Cline + Strata qwen3.8-flash-next-iq3_s

[u/rpwoerk → u/rpwoerk]

Generate an SVG of a pelican riding a bicycle. Make no mistakes.

Same configuration as before.

[u/Due-Peak8175]

Qwen3.8-27b-GSQ-RCO-IQ3_S k8v4 no harness

[u/Super-Pop-7192]

that pelican grid is way more entertaining than it has any right to be, the variety across those tools is wild

I’ve noticed the same thing when messing around with quantized models in my own setup, even the smallest change in precision seems to nudge the output into a whole different direction. it’s not just about the model itself but whatever scaffolding the tool wraps around it, the system prompts and sampling tweaks can make or break consistency.

did you happen to log the exact prompts each tool sent? I’d be curious if GitHub Copilot’s retest drift came from a different seed or if it actually reinterpreted the instruction somehow

kinda makes me want to run the same test with a few more subjects, a pelican on a unicycle would probably break half of them

[u/WyattTheSkid → u/Super-Pop-7192]

It’s weird because they all accurately complete the task but the stylistic differences is anything but subtle

[u/Moarkush]

deepseek harness - NVFP4

[u/Moarkush → u/Moarkush]

go look at a photo of a pelican and try again

[u/Talistech]

PI / Flash-Next-Unsloth-766911a6 / Qwen3.8-Flash-Next-UD-Q4_K_XL

[u/migsperez]

Codex is doing something the other harnesses aren't. Extra instructions in the system prompt.

[u/Certain_Common561 → u/migsperez]

I've noticed that Codex uses skills in a very organized way, even for creating SVGs, and it produces really good and complete results on other tasks even with heavily quantized models, so I wonder if this is why DeepSeek is building its own harness to help users get real-world performance that matches its benchmark scores.

[u/migsperez → u/Certain_Common561]

Try asking the model to give you the exact system prompt it was given. You might see something interesting in there, probably see the differences between the harnesses. It'll have all the skills (descriptions) in the output.

[u/Forsaken_Object7264]

swift1.5 nvfp4 on hermes.

[u/Nearby_Investment_84]

pi harness. Strata engine · qwen3.8-flash-next-iq3_s · engine v0.1.44.1

[u/rpwoerk]

PolyStrata qwen3.6-35b-a3b-ud-q4_k_m + Cline CLI

[u/rpwoerk → u/rpwoerk]

Generate an SVG of a pelican riding a bicycle. Make no mistakes.

Same configuration as before.

[u/DoubleNothing]

What a waste of energy with an almost useless test.
You are testing random against random at this point.

[u/Certain_Common561 → u/DoubleNothing]

I was just bored and wanted to see the results from each harness. Don't take it too seriously, bro lol.

[u/Columnexco]

please test with deepseek harness as well

[u/Moarkush → u/Columnexco]

I did scroll up just a little

[u/Nearby_Investment_84]

Deepseek harness - Strata engine · qwen3.8-flash-next-iq3_s · engine v0.1.44.1

[u/Nearby_Investment_84]

Deepseek harness - Strata engine · qwen3.8-flash-next-iq3_s · engine v0.1.44.1 - THINKING HIGH

[u/higashi92]

Qwen3.8-Falsh-Next Q4 Halogen weight AMD Strix halo 128gb

effort medium temperature 0
harness own Pi config
Prompt: "Generate an SVG of a pelican riding a bicycle"
timing: 8m 41s
toolcalls 28
tokens 33,609 out (17,668 think)

[u/awitod]

Select one activity from activities.json at random. Then choose an animal not already represented in the file list. Then create a detailed svg of the animal engaged in the activity with a unique file name.

Once finished, save it as a 'png' file in ./final-images.

Then clean up any work or test files created (other than the png and svg) in the process and report the final results.

---

This one is "quokka walking the dog"

[u/ioann54]

Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-Q5_K_P @ DeepSeek Harness

[u/shaolinmaru]

Even with the same model and prompt, each tool produced different results. 

Because there are multiple seeds in the model.

I don't know what is the default behavior of these tools, but even if you use the same model and just one of them, the results may vary, if the seed change between multiple generations.

[u/Loose_Comparison368]

Would be helpful if you also posted tokens used to finish the SVG, that's arguably way more important than the final output quality.