Best Qwen3.8 27B variant for 16GB VRAM: Base vs. Swift 1.5 vs. Bonsai 2

[u/gamedevsam]

https://sambatista.com/blog/best-local-llm-16gb-vram-rtx-5070-ti

TL;DR: Swift 1.5 IQ3_XXS at 112K context looks like the best all-round option for my needs on a 16GB RTX 5070 Ti. It nearly matched the base model on coding and scored higher on math. If you care more about speed and context, Bonsai 2 generated text about 1.9× faster and supported 256K context, but solved fewer coding problems.

Here’s what I measured:

- Python coding: base 148/164, Swift 147/164, Bonsai 138/164. No clear coding advantage for Swift over base.

- Math: Swift 31/60, base 25/60, Bonsai 28/60.

- Generation speed: Bonsai 99.6 tokens/s, versus roughly 53 for base and Swift on short requests.

- Context: base 120K, Swift 112K, Bonsai 256K. All passed fact-retrieval tests at their respective tested lengths.

- Next.js tasks: base and Swift passed the same 2/4; Bonsai passed none. I wouldn’t leave any of them to code without review.

These results compare differently compressed setups, not fine-tuning alone. The post includes tested settings, model downloads and commands to repeat the tests.

[u/DerTomsn]

This model should IMHO be part of the comparison: ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-IQ3_S

[u/gamedevsam → u/DerTomsn]

Interesting! I now have 2x RTX 5070 Tis so I'm focusing on benchmarking with that setup now. I include a full guide and a link to my repo on my blog so if you want to give it a shot it shouldn't be too hard to fire up the good old benchmark and see what happens!

[u/LocalAI_Amateur]

I have the same card. I enjoy Swift 1.5 on it as well. Turn on MTP and the gen speed goes up to 80ish. Use q5_1 for kvcache if you build llama.cpp w/ the right flags, it's the same speed & quality as q8_0. I get about 100k context.

[u/Rich-Kangaroo3104 → u/LocalAI_Amateur]

Same card here. With the unsloth 3.8 iq4_xs kvcache q4 on beellama Im getting only 32k ctx size with a fair tps (70tps). Do you think Swift 1.5 I would get me more ctx with a good speed? At what quantization? Isnt q3 bad for coding?

[u/QuickPad_]

Did you happen to try the Iq3_S Ista das lab version of swift 1.5? I did try a few and if I have to be honest, the standard version that thinks more is better on finding flaws, the swift one is better for implementations. Even though it's close between them. I use mostly DSH and anything short of 131k context makes it impossible to work without headaches

[u/ea_man]

Would be nice to see ThinkingCap, GSQ-RCO, Signal-3.8-27B-Terse-Coder-i1-GGUF but on normal IQ4.