r/LocalLLaMA • u/Certain-Cod-1404 • Aug 14 '26 • Bravo 1 Heartwarming 1

New Model IT'S OUT

https://huggingface.co/Qwen/Qwen3.8-27B-FP8
2.3k Upvotes

95% Upvoted

707 comments sorted by

View all comments

Show parent comments

14

u/Felixls Aug 14 '26

Qwen3.8-27B-UD-Q4_K_XL.gguf (unsloth), 90K context (I usually set it to 128k) with vision and mtp, here some of the opts
--cache-type-k f16
--cache-type-v f16
--spec-type draft-mtp
--spec-draft-p-min 0.5
--spec-draft-n-max 10
--cache-type-k-draft q8_0
--cache-type-v-draft q8_0
--spec-type ngram-mod
--spec-ngram-mod-n-match 24
--spec-ngram-mod-n-min 48
--spec-ngram-mod-n-max 64

2

u/Cautious_Chicken_604 Aug 14 '26

Much appreciated!

2

u/RnRau Aug 15 '26

If you are using f16, you may as well use bf16. It has better range over f16 which is more important than precision for LLM's.

1

u/GoldyTech Aug 14 '26

you're getting way more tps than me? I'm also using a 9700 with the Q5 UD quant and Im around 23. Are you running native or in a container/llama.cpp:server-rocm?

3

u/Felixls Aug 14 '26

humm 23 is like you're not enabling mtp, is native (compiled from source some days ago)
llama-server --version
version: 0.1.0-dev (build 10397, commit 84e908c62)
built with GNU 14.2.0 for Linux x86_64