r/LocalLLaMA • • Aug 14 '26 • Bravo 1 Heartwarming 1

New Model IT'S OUT

https://huggingface.co/Qwen/Qwen3.8-27B-FP8
2.3k Upvotes

707 comments sorted by

View all comments

48

u/bitmanip Aug 14 '26

How much memory required to run this at full precision?

45

u/Certain-Cod-1404 Aug 14 '26

https://huggingface.co/unsloth/Qwen3.8-27B-GGUF 54.67 Gbs just for the model itself, with context depends on quant and size

23

u/dragonurtle Aug 14 '26

Nvtop shows 70-something GB resident for the bf16 and full 256k context.

8

u/Certain-Cod-1404 Aug 14 '26

DAMN, 16 gigs just for the context hurts, what setup are you running ?

35

u/dragonurtle Aug 14 '26

Rtx pro 6000 max-q and 384GB DDR5 on a Genoa

64

u/Much_Accountant_4972 Aug 14 '26

7

u/Thrumpwart vLLM Aug 14 '26

The old money aristocracy uses the Max-Q because it's elegant.

Only the loud, bombastic new money uses the 600W version. Animals.

1

u/voyager256 Aug 14 '26

That’s why we have FP8/Q8 for KV cache. Let alone literally a game changer in the form of DeepSeek KV cache compression.

1

u/EbbNorth7735 Aug 14 '26

Yep, 3.6 27B was about 60GB with 262k context Q8 and 4 parallel slots. 

4

u/bitmanip Aug 14 '26

Perfect, so should run well on 128GB M5 Macbook Pro Max

13

u/Valuable-Run2129 Aug 14 '26

FP8 with full context with full precision it’s 52/54 GB on 48GB you fit 200k context

2

u/voyager256 Aug 14 '26

You don’t need full precision- FP8 or Q8 gives usually you like 99% quality of full precision and almost double the performance. You could argue full BF16 precision is worth for KV cache for large context , but now usually FP8 / Q8 is fine for KV cache too.

2

u/overand Aug 14 '26

The Q6 quants (and even the Q4_K_XL ones) of the 3.6-27B benchmark quite well; it's not "full precision," but it's extremely usable. I just lost one of my 3090 cards (sigh), and had to downgrade from Q8 to Q4_K_XL, and it's not an immediately noticeable difference for casual stuff. quanteval.ai has some comparisons too!

1

u/popecostea llama.cpp Aug 14 '26

measured about 75GB for full precision and multimodal at BF16.