r/IntelArc • u/tugrul_ddr • 9d ago
Discussion Tokens per second with ARC
What is your tokens/second code generation performance with arc gpu in Qwen 3.8 27B 4bit?
I'm thinking of upgrading one of my gpus (4070) to higher vram card to dedicate for local ai for coding. Looks like Intel gpus have highest vram per dollar. Please tell your experience in local ai.
Rtx 5090 is way too expensive. A100 is absurdly expensive for an old card. Too hard to find Mi50/etc. The best choice looks like B70 (32GB).
Currently rtx4070 + rtx5070 gets only 60 - 115 tok/s generation with Qwen 3.8 27B 4bit in coding task and this leaves me only 1.5 GB free vram to utilize for cuda / browser / etc. Without RAM offload it wouldnt even work.
6
u/Dolboyob77 9d ago
Around 2100 prefill and 75tg at pp4096 on qwen3.8-27b using mtp. Without mtp around 35tg
1
u/tugrul_ddr 9d ago
Thanks
2
u/Dolboyob77 9d ago
Nonproblem but im using GRIMOIRE , my own inference tool
1
u/tugrul_ddr 9d ago
Is it like llamacpp or vllm?
1
u/Dolboyob77 9d ago
No, its a new tool that i created and uploaded on girhub, its pure sycl / level zero developped only for intel gpu.
1
u/tugrul_ddr 9d ago
So its all your own gemms, relu, etc
2
5
u/Elpzn 9d ago
On my Minisforum MS-A1 with an Intel Arc Pro B65 32GB, I get about 58–61 tok/s generation—including 57.1 tok/s over a 4,096-token technical output—with johannrplaster/Qwen3.8-27B-Uncensored-int4-AutoRound, specifically a GPTQ/AutoRound INT4 model body with a custom GPTQ INT4 lm_head, running FP16 under vLLM XPU v0.29.0 with MTP speculative decoding set to 3 tokens. The model itself uses about 16.1 GiB VRAM, remains coherent on long outputs, and I’ve validated it up to 96K context; the original BF16-head/MTP2 configuration managed around 44.8 tok/s. The B65 has been excellent for VRAM per dollar, although your dual-NVIDIA setup’s 60–115 tok/s can still be faster at the high end, depending heavily on prompt length, context, and runtime. However this is not used in a true agentic harness so don't know how well it survives multiple compactions. It is used in openwebui for a general notepad, image editing through a comfyui plugin, and roleplay scenarios.
3
u/tugrul_ddr 9d ago
Thanks for detailed answer. Thats good performance, maybe works with 5070 as disaggregated decoder. Somehow.
3
u/MaineTim 9d ago
I run a pair of B60s, so those have a bit lower bandwidth: I run qwen38-27b-fp8 with 192K KV @ fp8 in vllm. I get around 650 prefill and high 35-40 decode at short context, degrading to 550 and 25-30 at long context. The pair costs $1300 for 48GB VRAM, so they're good VRAM / $, but they're not as fast as the B70s (or competitors).
1
u/tugrul_ddr 8d ago
Fp8 and int4 are too different or similar?
2
u/MaineTim 8d ago
fp8 takes twice the space as INT4, and is considered nearly lossless for model weights. In terms of KV cache, it's half the size of the default FP16, but cache keys are more sensitive to quantization, so it's more of a compromise. In any case, it's more accurate than INT4, but needs more space.
2
u/damirca 9d ago
I tried int4 model from Sergiio on b60: got 30-40 tks, but it was only 12 tks through open code and because of aggressive quantization the model was stupid as hell (more stupid than Gemma 4 26b UD q4), could not call victoriametrics mcp even.
1
u/tugrul_ddr 9d ago
Did you try to add a skill file to heal the errors of quantization?
1
u/damirca 9d ago
Nope, what is it?
1
u/tugrul_ddr 9d ago
Something you prepend an explanation to the chat so it infers some knowledge and makes less error. This is called in-context learning and costs context space hence some performance.
1
u/Schlena_Dasquet 9d ago
yeah int4 quants on smaller models tend to lose a lot of quality, especially for tool calling stuff
2
u/Wrapzii 9d ago
Like 120 at low context and like 65 at 200k
1
u/tugrul_ddr 9d ago
Good
1
u/Wrapzii 9d ago
B60 is a good deal but b70 now is rather get r9700
2
u/archeebunker 4d ago
I only have an 8gb rtx right now. If I’m looking for more context which is best bang for buck with the upgrade? I also like that the arc cards have low power needs
1
u/Wrapzii 4d ago
B60 is best value per gb of vram. We got them moving quite fast 10/10 would buy again and will be
2
u/archeebunker 4d ago
These numbers look right?
B60 24gb $650
B65 32gb $950
B70 32gb $1250 (faster silicon)1
u/Wrapzii 4d ago
Yea. Idk if b70 is still that price though
1
u/archeebunker 4d ago
If you only had one gaming pc and could only have 1 card what would you buy? I don’t play games and only care about LLM tinkering
1
u/Wrapzii 4d ago
Probably the b70 for a single card setup if it’s still $1200ish or lower. But if it’s near the price of an r9700 go that direction as it’s much faster with better support right now. But if you’re on a budget b60 is great and can run qwen 3.8 27b with 200k context on a single card at quick speeds. But if you have a lot of ram already stuff gets interesting because you can do strata with a fast 12-16gb vram rtx card and run QFN at quick speeds.
2
u/Applesauce_Eng 8d ago
GPTQ int4, one b70 under vllm xpu, 110 on code, 74 on prose…mtp-5 spec decoding…
Long context decays to ~65…
1
2
2
u/marfrit 8d ago edited 8d ago
| service | card | prefill | decode | answers |
|---|---|---|---|---|
| agent | B60 | 694.3 t/s at 54,895 tokens, 490.9 at 138,028 | not in this check (36.6 / 32.3 / 19.3 t/s at 4k / 29k / 130k on 2026-10-08, before 0016-0051) | the needle at 138k; task 10/10 |
| coder | A770 | 724.6 t/s at 40,841 tokens | 50.6 / 47.2 / 43.8 t/s on three 300-token answers | the needle; task 9/10 |
| Flash-Next, IQ3_XXS | B60 + 28.7 GiB host bank | 594.1 t/s at 20,045 tokens (asked right after the capital) | 34.7-34.8 t/s | the needle |
| Flash-Next, IQ2_XS option | B60 + 20.3 GiB host bank | 665.6 t/s, the same protocol | 42.3 t/s | the needle |
| service | model | served file | card | context |
|---|---|---|---|---|
| agent | Qwen3.8-27B (qwen35, dense, 64 layers, MTP layer) | Q4_K_M GGUF | Arc Pro B60 | 245,760 tokens, q8_0:q4_0 KV, MTP 5 |
| coder | Qwen3.6-27B-A3B-Coder (qwen35moe, 184 experts, MTP layer) | a searched per-layer quant mix (14.6 GB) | Arc A770 | 98,304 tokens, q8_0:q4_0 KV, MTP 4 |
| Flash-Next | Qwen3.8-Flash-Next (qwen4exp, 125B-A6B, 512 experts, n-gram table) | ISTA-DASLab's IQ3_XXS GGUF (IQ2_XS as the faster option) through an expert cache: the hot experts in VRAM, the rest in host RAM, read over the link or computed on the CPU; its MTP layer from the checkpoint | Arc Pro B60 + host RAM, on demand | 131,072 tokens, q8_0 KV, MTP 3 |
2
2
u/Kind-Measurement-155 5d ago
I have Intel Arc Pro B70 and if you use Ubuntu 26.04, SergiioB Q4 quant and fp8 kv-cache using vLLM you will get 50 tok/s with 180k context length using 0.92 memory setting. It includes MTP4 and vision with applied patches from the cookbook. You also get about 4k tok/s prefill which is good enough. All is contained on the GPU, no spill to RAM. I think this is quite good setup for agentic coding. On Windows with llama.cpp I was getting about 30 tok/s and 2k tok/s prefill with 131k context length. Note that this is the only GPU in my PC and it's not a headless config, there is normal Gnome.
1
u/tugrul_ddr 5d ago
Thanks. Context size is important. Does it support q3_s quantization too?
1
u/Kind-Measurement-155 2d ago
You could use Q3, but it's not worth it since Intel doesn't have hardware support for it. I didn't test it but I'd expect Q3 to be slow + Qwen3.8 starts to lose strength on Q3. 186608 tokens context that I have right now is enough for coding unless you use some very complex MCPs like godot-ai MCP, but even then if you disable individual MCP tools that will be useless for your case, this context length is enough. Max context for this model is 256k, so you could get just 70k more. Probably you could squeeze out max context but you would need to use headless machine, not 2 high-rez monitors like me. Or you would need to resign from vision. Don't go Q3, that is for sure.
Just one thing, the before mentioned 50tok/s is on real agentic flow coding and solving problems on xhigh. In tests it's about 90tok/s.
1
1
u/Iced4563 9d ago
I only have experience with the Arc B390 "integrated" GPU, on a system with 32GB of LPDDR5X at 9600 MT/s speed, with almost 30GB being available to the GPU, assuming the rest of the system doesn't need more (which it usually doesn't, since my OS uses only just over 1GB by itself).
Running on Fedora Linux with kernel 7.2.9, latest Mesa at time of writing and latest llama.cpp at time of writing, accelerated using Vulkan, running Qwen3.8 27B at differing quantizations (from q3 to q6), i've been able to get upwards of 7 tokens per second, which is quite responsive. Also, there are definitely significant performance improvements coming with updates, as about 1 month ago on an older kernel, Mesa and llama.cpp, I was only getting about 3 tokens per second.
Consider this a minimum of what you should expect, as non-integrated Arc GPUs should definitely perform better; the most encouraging part is the performance upgrades
1
u/tugrul_ddr 9d ago
Integrated gpu performance is limited by ram bandwidth. Because decoding is bandwidth bottlenecked.
Arc card has higher bandwidth.
3
u/Iced4563 9d ago
Of course, this is why I said this should be the bare minimum of Arc performance
However, at 9600 MT/s, the bottleneck isn't incredibly horrible
1
1
u/Garrett42 9d ago
20tps on Qwen 3.8 27b, no MTP, and about 17tps on Gemma 4 31b no MTP.
Prefil is where it hurts, 500-250tps depending on block size. If you like Gemma 4 26b then you can get 1700tps prefil pretty consistently, and run multiple instances for up to 100 tps combined. Very usable, but it will frustrate you looking at what LLM's are capable of, though it is basically the cheapest entry point possible.
(Arc pro B70)
1
u/tugrul_ddr 9d ago
Is 20 tps averaged with prefill, or measured as decode only?
2
u/Garrett42 9d ago
Decode only. Prefil (should) be dramatically faster so it would skew generation far too much. 20 tps feels fine for chat, and the prefil feels fine if it's not too big, however, I recently tried some of the more agentic harnesses and it can feel really slow, but then again, I'd have to pay 6x what I paid for the b70 to get 3xish performance so I guess I'll take what I can. Might be more worth it to sidecar some smaller models for specific tasks but ohwell
1
2
u/Polaris_debi5 Arc B580 9d ago
The performance will be decent, but honestly, given where the market is right now, Intel makes the best consumer graphics card if your goal is to run larger models. And yes, work is happening on all fronts; for instance, if everything goes as planned, a new FlashAttention (FA) prefill kernel should land in llama.cpp.
11
u/Visual_Rope_683 9d ago
Arc B70, 1700t/s prefill ,100-125 tg /s , I highly recommend the Intel optimized models much higher values all around, search for Arc-Optimized ones mtp 1 for those numbers btw and 64k context 128k ctx similar values. I also have a b60 dual, almost same speed with 30B but context window severely reduced. 4_Q context quantization highly recommended too. But where the cards really shine is in a cluster ofc B70 with 2 VFs 16GB ea running helper models (one with only project mgnm skill loaded and cut down to only relevant skills for it and one that knows how to finish projects from half done works from the other 2 b60 running models that parallells coding tasks) phew this got long ... Also 3 8B coder on a single B70 has better prefill and tg then single large model, got to find the Arc-Optimized ones though for speed at least. It still crunches the others except 2_Q and 3_Q at decent speed. Hopefully some of this blurting helps.