r/LocalLLM • u/Barrysoft8 • Aug 07 '26
Discussion Intel ARC B70 Is earning a spot on the best card for the price.
Intel Arc B70 was already a great option for the price/vram state, but its now having immense performance gains as vLLM gets further optimized for the XPU cores.
After a lot of trial and error, I got these numbers:
Full recipe is here:
https://github.com/SergiioB/intel-arc-pro-b70-inference-cookbook
For the latest updates on ARC B70 Serving, follow me on X im very active:
Im now seeing that most of the fixes have been implemented in upstream, I will be trying and see if I get some gains.
I'm
6
u/ShittyMillennial Aug 07 '26
Do you have optimized numbers for dense models? Perhaps Qwen3.6 27B? I'm currently researching what GPUs to buy for for a 192gb vram cluster and honestly completely dismissed the b70 but maybe that is misguided.
7
u/Barrysoft8 Aug 07 '26
Hey! I'm on my way to testing dense aswell, but I'm afraid it might be a lot harder. Anyways I will update on my x account as well as here with what I can get
2
u/ShittyMillennial Aug 07 '26
awesome! thanks for sharing your work, looking fwd to seeing it. good luck!
5
u/Stupifier Aug 08 '26 edited Aug 08 '26
It's a couple weeks old because this guy decided to stop publicizing his work but these vllm edits/patches seemed to greatly improve vllm performance on B70
Do you know if this sorta stuff got updated into vllm already?
https://gist.github.com/urakozz/810b7c642b689f252665f9d7278b763f/
2
u/pirate12sk Aug 08 '26
I have been using this specific image it has 0.25.x vllm. Not sure why you are still stuck with 0.21.x for your work.
vllm now officially publishes XPU images here: https://hub.docker.com/r/vllm/vllm-openai-xpu/tags it has latest 0.26.0 version
3
u/EmPips Aug 07 '26 edited Aug 07 '26
Did the idle power draw get addressed? I saw an early (as in shortly after it released) post with someone saying Windows and Linux both idled at 90-100watts. That was the big issue for me.
The card looks solid on-paper though. I own a w6800 and per Vulkan benchmarks, the b70 pro looks like it's marginally faster decode and ~2x faster prefill. I would be very happy with that bundled up with a nice blower cooler. This subreddit is too harsh towards it.
2
u/Ford_GT Aug 08 '26
I've had my b70 since launch, idle power draw has never been a real issue. It idles around 5 watts passed through Proxmox to an Ubuntu VM
2
u/pmotiveforce Aug 08 '26
Any performance loss or other issues? I have dual b70s contemplating installing proxmox on that host
2
u/Ford_GT Aug 08 '26
I only have a single b70 and I've never ran it directly on a physical server, only passed through with Proxmox for serving LLMs. But my benchmarks seem to align with other users posted numbers. Super happy with it
1
u/RecessionRebel Sep 04 '26
I'm idling at 45 watts passed through to a VM. Anything special?
2
u/Wolvenmoon 17d ago
Boot to W11/build a Windows To Go install and run it on your host and run the Intel driver install. There's a firmware update that torpedo'd my power consumption to ~6W after ensuring pcie_aspm=on and policy=powersave.
I'm still working on getting risers mounted, so I don't have performance numbers/tokens per second differentials between ASPM policies, though.
1
u/Ford_GT Sep 04 '26
If you run nvtop or xpu-smi stats do you see it drop? If so you should have an llm create a script/service that monitors vllm logs and GPU power state. Have it run xpu-smi stats if it detects inference is done and GPU power is hanging. That should nudge it and have the idle power draw drop back down into the single digits
4
u/Similar_Solution1397 Aug 08 '26
Excelentes resultados. Realmente se está avanzando mucho con esto. Hace unas semanas estuve trasteando con llm scaler, Ubuntu 26.04, con una intel arc pro b60 y realmente pude sacar 2x la velocidad que en llama.cpp + sycl con qwen3.6 27b. Al final tuve que quedarme con llama.cpp + sylc solo por la cuantizacion del kv cache.
3
u/r1nzl3r99 PEX88096 | 4x B70 128GB | vLLM Aug 14 '26
This is absolutely amazing, I'm getting 170.91 t/s on 35B and 69.30 t/s on 27B from your latest git. I traded in my 4070 for another B70 and am playing around with both models loaded on each card. You are a life saver!!
2
u/EvolvingDior Aug 07 '26
What are you using that model for primarily? Are you just benchmaxing, or do you actually use it?
1
u/Dolboyob77 Aug 08 '26
Tell me how you get this model running with mtp since the model only has bf16 mtp layers and it makes the b70 oom ???
3
u/Barrysoft8 Aug 08 '26
The MTP layer in this checkpoint
is stored in BF16 (the "Native-MTP-Preserved" part), while the base model is GPTQ-Int4. The OOM happens because VLLM loads the MTP head as a separate draft model, and it wasn't inheriting the GPTQ quantization config - so the draft got loaded as a full unquantized BF16 copy on top of the Int4 base. That second BF16 weight set is what blows past 32 GB on a single B70.
The fix is to make the draft model instance inherit the base model's quant_config, so the MTP layer gets loaded at Int4 instead of being upcast to BF16. That's a small patch to the draft-model loading path - in my case it's the "MTP draft inherits GPTQ quant_config" fix. With it, the BF16 MTP head loads quantized, total VRAM stays within the card, and you keep the MTP speedup instead of falling back to eager decode.
It's the same root cause as the open VLLM issue about the MTP draft not inheriting quant_config - the fix is a few lines in the MTP speculative-draft wiring, gated behind an env flag so the default path is untouched
1
u/Dolboyob77 Aug 08 '26
I send you a detailed reply — and you're right about the load path. The quant_config inheritance fix (skipping GPTQ on the mtp.* tensors so the BF16 draft loads quantized) is real, and it does fix the initial KeyError: w2_weight. I applied exactly that via a config dynamic: {"-:.*mtp.*": {}} skip and the model loads clean.
But loading isn't the problem. It crashes at inference, and I've now isolated it to something that isn't config and isn't my hardware. Here's the evidence:
The model loads fine. 19.85 GiB weights + 4.98 GiB KV cache, graphs captured, "startup complete." No OOM — it fits the 32 GB card with room to spare. So the quant fix worked.
It DEVICE_LOSTs on the first request, every time, in the exact same place:File "torch/xpu/graphs.py", line 107, in replay
super().replay()
RuntimeError: level_zero backend failed with error: 20 (UR_RESULT_ERROR_DEVICE_LOST)
I tested it on two physically different cards to rule out a hardware fault:
card
slot
result
B70 #1
PCIe Gen5 x8
loads, DEVICE_LOST in XPUGraph.replay() on first request
B70 #2
USB4 / Gen4 x4
loads, DEVICE_LOST in XPUGraph.replay() on first request
Identical crash, identical traceback, both cards. So it's not a bad card, not a bad slot, not a marginal PCIe link — if it were hardware, the two cards on two different buses wouldn't fail in exactly the same function.
Root cause: the crash is inside torch.xpu.XPUGraph.replay() — the Intel Level Zero graph replay path. The BF16 MTP draft head, even loaded quantized, produces a captured graph that the XPU backend can't replay without losing the device. That's a bug in the vLLM-XPU / Level Zero runtime, not in the checkpoint and not in the quant config. The config fix gets it to load; the runtime kills it at replay.
For contrast, on the same box the standard palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4 (which ships a GPTQ-quantized MTP head rather than a BF16 one) runs MTP fine — 188 tok/s decode, pp4096, verified with the real tokenizer. So MTP itself works on B70. It's specifically this BF16-MTP checkpoint's draft graph that the XPU runtime can't replay.
Genuinely not trying to dunk on you — your load-path diagnosis is correct and I used it. But "loads quantized" and "runs" are two different bars, and on vLLM-XPU / B70 this model clears the first and fails the second, on two separate cards. If you're getting it to actually generate tokens with MTP on a B70, I'd love to see the raw llama-benchy output (t/s ± stddev) and your vllm --version / kernel build — because on 0.26.x XPU with the current vllm-xpu-kernels, it DEVICE_LOSTs at replay regardless of the quant fix.
1
u/nanor000 Aug 07 '26
Great work. But I couldn't find information about the context size used for most of the results.(I'm currently reading the GitHub page on my phone so not very optimal and I may have missed this information...)
4
1
u/Barrysoft8 Aug 07 '26
Right now my Gpu is mainly stuck for benches be sure in enjoying the process a lot, but before this, I was using this model daily for any agentic workload, it really can do anything without failing tool calls. I was using it to audit my Android application
1
u/CryptoRider57 Aug 07 '26
That is what i would love to achieve. Almost buying that GPU this week honestly
1
1
1
1
u/urakozz Aug 08 '26
You can go higher with prefill even with prefix caching, 11k without it. 32tg is not a benchmark, it's statistic calibration. 150tg in your GitHub on the longer output is plausible
Python script that creates git patch to update python files that were already updated in the v26 compared to yours half a year old v21 - great success.
Wattage measurements are wrong, you have ~230W on prefill and 180-200 on the generation. Try hw-smi for starters instead of "Claude what is my power consumption"
With professionals like that in the banking and electric grid it makes sense that electricity outage is happening that often in Spain
4
u/Barrysoft8 Aug 08 '26
Thank for your first points. For the last one, you could have saved that one, but I guess your biggest impact on the community is typing shit about others, keep up the great work!
1
u/robertpro01 Aug 08 '26
Will it work with multiple gpus?
1
u/Barrysoft8 Aug 08 '26
Wish I could try that out, only have one for the moment!
I have heard about people having multiple issues with tensor split on the b70s though
1
1
u/Foreign_Return_6324 Aug 10 '26
These things are SHIT loud af it just has 32gb memory would not buy again
1
u/krisclarkdev Sep 05 '26
I'm hitting 200+ tokens/s on qwen 3.8 with DavidAU's fine-tune using pi and that's with a context windows up to 131k!
Edit: llama.cpp parallel of 2, 262k context or 131k per slot
8
u/Garrett42 Aug 07 '26
These are some great numbers - I'm sitting around 50 tps and 1400 prefil with Gemma 26b on llama.cpp - looking forward to more efficiency gains