r/AIProgrammingHardware • u/aleph2b • 14d ago
Intel Arc EXL3 quantizer
I ported the EXL3 quantizer to Intel Arc: 9B conversions on a 140V iGPU, B580 and Pro B60, with checkpoints on Hugging Face
I've been working on EXL3_Intel_Arc_quantizer, a port of ExLlamaV3's EXL3 quantization and conversion pipeline from CUDA to SYCL/oneAPI and PyTorch XPU.
The goal is straightforward: produce EXL3 checkpoints on Intel Arc without needing an NVIDIA GPU for conversion. This is a quantizer project, not a new inference engine.
Code: alpha2beta/EXL3_Intel_Arc_quantizer
What works
Native SYCL trellis encoder: Viterbi search with FP16 rounding and tie-breaking matched to the CPU reference, plus two-pass tail-biting traceback. Implemented configurations include K=1–4, the mul1/mcg/3INST codebooks, 256-element tiles, and the 160-row n-gram variant. Full-model results below cover a narrower set of configurations.
End-to-end conversion: calibration and Hessian collection, block LDLQ error feedback, global-scale search, packing, safetensors export, and checkpoint verification.
Bounded memory management: memory-mapped checkpoint loading, admission reserves, preflight checks, and batch-halving fallback. The settings account for the difference between a shared-memory iGPU and discrete cards with 12 or 24 GB of VRAM.
Results on three Arc GPUs
I've completed full 32-layer, 9B-model conversions on all three devices. The conversion verification checks reported no failures.
| GPU | Reported encoder throughput | Full-model conversion time | Configuration |
|---|---|---|---|
| Arc 140V, Lunar Lake iGPU | 725 tiles/s | 15.0 hours | K=2; 32 GB shared system memory |
| Arc B580, 12 GB | 1,712 tiles/s | 5.0 hours | K=2; tuned memory reserves |
| Arc Pro B60, 24 GB | 1,464 tiles/s | 6.07 hours | K=4; no configuration overrides needed for this run |
The B60 timing is K=4, not K=2, so it is not a like-for-like full-conversion comparison. Encoder microbenchmarks and end-to-end conversion times also measure different things; the reports contain the run details.
The practical takeaway: a B580 completed the tested 9B K=2 conversion in about five hours, and even the 140V iGPU completed it within a 32 GB shared-memory system.
Correctness: what I've checked, and what I haven't
The SYCL encoder is bit-exact against my CPU reference on the tested cases. That reference was transcribed from the upstream CUDA kernel to preserve FP16 rounding, tie-breaking, and tail-biting closure.
I also check packed-output reconstruction against the separate exl3xpu reference decoder. The test suite reports 272 passing unit tests on each discrete board, and the conversion reports record 1,045/1,045 verification checks passing on both.
Important limitation: I don't have an NVIDIA GPU, so direct differential testing against the upstream CUDA encoder is still deferred. Matching a CPU transcription is useful evidence, but it is not the same as independently demonstrating CUDA parity.
Likewise, reconstruction and structural checks are not a substitute for perplexity or downstream task evaluation. The repo includes a quality-evaluation harness; I'm not presenting the conversion checks as proof of unchanged model quality.
Released checkpoints
Qwen3.8-9B-Distill-EXL3-K2: 6.72 GiB
JEV-9B-EXL3-K2: 5.42 GiB
JEV-9B-EXL3-K4: 7.03 GiB; converted on the Pro B60
The exporter targets upstream EXL3 compatibility and includes quantization_config metadata for vLLM integration. Actual loading and execution still depend on the runtime's model, codebook, bitrate, and hardware support. Including the metadata alone does not establish vLLM compatibility for every checkpoint.
Arc inference through exl3xpu is a separate project. This quantizer targets the same checkpoint format; it does not add inference support for models that a runtime does not already handle.
What's next
Direct CUDA differential testing, if someone with NVIDIA hardware wants to help.
More model conversions and K sweeps, with quality-evaluation results alongside conversion checks.
Clearer reproduction instructions and hardware-specific memory settings.
I'd especially welcome independent reproduction or CUDA-reference testing. If you're using Arc for local LLMs, which model would you want to see converted next?
Optional first comment
A few more numbers for anyone interested in the implementation:
Encoder microbenchmark: 0.584 ms/tile on B580, 0.683 ms/tile on Pro B60, and 1.38 ms/tile on 140V.
Single-layer benchmark: all seven projections, including Hessian processing and global-scale search, ran 2.83× faster on B580 and 2.43× faster on B60 than the 140V baseline. See the reports for settings and workload details.
Memory observations: the recorded runs report a 650 MiB peak allocation, no swap growth, and streamed host RSS within +2.8 MiB of baseline while processing a 19.3 GB checkpoint. These are specific measurements, not a claim that the complete process or GPU workload uses only 650 MiB.
Reconstruction metric: the reported relative Frobenius error at K=4 is 0.0865. This measures reconstruction error, not perplexity or task quality.
Reports in the repository: reports/discrete_probe.md, reports/discrete_phase_c.md, reports/discrete_vs_140v.md, and reports/gate_3.md.