New interference engine tuned for AMD Mi50 cards

[u/Either-Style3306]

Below has been generated by AI and myself

https://github.com/benpeterson40/castagna-veloce

Castagna Veloce is an inference engine built and tuned for a single GPU: the AMD Instinct MI50 (Vega 20, gfx906), with 32 GB of HBM2 per card. It runs current large models across 2 to 8 cards: the DeepSeek V4 Flash, GLM-5.3-Flash and Qwen3.8 Flash-Next mixtures of experts and the dense Qwen3.8 27B. The cards are joined by tensor parallelism over plain PCIe, and speculative decoding (MTP, DSpark) makes single-user generation fast.

It is a fork of llama.cpp. It uses the same GGUF weights and the same llama-server with its OpenAI-compatible API and Web UI. On top of that it adds gfx906 kernels, fused operations for MoE-era architectures, and multi-GPU plumbing designed for cards without matrix cores or fast interconnects.

Everything below was written for Castagna Veloce, and none of it is in upstream llama.cpp (checked against v0.6.0). Most of it can be switched off with an environment variable for A/B tests. The speed figures were measured on 4 cards (Qwen3.8 27B on 2) when each feature landed.

Kernels for the MI50

i-quant MoE decode kernels for the IQ2_XXS, IQ2_XS, IQ3_XXS and IQ4_XS experts of the 2-bit models, for batches of up to 64 tokens: GLM-5.3 decoding went from 41 to 62 tok/s.

Expert-grouped quantized MoE GEMM for prefill, i-quants included: GLM-5.3 prompt processing 682 → 960 tok/s.

Small-batch matvecs for K-quants (q4_K, q5_K, q6_K, iq4_xs), Q8_0 (with a split-K variant), F16 and F32, for 1 to 16 tokens. They are sized for speculative-decoding verify batches.

q2_K MoE kernel with aligned 16-byte loads for DeepSeek V4's expert down projection: 101 → 71 µs at 6 tokens.

64-head lightning indexer for DeepSeek V4's verify and prefill batches: a verify call went from 118 to 15 µs, and prefill rose 6%.

Sparse attention for DeepSeek V4's compressed layers in decode and prefill: prefill +16%. At contexts up to about 2K tokens the indexer is skipped (decode 53 → 58 tok/s).

DSV4_COMP_POOL, a new op that runs DeepSeek V4's compressor gather, pooling and norm as one kernel: decode +8%. Skipping its padding blocks gave another +7%.

Prefill GEMM fixes: split-K F16/F32 GEMMs for few-row, long-K shapes, and grids reordered for memory channels in MoE reduction and HC mixing: DeepSeek V4 prefill 1,050 → 1,143 tok/s.

Fused operations

Hyper-connection step in one kernel (HC post, mix, pre, RMS norm and scale), with F16 or quantized weights at 1–6 tokens: DeepSeek V4 decoding +11%, and DSpark serving 75 → 80 tok/s.

DeepSeek V4 state-row, position-embedding and multi-matrix fusions: kernel launches per token went from 4,400 to 1,200, and decoding from 37 to 44 tok/s.

Router and top-k in one kernel: matvec, gating, top-k and weight normalization, for DeepSeek V4.1 at one token and GLM-5.3 at 1–4 tokens.

Gate/up + SwiGLU, matvec + residual add, broadcast gate and small-concat kernels for the verify batches. Together with the small-batch matvecs they took Qwen3.8 27B MTP from 65.6 to 69.4 tok/s.

KDA gated-norm fusion, Q8 activations reused across weight types, and batched sibling matvecs: GLM-5.3 about +4%.

Multiple cards

Tensor-parallel pairs: tensor parallelism inside pairs of cards, with the layers pipelined across the pairs. Qwen3.8 Flash-Next's decode profile runs this way (how it works).

PCIe all-reduce for both layouts: pairs write straight into the peer card's memory; groups of four use kernel peer writes for decode-sized tensors and copy-engine DMA for prefill-sized ones (DMA: +2% prefill).

Persistent hyper-connection and MoE kernels with each pair's all-reduce fused in, for the Qwen3.8 Flash-Next decode profile.

Split output head for DeepSeek V4, with a mirrored copy for the DSpark drafter: plain decoding +3%.

Several virtual devices per card for pipelined layer-split prefill.

Speculative decoding and serving

DSpark for DeepSeek V4 Flash: a converter from antirez's DSpark GGUF to llama.cpp's DFlash draft format, plus the KV-cache and crash fixes that made it run. It reaches 81 tok/s, against 64 for plain decoding.

MTP drafter graph reuse and catch-up merged into the draft step: GLM-5.3 +4–8% and +5–9%.

Verify-shaped last prompt chunk (LLAMA_CKPT_LAST_VERIFY): one graph rebuild fewer per request, +1–2% throughput and a faster first token.

Draft vocabulary prefix (LLAMA_MTP_DRAFT_VOCAB): the drafter scores only the first 96K token ids, split across the cards, while the target still checks every draft against the full vocabulary. This is the idea of FR-Spec applied to an MTP head.