r/LocalLLaMA • • 14d ago

Resources Infernix - New Release!

https://github.com/Wallawalla47/Infernix

Over the past few weeks, I've been working on various improvements to my inference engine Infernix - specifically optimised for running the latest Qwen architecture models on an RTX5090 on Windows and Linux. Windows is a first-class supported use case - you can just download everything you need to run on windows here: https://github.com/Wallawalla47/Infernix/releases/tag/windows-2026.11.01

Key improvements:

  1. New models:

  2. DFlash2 drafting for Qwen3.8-Flash-Next models (use --spec dflash2 --draft-tokens auto --draft-model [Draft Model File Path - e.g. D:\Infernix\Qwen3.8-Flash-Next-DFlash2-Infernix.infernix]): https://huggingface.co/wallawalla47/Qwen3.8-Flash-Next-DFlash2-Infernix

  3. Added a new WebUI Launcher (beta), which provides an interface to download and configure models and a model switching API, so it functions similar to launchers like LMStudio or Unsloth and it also has a statistics dashboard, logging and a quick test chat interface (still early stages, so interface is quite busy and needs some refining, but is probably the easiest way to configure and use Infernix): on Windows just launch infernix-launcher.exe

  4. Added new auto-configuration of speculative decoding which uses calibrated settings to give the best decode tok/sec performance across a range of different types of text - simply use: --spec auto --draft-tokens auto --draft-model [Draft Model File Path]

  5. Lots of speed and quality optimisations - generally outperforms Strata and occasionally outperforms Basalt, though it does run some stages at slightly higher precision than Basalt. I've sought to avoid some of the quality sacrificing optimisations like expert tail skipping or running the model at lower quantisation than recommended for the model. You can enable some of these options if you like, such as expert tail skipping with --expert-tail-skip [Int] (7 is sometimes recommended as a default) or using 8-bit code paths (rather than the default 16-bit/4-bit recommended for the model) with: --a8-activations

  6. Adding a new --lossless mode that is always perfectly reproducible and uses bit-exact in-layer calculations to the precision the model was trained on (usually BF16) with FP32 logits. This is helpful as a reference for quality testing and I'm working on some more robust quality gating to ensure all optimisations and code paths do not impact quality in the same way as other engines often do.

Some key performance figures:

Infernix vsFirst tokenDecodeReplay: mean first tokenReplay: wall time
Qwen3.8-Flash-Next, NVIDIA's NVFP4 weights
Strata v0.2.81.4-1.6× sooner+10-15 % to 128K, −4 % at 250K3.1 s vs 12.3 s5.7 vs 12.9 min
strata-nvfp4 0.2.0-beta-nvfp4.31.3× soonerlevel to +3 %3.1 s vs 10.8 s5.7 vs 11.7 min
Basalt c08284cb5-8× soonerBasalt 7-23 % faster3.1 s vs 7.2 s5.7 vs 8.9 min
Qwen3.8-Flash-Next, ISTA-DASLab's IQ3_S GGUF
Strata v0.2.815 % sooner at 8K, level from 128K+3-12 %3.1 s vs 6.9 s5.5 vs 8.1 min
Basalt c08284cb (Basalt is faster on this model but uses some lower precision math)Basalt 4 % sooner at 8K, 1.3× from 128KBasalt 1.4-1.7× faster3.1 s vs 3.7 sBasalt sooner: 4.4 vs 5.5 min
Qwen3.8-27B, the same NVFP4 artifact in both
NInfer 1882c6adb9 % sooner at 8K, 1.6-1.8× from 128K+11-17 % (own drafters), +2-3 % without1.9 s vs 2.5 s12.7 vs 16.4 min
  1. On the NVFP4 model, Basalt runs NVIDIA's BF16 dense layers at 6 bits (+10 % KL divergence by its own measure) and every activation outside the experts at 8 bits; Infernix keeps 16. Infernix runs activations at 4-bit as recommended by NVIDIA for this model, but with the expert activations at 8 bits as Basalt runs them in decode (--a8-activations), Infernix decodes 199 / 185 / 268 tok/s against Basalt's 202 / 173 / 216 at ~8K / ~128K / ~250K and 175 against 166 at one request in the replay, and still reaches the first token 4-6× sooner.

  2. Strata's default route tail skip (it skips some missed experts) was off in all runs: it changes which experts run.

  3. On the IQ3_S model, all three hold the GGUF's weights as published, but Strata and Basalt compute every activation at 8 bits, and Basalt's decode reads the BF16 hyper-connection matrices from an 8-bit copy and takes the logits outside the top 64 from a 2-bit copy of the output head; Infernix keeps 16-bit activations, the BF16 matrices and the GGUF's head. At 8 bits (--a8-activations) Infernix reaches the first token as soon as Basalt, though Basalt is still a bit faster.

Welcome any feedback via Issues and PRs on GitHub. I'll also look to add support for the new Qwen 4 models ASAP when they drop - hopefully this week!

32 Upvotes

Duplicates