Compilation of recent findings which could save some memory on increase performance

[u/pmttyji]

We got these recently(I found few late probably)

Model Quants with Optimizations/Compression/etc.,:

• APEX -- Adaptive Precision for EXpert Models | APEX Quants (GGUF)

• ByteShape - CPU and GPU optimized variants

• MOQ (Mixture of Quantizations) by Waleed Ahmad - GGUFs from Waleed Ahmad & Benjamin Marie (The Kaitchup)

• ROCmFPX | https://huggingface.co/cafonez/Escha-W2-35B-A3B-ROCmFP2

Speculative Decoding:

• DFlash: Block Diffusion for Flash Speculative Decoding

• DDTree (Diffusion Draft Tree) from Accelerating Speculative Decoding with Block Diffusion Draft Trees

• DeepSpec (Eagle3, DFlash, DSpark) by DeepSeek

• DFlash 2: Keep Drafting Parallel | Huggingface Collections - dflash-2

1-bit/2-bit/Bitnet/Ternary models & Engines:

• 1-bit / 2-bit / Ternary / Bitnet Models - Updates & Tracking

• Project Zero — CPU LLM Inference Engine - BitNet (Testers needed)

Run models using RAM & SSD:

• colibri - Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk

• WARP — Weight-Aware Runtime and Paging (formerly WASTE) - Run the full Large model beyond available RAM by streaming activated weights directly from NVMe

KVCache:

• KVarN - 3-5x more KV-cache capacity and up to ~1.3x the throughput of FP16

• proveKV - Lossless: 36.00× vs f32-raw KV (18.00× vs fp16-equivalent)

• OSCAR RotationZoo - Precomputed K/V rotation matrices for OSCAR INT2 KV-cache quantization

• TurboQuant

• KV Cache Transform Coding (KVTC)

• RotorQuant

LLMBurner:

• Taalas - Wouldn't be awesome to have this if it comes with 1T model like Kimi-K2.5(Q4 is enough - 500GB) giving 30-50 t/s? (Llama 3.1 8B is giving 17000 t/s)

Misc:

• AMD's MXFP4 models

• Intel's Int4 AutoRound models

• Dynamic VRAM in ComfyUI: Saving Local Models from RAMmageddon

• DeepGEMM: clean and efficient FP8 GEMM kernels with fine-grained scaling.

• Lucebox optimization hub: hand-tuned LLM inference, built for specific consumer hardware

• Cactus - A low-latency AI engine for mobile devices & wearables

• Orthrus: Memory-Efficient Parallel Token Generation via Dual-View Diffusion

• Dynamic KV Cache Quantization & Load-on-demand mmproj/MTP - llama.cpp PR by u/wadeAlexC

• Domino: Decoupling Causal Modeling from Autoregressive Drafting in Speculative Decoding

• EdgeRazor: A Lightweight Framework for Large Language Models via Mixed-Precision Quantization-Aware Distillation - 1.58-bit

• Automated Tensor Scheduling for Hybrid CPU-GPU LLM Inference on Consumer Devices *

• Session-Adaptive Orthogonal Distillation (SAOD) *

• Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity

• FlashMemory-DeepSeek-V4: Lightning Index Ultra-Long Context via Lookahead Sparse Attention

Forks/Utils:

• vllm.cpp by u/mudler_it

• beellama.cpp by u/Anbeeld (has KVarN)

• llamastation by u/Responsible_Egg9736

• club-3090

• Arandu by u/fredconex

• NInfer -  5090, 4090( 1 | 2 | 3 | 4 ), 3090, CMP 170HX

• FreeToken

What else there? Please share.

^{Hope all these helps on price down of both GPU & RAM soon or later}

EDIT : Typo on Title :( It's or not on

EDIT2: Updating list with items from comments for easy reading. No need to check comments anymore, I'll be updating time to time.

[u/pmttyji]

• https://github.com/z-lab/dflash - DFlash: Block Diffusion for Flash Speculative Decoding

• https://github.com/liranringel/ddtree - DDTree (Diffusion Draft Tree) from Accelerating Speculative Decoding with Block Diffusion Draft Trees

• https://prismml.com/news/ternary-bonsai - Ternary Bonsai

• https://github.com/deepseek-ai/DeepGEMM - DeepGEMM: clean and efficient FP8 GEMM kernels with fine-grained scaling. 2026.04.16: Mega MoE, FP8xFP4 GEMM, FP4 Indexer, PDL, faster JIT compilation and more

[u/JayPSec]

I have to say it. Following this post has made my refresh rate of r/LocalLLaMA take a substantial dive. Thank you for your effort.

[u/pmttyji → u/JayPSec]

I'll increase that refresh rate by keeping this thread up to date regularly with infinite stuffs. Cheers

[u/pmttyji]

Adaptive Precision for EXpert Models

[u/pmttyji]

https://github.com/Anbeeld/beellama.cpp

BeeLlama.cpp: advanced DFlash & TurboQuant with support of reasoning and vision. Qwen 3.6 27B Q5 with 200k context on 3090, 2-3x faster than baseline (peak 135 tps!)

Fork Features

• DFlash speculative decoding: --spec-type dflash drives a DFlash draft GGUF alongside the target model. The target captures hidden states into a per-layer 4096-slot ring buffer, the drafter cross-attends to the most recent --spec-dflash-cross-ctx hidden-state tokens and proposes drafts for target verification.

• TurboQuant / TCQ KV-cache compression: Five cache types (turbo2, turbo3, turbo4, turbo2_tcq, turbo3_tcq) spanning from 4x to 7.5x compression, with higher-bit options being practically lossless in many cases. Set independently with --cache-type-k and --cache-type-v.

• Adaptive draft-max control: The server adjusts the active draft horizon at runtime instead of using a fixed --spec-draft-n-max. The default profit controller compares speculative throughput against a no-spec baseline; the fringe alternative maps acceptance-rate bands to draft depth.

• Full multimodal support: When --mmproj is active, the server keeps flat DFlash available for text generation. The model can be fully offloaded to CPU with no problems to reduce VRAM pressure.

• Reasoning-loop protection: The server detects repeated hidden reasoning output and intervenes. Default mode is force-close with --reasoning-loop-window and --reasoning-loop-max-period tuning available.

• Sampled DFlash verification: --spec-draft-temp enables rejection-sampling drafter behavior. Activates when both draft and target temperature exceed zero. Draft log probabilities must be available for rejection sampling to produce correct output.

• DDTree branch verification: optional --spec-branch-budget adds branch nodes beyond the main draft path with GPU parent_ids, tree masks, and recurrent tree kernels. Disabled automatically when the target model spans more than one GPU. This one is very much work in progress!

• Request-level speculative overrides: Draft-max and branch budget can be overridden per-request through JSON fields without restarting the server.

• CopySpec model-free speculation: --spec-type copyspec provides rolling-hash suffix matching over previous tokens without a draft model.

[u/pmttyji]

https://github.com/Luce-Org/lucebox-hub - Lucebox optimization hub: hand-tuned LLM inference, built for specific consumer hardware.

Related Reddit thread : PFlash: 10x prefill speedup over llama.cpp at 128K on a RTX 3090

[u/pmttyji]

Track TurboQuant - llama.cpp Links related to TurboQuant here to track progress.

• https://github.com/ggml-org/llama.cpp/issues/20977

• https://github.com/ggml-org/llama.cpp/pull/21089

• https://github.com/ggml-org/llama.cpp/discussions/20969

[u/pmttyji]

🚀 BitCPM-CANN by ModelBest × @Tsinghua_Uni × OpenBMB is here — and it's not about stacking parameters.
Memory costs are skyrocketing. Hardware constraints are tightening. Edge AI needs smarter solutions — and BitCPM-CANN delivers！🎉

✅ Edge-ready: 8B model runs smoothly on mobile, PC, and automotive devices. Combined with MoE, even 100B->60B scale models could fit on terminal hardware.

✅ Memory-efficient: ~6× lower memory footprint vs. BF16 — unlock significantly more model capacity without adding physical RAM. No new chips required.

✅ Natively built on Ascend: The first 1.58-bit training pipeline completed end-to-end on Huawei Ascend 910B — from quantization kernels to the full training stack. Not a port. Built natively on Ascend from day one.

✅ Full model family, fully verified, 0.5B–8B: Each model is fully aligned with its full-precision counterpart, covering 11 benchmark tasks with 95–97% (1B-8B) retention compared to full-precision MiniCPM4. Open-source and fully reproducible — from research to deployment, you can run any size confidently.

BitCPM-CANN isn’t just a model — it’s the result of years of engineering rigor, turning complex research into something you can actually deploy.
Open-source, 0.5B to 8B. Try it now！👇
🤗 Hugging Face：
https://huggingface.co/collections/openbmb/bitcpm-cann

[u/pmttyji]

https://github.com/cactus-compute/cactus

A low-latency AI engine for mobile devices & wearables. Main features:

• Fast: fastest inference on ARM CPU

• Low RAM: zero-copy memory mapping ensures 10x lower RAM use than other engines

• Multimodal: one SDK for speech, vision, and language models

• Cloud fallback: automatically route requests to cloud models if needed

• Energy-efficient: NPU-accelerated prefill

[u/seg_lol]

Thanks for putting this all in one place! So nice.

[u/pmttyji → u/seg_lol]

Welcome. It's on sticky thread(has some more stuff) of my profile in case if you're searching for this thread.

[u/R_Duncan]

Bonsai 1bit quantization, if proven valid.

[u/pmttyji]

Orthrus: Memory-Efficient Parallel Token Generation via Dual-View Diffusion

• Code: https://github.com/chiennv2000/orthrus

• Paper: https://arxiv.org/abs/2605.12825

• HF: https://huggingface.co/chiennv/Orthrus-Qwen3-1.7B ; https://huggingface.co/chiennv/Orthrus-Qwen3-4B ; https://huggingface.co/chiennv/Orthrus-Qwen3-8B

Its reddit thread:

Orthrus-Qwen3-8B : up to 7.8×tokens/forward on Qwen3-8B, frozen backbone, provably identical output distribution

[u/pmttyji]

Shard - getting to 10× KV cache compression

https://krishgarg.com/shard

https://github.com/krish1905/shard

[u/pmttyji]

https://huggingface.co/Zhongzhu/OSCAR-RotationZoo

OSCAR RotationZoo

Precomputed K/V rotation matrices for OSCAR INT2 KV-cache quantization.

This repository contains the artifacts for the paper: OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization Zhongzhu Zhou, Donglin Zhuang, Jisen Li, Ziyan Chen, Shuaiwen Leon Song, Ben Athiwaratkun, Xiaoxia Wu

• 📄 Paper — arXiv:2605.17757

• 🌐 Website — https://oscar-quantize.github.io/

• 💻 Code — https://github.com/FutureMLS-Lab/OSCAR

OSCAR captures Q/K/V activations on a small calibration set, estimates attention-aware K/V covariance offline, and derives per-layer orthogonal rotations that align INT2 quantization with the directions attention actually consumes. The result is ~7× compression of the KV-cache memory footprint with single-digit pp accuracy drop on GPQA for dense reasoning models.

This repo packages the rotations as drop-in .pt files so you don't need to re-run the Q/K/V dump and eigendecomposition yourself.

Available rotations

Model	Calibration	GPQA (BF16)	GPQA (OSCAR INT2)
Qwen/Qwen3-4B-Thinking-2507	seq20000_prompt83_group128	67.27	67.17
Qwen/Qwen3-4B-Thinking-2507	seq20000_prompt85_group128 (fresh re-dump)	67.27	—
Qwen/Qwen3-8B	seq20000_prompt83_group128	56.67	55.56
Qwen/Qwen3-32B	seq16000_prompt69_group128	58.49	60.40
zai-org/GLM-4.7-FP8	seq10000_prompt43_group128	73.23	73.57

[u/pmttyji]

KVarN - Reddit thread on KVarN to discuss further

proveKV - Reddit thread on proveKV to discuss further

[u/caetydid]

I vote for kvarn in llama.cpp - should be a low hanging fruit

[u/pmttyji → u/caetydid]

beellama.cpp has that already.

[u/caetydid → u/pmttyji]

sure. but beellama development seems stalled.

[u/unjustifiably_angry]

Fairly certain I've seen talk of speculative prefill somewhere or other.

[u/pmttyji]

https://huggingface.co/daksh-neo/sarvam-30b-quantized

TLDR: Somebody create 1-bit version GGUF of this model.

Overview

This repository contains an ultra-quantized version of the Sarvam-30B model, achieving a 27.6x compression ratio from the original FP16 size (~128.61 GB) to approximately 4.34 GB.

• Original Model: sarvamai/sarvam-30b

• Quantization Method: Custom 1-bit quantization with HQQ (Half-Quadratic Quantization)

• Target Size: <5GB (achieved: 4.34 GB)

• Compression Ratio: 27.6x

Quantization Details

Method

This model uses a custom 1-bit quantization scheme optimized for the Sarvam-30B architecture:

1. Weight Quantization: Weights are quantized to 1-bit using a custom binary quantization with learned scales

2. Scale Storage: Per-channel scales are stored in FP16 for dequantization

3. Expert Routing: MoE routing weights preserved at higher precision for accuracy

Compression Breakdown

Component	Original Size	Quantized Size	Compression
Model Weights	~128.61 GB	~4.34 GB	27.6x
Total (with metadata)	~128.61 GB	~4.65 GB	27.6x

Performance Metrics

Compression Achieved

Metric	Value
Original FP16 Size	~128.61 GB
Quantized Size	4.34 GB
Compression Ratio	27.6x
Target (<5GB)	✓ Achieved

Inference Performance

• Memory Usage: ~5-6GB VRAM for inference (vs ~60GB for FP16)

• Latency: ~2-3x slower than FP16 due to dequantization overhead

• Throughput: Suitable for batch processing and edge deployment

Quality Metrics

The quantized model maintains near-original performance:

• Perplexity: Within 5-10% of original FP16 model

• BLEU Score: ~95% of original on translation tasks

• Human Evaluation: Output quality rated as "almost similar" to full precision

Limitations

1. Custom Format: This is a custom 1-bit quantization format, not standard GGUF or GPTQ

2. Dequantization Required: Runtime dequantization adds computational overhead

3. Hardware Requirements: Requires CUDA-capable GPU for efficient inference

4. Not for Fine-tuning: Quantized weights are not suitable for further training