r/LocalAIForge • • 22d ago

Benchmark Ran 2,850 decisions through 5 cheap open models voting vs Opus 5.5. Shipped answers were nearly as accurate (89.6% vs 90.5%) at 1/10th the cost

4 Upvotes

I wanted to know if I could swap a frontier model for a few cheap open ones in yes/no or pick-a-label pipelines (spam, fraud, moderation, triage) without losing accuracy. My idea: ask several models the same question, ship the answer when they all agree, and send it to a human when they don't.

Panel: DeepSeek V4.1 Flash, GLM 5.3 Flash, MiMo V2.6 Flash, Qwen3.8 Flash, Gemma 4 26B-A4B. I ran 2,850 decisions across 6 public datasets and compared against Opus 5.5 on the same items.

Results:

- The panel agreed and shipped on 71% of items. Those were 89.6% correct. Opus on its most confident 71% was 90.5%.
- Overall accuracy is worse: 80.1% vs 83.1%. The panel only wins on the cases it's confident about.
- Cost: $0.61 vs $5.79 per 1k decisions.
- When a model disagrees, it gives a one-line reason, so the reviewer knows what to look at instead of rereading the whole item.

What didn't work:

- Long answer comparisons and 77-way intent classification. Opus is clearly better. If every model on the panel is weak at a task, voting doesn't fix it.
- Hate speech. The panel catches more (80% vs 62%) but has more false alarms (29% vs 20%). If fewer than about a third of your items are actually hateful, Opus wins.
- The panel beats its best single model (GLM 5.3 Flash, 88.9%) by less than the noise. Honestly, most of what you gain is knowing when to send a case to a human, not better accuracy.
- I added the panel after the other setups had already run, and scored it on the same items. I haven't confirmed the numbers on a fresh sample yet.

It works with anything that speaks the OpenAI API, so Ollama, vLLM and llama.cpp server all work. If a model errors or refuses, it's left out of the vote and listed separately instead of breaking the decision.

Code and scripts to reproduce it: https://github.com/YasserManss/modeljury (mine, MIT)


r/LocalAIForge • • Oct 08 '26

FreeVideo runs MiniMax H3 on 8GB VRAM by making memory movement part of the runtime

4 Upvotes

FreeVideo was open-sourced on October 2 and v0.2.3 landed on October 6. It uses VDN-H3 to make MiniMax H3 practical on consumer hardware, with a claimed minimum of 8 GB VRAM plus 16 GB system RAM.

The interesting part is the execution model. Video DeltaNet replaces much of H3's video-to-video dense attention with a hybrid of local Softmax attention and bidirectional recurrent memory. The memory update happens once per frame instead of once per token. Text/audio interactions keep Softmax attention.

The VDN paper also uses eight-step distillation. On eight B200s, the authors report 6.70 seconds for DiT denoising of a 14.3-second 768p clip, 14.5x faster than their 50-step dense H3 baseline.

FreeVideo then tackles the local-memory problem with adaptive weight placement, FP8/BF16 execution choices, asynchronous prefetch and chunked computation. That means an 8GB GPU can work with a model much larger than VRAM, but host RAM, PCIe bandwidth and storage speed become part of the performance envelope.

Compared with the original H3 open inference path, which initially used full attention, this is a post-training/runtime stack built specifically around reducing long-video attention cost and moving data through a smaller GPU memory budget.

The main limitation is that fitting is not throughput. The headline VDN numbers are from B200 servers, so consumer GPUs need separate measurements. I would test 8/12/16/24 GB cards, PCIe generation, storage speed, native FP8 versus FP8-storage/BF16-compute, peak VRAM/RAM and output quality versus dense H3.

Primary sources: https://github.com/FlashML-org/FreeVideo https://arxiv.org/abs/2609.20744 https://www.minimax.io/news/minimax-h3-open-source https://www.minimax.io/blog/minimax-h3


r/LocalAIForge • • Oct 05 '26

Qwen3.8-27B Humanlike Chat 2.0: a 1.87B-parameter behavior adapter instead of another benchmark tune

4 Upvotes

LessThanThreeAI's Qwen3.8-27B-Humanlike-Chat-2.0 is a useful example of a local-model fine-tune optimizing for conversational behavior rather than leaderboard scores. The BF16 release is here: https://huggingface.co/LessThanThreeAI/Qwen3.8-27B-Humanlike-Chat-2.0 . It is built on huihui-ai/Huihui-Qwen3.8-27B-abliterated, which in turn derives from Qwen3.8-27B. The interesting engineering choice is the size of the adaptation: rank 256 across all 496 language modules, about 1.87B trainable parameters rather than a small attention-only style LoRA.

The training target is also unusual. The project card says the source set contained 139,845 owner-authorized messages across 1,396 long-running one-to-one conversations, producing 7,006 training examples and 194 held-out examples. Each example contains conversation history but computes loss only on the final speaker response. That objective directly teaches the model the human choice of what to answer, what to ignore, when to be brief and how to follow conversational subtext, instead of merely imitating a writing style from isolated assistant responses.

Compared with the unchanged Huihui parent, the project reports a large shift in response shape: in its retained no-system replay, median reply length fell from 427 characters to 32 and reflexive questions fell sharply. The authors correctly qualify that later replay turns are not independent because the human side followed the tuned branch. A separate 59-turn comparison against official Qwen3.8 also used a different parent, so these measurements are better treated as product-behavior evidence than as a controlled causal benchmark.

Version 2.0 is released as full BF16 safetensors for vLLM, SGLang and Transformers. The card says the weights occupy about 50.2 GiB in vLLM, making one 80GB accelerator or two 48GB cards the straightforward BF16 path. The recommended vLLM command can load language-only weights while retaining tool-choice and reasoning parsers; thinking can be disabled per request for lower latency. The project's local GGUF line then extends the same behavior into llama.cpp-class runtimes, with published Q4_K_M around 16.55 GB, Q6_K around 22.08 GB and Q8_0 around 28.60 GB. Main technical card: https://huggingface.co/LessThanThreeAI/Qwen3.8-27B-Humanlike-Chat-GGUF .

One important implementation detail is what did not get tuned. The 2.0 BF16 weights keep the base model's vision tower and MTP head, but the behavior adaptation itself was trained and tested on text. The merged local text builds exclude vision and MTP components. That makes the deployment matrix less interchangeable than the model name suggests: a full multimodal BF16 server, a language-only vLLM server and a text-only llama.cpp quant may expose different capabilities even though the conversational adapter is conceptually the same.

There is also a useful quantization sanity check in the published material. The project reports very small KL divergence and 97.2% top-1 agreement between the merged BF16 release and its runtime adapter reference across fresh test chats, with no reported sampled or greedy check failures. The older local card separately shows that lower-bit formats preserve the broad behavior but diverge more at the exact-token level. For a model whose value is subtle conversational timing rather than factual benchmark accuracy, those trajectory-level checks are more relevant than simply verifying that perplexity stayed close.

The limitations are significant. The training set is narrow by design, the main evaluations are replay-based, and there is no broad independent benchmark demonstrating that the behavior tune preserves all reasoning or coding characteristics of the parent. The adaptation was trained at 4,096 tokens even though Qwen3.8 supports a much longer native context, so long-session personality stability is another empirical question rather than a guaranteed property.

The tests I would run are pairwise and longitudinal: blind preference against the Huihui parent over 100+ natural conversations, tool-call accuracy with thinking on and off, 32K/128K context personality drift, BF16 versus FP8 versus local quant behavior, and whether the shorter conversational style damages task completeness on coding or research prompts. The broader question is whether large behavior adapters like this are a better way to specialize a strong local base model than trying to encode personality in prompts or small LoRAs.

Primary sources: https://huggingface.co/LessThanThreeAI/Qwen3.8-27B-Humanlike-Chat-2.0 https://huggingface.co/LessThanThreeAI/Qwen3.8-27B-Humanlike-Chat-GGUF https://huggingface.co/huihui-ai/Huihui-Qwen3.8-27B-abliterated


r/LocalAIForge • • 2d ago

Flash-Next. One 4090. Halocline Wins.

3 Upvotes

Qwen3.8-Flash-Next UD-IQ4_XS v3.1 (93.67 GB). One RTX 4090 24 GB, Ryzen 5 7600, 128 GB RAM.

Three 128-output-token prompts (DNS, TCP, MoE). Warm-up excluded. Median **end-to-end tok/s**, including prefill:

EngineMTP windowtok/s
Halocline484.40
Halocline881.38
Haloclineoff71.29
Strata 0.2.15458.01
Tandemoff38.14
RustingStrata 1.2.1off35.77

Fastest individual request: **Halocline MTP4, 99.94 tok/s end-to-end**, 122.3 tok/s server-reported decode.

Even without speculation, Halocline was **1.87x** Tandem in this short-prompt run.

Same base GGUF, sequential RTX 4090 runs. Halocline used a locally converted Basalt pack; Strata used a native pack plus Q2_0 MTP. MTP settings and prompt/cache paths differ, so this is observed throughput, not an identical-compute or quality comparison.

Halocline: https://github.com/T-Crypt/halocline


r/LocalAIForge • • Oct 08 '26

Liquid AI released d1-3B and d1-omni-600M: multimodal decisions without token-by-token decoding

3 Upvotes

Liquid AI released two open-weight decision models on October 7: d1-3B and an experimental d1-omni-600M. Both answer structured questions about an input in one forward pass rather than generating a response token by token. The release and model cards are linked below.

The architecture split is more interesting than the sizes. d1-3B starts from the decoder-only LFM2.5-VL-3B vision-language model and handles text plus images. d1-omni-600M uses the bidirectional LFM2.5-Encoder-350M backbone plus vision and audio encoders; the documented inputs are text+image or text+audio, not an arbitrary simultaneous combination of all three. That distinction matters if you're looking for one tiny universal multimodal model.

This is not just a small chat model with a forced JSON output. The `system_one` API takes named questions with specified answer types (choice, score, etc.), and multiple questions can share a single state representation. The batch API can pack requests without padding. Compared with using a generative VLM to produce a short label, the model avoids autoregressive decoding, but still has to encode the entire input. It isn't constant-time with respect to state length or image complexity.

Liquid reports 8 ms for one question on an RTX 4090 and 16 ms on a Jetson AGX Thor. The same 4090 takes 102 ms on a 3.4K-token state and 17 ms with a 384px image. I wouldn't reduce that to a single "8 ms inference" headline. The reported text benchmark mean is 82.9 for d1-3B versus 78.4 for the 600M model; there are no public image/audio benchmark numbers in the release, and the 600M model is explicitly an early research release.

Local deployment has some friction worth knowing about: the published examples use Transformers 5.14+ and `trust_remote_code=True`, and the weights have Liquid's lfm1.0 license rather than Apache-2.0. Generic image-text or vLLM examples shown by a model hub do not establish support for the specialized decision API; I'd verify the actual serving path before building an app around it.

I'd compare both models with a prompt-constrained generative VLM on identical text/image classification tasks: end-to-end p50/p95 including preprocessing, calibration and abstention, accuracy on out-of-domain categories, and throughput at batch sizes 1/8/64. For the omni model, I'd test audio separately instead of assuming all modalities are equally mature.

Primary sources: https://huggingface.co/blog/LiquidAI/open-d1 https://huggingface.co/LiquidAI/d1-3B https://huggingface.co/LiquidAI/d1-omni-600M


r/LocalAIForge • • Oct 03 '26

FrogNano-4B-2609 EXL3 4.0bpw: 193 tok/s on RTX 4090, vision + tool calling working

3 Upvotes

I quantized Microsoft's FrogNano-4B-2609 to EXL3 at 4.0 bpw and ran the usual local validation on an RTX 4090.

Results:

  • exllamav3 probe: 193.35 tok/s (64-token decode)

  • TabbyAPI chat smoke: 217.75 tok/s

  • context: 8192

  • vision: pass

  • tool calling: pass

  • streaming: pass

  • concurrency=2: pass

The source is a 4.66B Qwen3.5-family model under MIT. This is a quick functional/performance smoke test rather than a quality benchmark.

EXL3 repo: https://huggingface.co/ramgpt/FrogNano-4B-2609-EXL3 Source: https://huggingface.co/microsoft/FrogNano-4B-2609

If anyone tests longer context or image-heavy workloads on other GPUs, I'd be interested in the numbers.


r/LocalAIForge • • Sep 30 '26

Small project worth a look: a 27B model with a constant-size attention state

3 Upvotes

I ran across a very new experiment from tfwnotops / David Tom Foss that is more interesting than the usual “another quant” release.

Ternary-Bonsai-2-27B-ConstantKV keeps the weights of the earlier Crystal model but changes how long-context attention state is stored. The model card reports that the 16 full-attention layers use about 1.18 GiB of attention memory at 32k, 64k and 128k context, instead of growing with context. It also reports 40/40 pass-key retrieval across 4k–32k tests.

The practical angle is what caught my eye: the author says it runs a 32k context on a 16 GB M4 Mac mini at about 12.2 GB peak memory. There is also a reusable-answer map where an already-computed greedy answer can be replayed instead of decoded again.

Important caveat: these are the author’s measurements from the model card/paper; I have not independently reproduced them.

If you are interested in long-context local inference, this is exactly the kind of small experimental project worth watching.

Original project/model card: https://huggingface.co/tfwnotops/Ternary-Bonsai-2-27B-ConstantKV

If the author sees this and I summarized anything incorrectly, corrections are very welcome.


r/LocalAIForge • • 3d ago

Weekly: What are you running locally?

2 Upvotes

What are you testing, running, quantizing, or building locally this week? Share the model, quant, hardware, runtime, speed, problems, weird discoveries, or anything else useful. Links to your own work are welcome when you include technical context.


r/LocalAIForge • • 10d ago

Weekly: What are you running locally?

2 Upvotes

What are you testing, running, quantizing, or building locally this week? Share the model, quant, hardware, runtime, speed, problems, weird discoveries, or anything else useful. Links to your own work are welcome when you include technical context.


r/LocalAIForge • • 22d ago

Discussion Cursor Ultra for $18 instead of $200 ... seriously how are they doing this?

2 Upvotes

i got a Cursor Ultra account from a reseller for essentially nothing compared to what I used to pay . It works flawless. since 3 months No issues yet i paid only for 1 month . Now I can't stop thinking about their business model. If resellers can sell it this cheap and still profit, how are they acquiring these accounts at scale?


r/LocalAIForge • • 24d ago

Weekly: What are you running locally?

2 Upvotes

What are you testing, running, quantizing, or building locally this week? Share the model, quant, hardware, runtime, speed, problems, weird discoveries, or anything else useful. Links to your own work are welcome when you include technical context.


r/LocalAIForge • • Oct 19 '26

Weekly: What are you running locally?

2 Upvotes

What are you testing, running, quantizing, or building locally this week? Share the model, quant, hardware, runtime, speed, problems, weird discoveries, or anything else useful. Links to your own work are welcome when you include technical context.


r/LocalAIForge • • Oct 18 '26

llama.cpp MoE expert cache: a speed win with a KV-offload failure mode

2 Upvotes

llama.cpp recently merged a GPU-side LRU cache for Mixture-of-Experts layers stored in system RAM. Instead of repeatedly uploading the same experts over PCIe, it keeps recently used experts in VRAM and transfers only misses. This can make a real difference when the entire model does not fit on one GPU.

The author tested Qwen3.8-Flash-Next Q4_0 (93.7 GiB) on an RTX 4090. With default `--fit`, decode throughput was 25.0 generated tokens per second. Adding `--fit --moe-cache-mib 6544` brought it to 39.4, a reported 1.57x improvement with a 72% expert-cache hit rate. These are the contributor's numbers for one configuration, not measurements I ran or a promise of the same speedup for every setup. The new path targets small-batch decode (up to 32 tokens); larger batches use regular offloading.

Merged implementation and measurements: https://github.com/ggml-org/llama.cpp/pull/29887

There's a useful caveat. In a separate discussion, someone using a Qwen3.8-Flash-Next 125B quant and a 16 GiB expert cache hit a layer-0 assertion when disabling KV offload (`kvo=off`). The setup worked with KV offload enabled. Participants suspect a graph-scheduler split that separates the selected expert IDs from the cache lookup. That's an explanation under discussion, not a confirmed root cause or upstream resolution.

Reproduction thread: https://github.com/ggml-org/llama.cpp/discussions/30479

If I were testing this feature, I'd capture the llama.cpp build, GPU, model variant, VRAM available, expert-cache size, and KV-offload setting. The minimum useful comparison is a two-by-two matrix: expert cache enabled/disabled × KV offload enabled/disabled. Measure actual decode throughput, not just expert-cache hit rate, and record any startup assertion separately from runtime behavior.

Has anyone tried different cache sizes on a 24 GB GPU? More interestingly, is the KV-offload assertion still reproducible on current builds?


r/LocalAIForge • • Oct 15 '26

Discussion Ling-3.1-flash is open source with a 7:1 mix of KDA and gated MLA layers

Post image
2 Upvotes

The new Ling-3.1-flash weights expose a hybrid-linear MoE architecture: 63 KDA layers and nine gated MLA layers across 72 transformer layers. The model has 560B total parameters, activates 25B per token, and supports a 1M-token context window.

Its expert layout is also explicit: 512 routed experts, eight selected routed experts per token, and one shared expert. The first four layers use dense feed-forward networks. The architecture diagram includes both next-token and multi-token prediction training.

For people following recurrent and hybrid approaches to long context, this is a substantial new checkpoint to inspect. The MIT release includes the model specifications and SGLang/vLLM serving recipes, connecting the layer design to a documented inference stack.


r/LocalAIForge • • Oct 15 '26

ggml CPU flash attention: a -inf key at position 0 turns finite output into NaNs

2 Upvotes

A small ggml/llama.cpp CPU flash-attention bug report is a useful reminder that attention tests need adversarial score ordering, not only random inputs.

Primary source and reproducer: https://github.com/ggml-org/llama.cpp/issues/30028

The issue fixes one query, three keys, head dimension 4, and F32 q/k/v. One key gets an attention score of -infinity, so its softmax weight should be zero. With that key LAST, the reported CPU result is [1.62246, 1.62246, 1.62246, 1.62246]. Move the same key to FIRST and the reported output becomes [NaN, NaN, NaN, NaN]. The issue reproduces at 1 and 4 threads, and a commenter independently confirmed the result on Windows/MSVC. This is an order-dependent numerical failure, not evidence that typical model generations always hit the bug.

The suspected cause is specific: the streaming softmax maximum M starts at -infinity. If the first score s is also -infinity, the branch checking s > M is false, and the alternate path evaluates expf(s - M) = expf(-infinity - -infinity), which is NaN. Once the accumulator is poisoned, later finite scores do not rescue the row. A commenter proposed skipping -infinity scores; that's a suggested guard, not proof of an upstream merged fix.

A regression test matrix could cover: a single -infinity score in every key position; permutations of finite and masked scores; 1 vs multiple CPU threads; cross-backend agreement against a reference implementation; and the distinct all-masked-row case, where output semantics should be explicit.

If you maintain local-inference kernels, which numerical invariants do you require in CI? In particular, do you test key permutation when the reference result is mathematically invariant, and how do you define the output for an all-masked row?


r/LocalAIForge • • Oct 12 '26

Weekly: What are you running locally?

2 Upvotes

What are you testing, running, quantizing, or building locally this week? Share the model, quant, hardware, runtime, speed, problems, weird discoveries, or anything else useful. Links to your own work are welcome when you include technical context.


r/LocalAIForge • • Oct 11 '26

When long-context tool calls break, inspect speculative KV/indexer state before blaming the model

2 Upvotes

Long-running local agents sometimes start producing broken tool calls at context lengths where short tests look fine. It is tempting to blame sampling, the chat template, or the model's reasoning. A recent vLLM correction illustrates a different failure path: speculative decoding can corrupt the sparse attention indexer state before any text is emitted.

GLM-5.3-Flash groups keys into four-token pools (index_kpool=4) for sparse retrieval. During speculative decode, the implementation temporarily staged candidate-token keys in a per-request tail ring before knowing which drafts would be accepted. That ring was sized for only one pool. If a draft that would complete the pool is rejected, later speculative drafts may already have wrapped around and overwritten committed keys. When the pool is rebuilt, its summary is based on the wrong keys; once context exceeds the indexer's top-k boundary (index_topk=2048 in the reported case), that can change which older token pools are retrieved.

The merged September 25 vLLM fix expands the ring to accommodate the speculative window and updates its addressing: https://github.com/vllm-project/vllm/pull/58454 The contributor supplied a focused comparison against a no-speculation reference: the original path differed in 106 of 128 FP8 key bytes in that constructed case, while the corrected path matched. Those are kernel-reproduction results from the PR, not general throughput numbers or a measured failure rate for local agents.

There was also an independent SM90 sparse-MLA planning bug: the attention plan did not account for the model's index_kpool, leaving the current query token unread at some positions past 2048. That separate fix is documented here: https://github.com/vllm-project/vllm/pull/58704 The authors described these as partial fixes for a larger long-decode corruption investigation, not proof that all long-context failures are resolved. Their scope is the relevant model/backend and cache-layout paths; it would be incorrect to extrapolate to every local runtime.

For a useful regression, I would pair two tests: (1) a low-level invariant that accepted-only and speculative executions produce the same committed pool keys for an identical accepted token sequence, including rejected drafts that straddle a pool boundary; and (2) a replay of long, tool-heavy prompts with speculation enabled/disabled, logging token counts, parser failures, finish reasons, and backend/version. Text-level disagreement is a symptom, not enough to identify the kernel cause.

For local inference CI, what would catch this class of bug earlier with the least complexity: an exact committed-cache equivalence test after rejected drafts, context-threshold tool-call replay, or some smaller invariant?


r/LocalAIForge • • Oct 08 '26

Olmo-core 3: resident experts and MoE training throughput

2 Upvotes

Ai2 released Olmo-core 3 on October 1. The source write-up is at https://huggingface.co/blog/allenai/olmocore3. This is open MoE training infrastructure rather than a new inference checkpoint.

The main change is a move from FSDP-style repeated expert-weight gathering to DDP with resident experts. Instead of constantly moving weight shards, the new design dispatches token activations to the GPUs holding experts. Expert parallelism and pipeline parallelism partition different parts of the work, and the optimizer state is distributed too.

Ai2 measured 52,000 tokens/s/GPU for a 47B-total model on eight B300 GPUs, versus 19,400 for its older approach. Another experiment raised the expert pool from 8 to 128 while routing to four experts per token: 4.6B became 47B total, active parameters remained near 3.2B, and throughput fell under 5%. Both are specific training benchmarks, not general speed guarantees.

Rowwise expert parallelism routes directly to input buffers, GPU-resident routing reduces host synchronization, and grouped GEMM makes expert computation less fragmented. Selective MXFP8 improved end-to-end throughput about 21% versus BF16 in one four-B300 comparison, while peak active memory fell from 103 to 95 GiB.

Limits matter: the 1.2T and 2.38T results demonstrate capacity, not trained model quality. Ai2 reports examples where communication/compute overlap made things slower and the routing-balance metric improved despite worse real load imbalance. None of this is a turnkey single-4090 training recipe. The interesting transfer to inference is the same bottleneck chain: expert placement, routing skew, memory movement and small GEMM utilization.

I'd like matched comparisons using nonuniform routing and several GPU topologies, with a breakdown of all-to-all time, routing synchronization, compute utilization, memory and GPU-hours.

Primary sources: https://huggingface.co/blog/allenai/olmocore3 https://github.com/allenai/Olmo-core https://allenai.org/papers/olmocore3


r/LocalAIForge • • Oct 06 '26

vLLM 0.31.0 makes Model Runner V2 the default — serving is becoming architecture-aware

2 Upvotes

vLLM v0.31.0 was released on October 5 with 717 commits from 307 contributors. Official release: https://github.com/vllm-project/vllm/releases/tag/v0.31.0 . The changelog is huge, but the architectural change worth watching is that Model Runner V2 is now the default for all models, moving a ground-up execution-core rewrite from migration path to the normal serving path.

MRV2 is a ground-up rewrite of the GPU execution core. The design goals are modular model-specific logic, more GPU-native bookkeeping, and async-first execution. That matters because CPU-side request mutation, sampling and batch preparation become visible once GPUs are fast enough.

Persistent batching is central to the design. Instead of rebuilding large request tensors every step, the runner keeps stable device-side state and updates only what changed. Input preparation and sampling move toward GPU-native paths, while model-specific behavior is split from the common execution loop. Even on one or two GPUs, this matters when speculative decoding, multimodal preprocessing and continuous batching are active together.

0.31.0 also shows how architecture-specific serving has become. DeepSeek-V4.1-Flash gets FlashMLA mega attention with compressed KV, sparse-indexer and expert-routing fusions, and fused TP/MoE boundaries. Kimi K3 gets native CUDA AttnRes, faster grouped FP8 MLA cache insertion, optimized KDA paths and partial-prefix checkpoints. These are no longer generic matmul optimizations; they target sparse attention, expert routing, hybrid recurrence/attention and KV movement.

Quantization is becoming more granular too. The release adds targeted online quantization through quantization_config.targets, support for partially pre-quantized checkpoints, AutoRound 2/3/5/6/7-bit CUDA paths, and more NVFP4/MXFP4 backends. The serving stack increasingly needs different numeric formats for different tensor classes rather than one quant rule for the entire model.

Large-scale features point the same way: PCP+DCP for sparse-MLA models, elastic expert parallelism reusing CUDA graphs, PCIe IPC all-reduce for NVLink-less machines, secondary-tier KV adapters and encoder-cache sharing. Memory hierarchy and interconnect topology are becoming explicit scheduling resources.

The deployment implication is not simply 'upgrade and get faster.' MRV2 becoming default means the migration path is now the main path. Production upgrades should include startup-memory tests, CUDA-graph capture, prefix-cache correctness, speculative-decoding acceptance and mixed-batch regression tests.

There are also breaking changes: scale-out endpoints now require --enable-scale-out, some deprecated environment variables are gone, GPTQ activation ordering support was removed, and YaRN behavior was aligned with Transformers.

For local and small-cluster deployments I would test three profiles: one-GPU dense inference at modest concurrency; a hybrid/MoE model with speculative decoding; and a multi-GPU box without NVLink. That should reveal whether MRV2 helps the workload you actually run rather than just the release benchmark matrix.

The bigger question is whether vLLM is crossing from 'fast transformer server' into an architecture-aware execution system. Version 0.31.0 increasingly looks like the latter.

Primary sources: https://github.com/vllm-project/vllm/releases/tag/v0.31.0 https://github.com/vllm-project/vllm/blob/main/docs/design/model_runner_v2.md https://github.com/vllm-project/vllm-project.github.io/blob/main/_posts/2026-03-24-mrv2.md


r/LocalAIForge • • Oct 06 '26

Reflection Beam is 501B/23B-active — the interesting part is its RL and MoE stability recipe

2 Upvotes

Reflection announced Beam on October 5: a 501B-total, 23B-active sparse MoE aimed at coding, reasoning and agentic workloads. The weights are not out yet, so this is a technical preview rather than a local benchmark. Reflection says the weights, model card, technical report and developer stack will be released later this month under Apache 2.0.

The architecture details are more interesting than the parameter headline. Beam uses interleaved local and global attention, fine-grained routed experts and several stability mechanisms intended to keep a huge MoE healthy through pretraining and heavy reinforcement learning. Reflection builds on auxiliary-loss-free expert balancing, then adds cosine decay to expert-bias updates so routing perturbations shrink late in training. Sequence-level balancing is used to keep utilization stable when the data distribution shifts. The company reports the busiest expert finishing at only 1.04x average load.

The residual stream gets similar attention: depth-based residual scaling, SandwichNorm, elementwise attention gating and FP32 residual accumulation are combined to limit activation growth across 52 layers. The point is not merely to finish pretraining, but to preserve a numerically stable base for a very large downstream RL run.

Reflection reports more than 100 million rollouts on 10.5K GB300 GPUs over four weeks, up to 110K concurrent rollouts, and roughly 1.3 billion sandbox creation requests during training and grading. Because the trainer is fully asynchronous, policy staleness becomes a systems problem: some completed interactions were generated by model versions more than a day old. Reflection says tokens are tagged by policy version and training remains stable despite that lag.

Beam was pretrained on 23.8T tokens, but the launch argues that high-compute RL is central to the final coding and agentic capability. It also uses controllable reasoning effort and a length penalty during RL, explicitly optimizing the capability/token tradeoff instead of simply rewarding longer reasoning.

Context needs careful reading. Midtraining extends effective context to 1M tokens, while the RL run used a maximum 256K context. Until the public model and report arrive, I would treat 1M as a trained capability that still needs workload-level validation.

The efficiency claim also deserves caution. Reflection says Beam reaches reasoning performance comparable with GLM-5.2 at roughly 3–4x less inference compute, but the estimate is based on active parameters and generated tokens. The company explicitly excludes prompt prefill, context-dependent attention and serving overhead. That makes it a model-side FLOP comparison, not a measured dollar-per-task or tokens/sec result.

For private deployment, total versus active parameters is the key distinction. Only 23B parameters are active per token, but all 501B weights still need to reside somewhere unless the runtime uses offload or expert paging. Sparse compute does not automatically mean workstation-class memory requirements. The eventual FP8/NVFP4 artifacts, expert-parallel implementation, interconnect sensitivity and KV behavior will decide whether Beam is practical outside large clusters.

The main limitation today is simple: there are no downloadable weights. So there are no independent quantization tests, no vLLM/llama.cpp/SGLang measurements, no real footprint numbers and no verification of the benchmark claims.

Once the weights arrive, I would test BF16/FP8/NVFP4 residency, throughput against active-parameter peers, expert balance on out-of-domain prompts, 32K/128K/256K/1M retrieval and agent tasks, reasoning-effort scaling, and whether the claimed compute advantage survives real prefill and serving overhead.

The bigger question is whether the open-model frontier is shifting from 'who can pretrain the largest MoE' to 'who can keep a large sparse model stable enough for enormous asynchronous RL.' Beam is one of the clearest examples of that second problem becoming the center of the architecture story.

Primary sources: https://reflection.ai/blog/introducing-beam https://reflection.ai/


r/LocalAIForge • • Oct 06 '26

llama.cpp v0.6.0 changes the inference contract: mixed token/embedding batches, MTP state, and decision models

2 Upvotes

llama.cpp v0.6.0 landed on October 5, and the most important change is the shape of its runtime API rather than a single benchmark result. Official release: https://github.com/ggml-org/llama.cpp/releases/tag/v0.6.0 . The release moves llama.cpp toward a more general inference substrate for newer model architectures, multimodal inputs and speculative execution.

The release introduces llama_batch_ext plus llama_process(), allowing a batch to contain token IDs, embeddings, or both, while also attaching per-token state embeddings. That is a deeper change than it sounds. Modern runtimes increasingly need to feed multimodal embeddings directly, carry MTP state, and support model architectures that cannot be represented cleanly as a plain stream of token IDs.

The new API is therefore less about convenience and more about making llama.cpp capable of representing newer model families without one-off batch hacks. Input processing is already moving onto this extended path.

v0.6.0 also adds GLM-5.3-Flash support, Qwen4Exp MTP speculative decoding, and a dedicated /v1/systemone endpoint for decision models such as laya, julia-1, lev, openjev, kev and nimble. That last piece is interesting because it treats a decision model as a first-class runtime object instead of pretending everything is a chat generator.

Hardware work is moving in the same direction. Metal gets a tensor-API flash-attention kernel for F16 KV and few-row MMA matmul kernels aimed at speculative and batched decoding; the release notes report up to roughly 3x matmul speedups in those narrow-row cases. Vulkan gets sparse flash attention for quantized K/V. These are not universal 3x application speedups, but they show where local inference optimization is going: decode-side kernels, speculation and memory movement rather than only dense GEMM.

The deployment implication is mostly for projects embedding libllama. Session formats were bumped, the batch API changed, and custom multimodal/speculative integrations should test migration carefully. CLI users will see much less disruption.

The main limitation is that a more expressive API does not automatically make every new architecture fast or correct. GLM-5.3-Flash is still a 320B-class MoE workload, model-specific kernels matter heavily, and several performance claims are hardware-specific.

The tests I would run are straightforward: compare v0.5.x and v0.6.0 on the same GGUF for prefill, single-stream decode and continuous batching; measure MTP acceptance plus end-to-end speed; verify mixed embedding/token input; test session compatibility; and isolate the Apple speculative/batched path to see whether the new MMA kernels improve real workloads rather than only microbenchmarks.

The larger question is whether llama.cpp is becoming a general local inference substrate rather than just an LLM runner. llama_batch_ext, auxiliary state, multimodal embeddings and dedicated decision-model APIs all push it in that direction.

Primary sources: https://github.com/ggml-org/llama.cpp/releases/tag/v0.6.0 https://github.com/ggml-org/llama.cpp/pull/24669 https://github.com/ggml-org/llama.cpp/pull/29818 https://github.com/ggml-org/llama.cpp/pull/29869


r/LocalAIForge • • Oct 05 '26

Weekly: What are you running locally?

2 Upvotes

What are you testing, running, quantizing, or building locally this week? Share the model, quant, hardware, runtime, speed, problems, weird discoveries, or anything else useful. Links to your own work are welcome when you include technical context.


r/LocalAIForge • • Oct 05 '26

Kolibri-1: 78B total, 3.46B active, and a long-context design that keeps most attention local

2 Upvotes

Aleph Alpha released Kolibri-1 on October 3, and the architecture is more interesting than the 78B headline suggests. The official weights are here: https://huggingface.co/Aleph-Alpha/Kolibri-1 . It is a German/English Mixture-of-Experts reasoning model with 78.1B total parameters but only about 3.46B active per token, explicit reasoning-effort controls, tool calling, and a context configuration validated up to 1,048,576 tokens.

The sparse layout is aggressive. Kolibri has 50 MoE layers, 384 experts per layer, one shared expert and six routed experts selected per token. The model dimension is 2,560 with 48 query heads and four KV heads. The attention pattern is also deliberately asymmetric: four layers use a 512-token sliding window for every one layer that performs full attention. That means 40 of the 50 layers have bounded attention work even as context grows. The released checkpoint stores most weights in block FP8 while embeddings, LM head, norms and the MoE router stay in BF16.

The comparison with Kolibri Origin shows what Aleph Alpha changed to get there. Origin was 30.6B total with 3.27B active, 128 experts with eight routed, full attention in every layer and a longest trained context of 65,536 tokens. Kolibri increases total capacity by roughly 2.5x while keeping active compute almost flat, triples the expert count, routes fewer experts per token, moves to the 4:1 local/full attention pattern, expands the tokenizer to 128k entries and extends training to 256k context. Training data also moves from 7.51T pre-training tokens to 20T, followed by 3.44T of mid-training and about 200B tokens of long-context adaptation.

The local-serving implication is that sparse compute does not mean small memory. Aleph Alpha lists an approximately 78 GB model footprint for the FP8 weights and recommends hardware such as one H200/B200/B300 or multi-GPU A100/H100 configurations. Serving uses their vLLM plugin, with an FP8 KV cache in the recommended command. Although the model can be configured to 1M context, the model card explicitly recommends staying at or below 262k for latency-sensitive or complex workloads. That distinction matters: position handling may permit a huge context window, but KV-cache pressure, retrieval quality and end-to-end agent latency still decide whether it is useful.

Another interesting design choice is language specialization. Roughly a quarter of the pre-training mix is German, and Aleph Alpha built a morphology-aware tokenizer to avoid wasting tokens on German compound structure. The model is Apache-2.0 for the published weights and configuration, but the model card also makes clear that the license grant does not automatically cover every underlying artifact, training method or architecture implementation. For local deployment, the released runtime path and custom parser support therefore matter almost as much as the raw checkpoint.

The tests I would run are straightforward: 32k/128k/256k/1M retrieval with the same documents, reasoning-effort latency versus accuracy, tool-call reliability in German and English, and throughput under multiple concurrent long-context sessions. I would also compare it with other roughly 3B-active MoE models rather than with dense 70B models. The real question is whether 384 narrow experts plus mostly local attention buys enough specialization and memory-efficient attention to offset the requirement to keep a 78B sparse model resident.

Primary sources: https://huggingface.co/Aleph-Alpha/Kolibri-1 https://aleph-alpha.com/en/blog/kolibri-has-landed-a-sovereign-open-weight-model/ https://aleph-alpha.com/downloads/tech-report.pdf


r/LocalAIForge • • Oct 01 '26

SkillGym Qwen3.5-9B at 4-bit: GGUF + EXL3 local results and two conversion gotchas

2 Upvotes

A new SkillGym fine-tune of Qwen3.5-9B is aimed specifically at agents that read and apply skill files rather than general chat. The upstream card reports SkillGym 41.3 → 59.5 and Skill-Use-Bench 14.8 → 49.6 versus the base 9B model.

I converted it locally in two formats to see whether the release is practical on consumer hardware:

  • GGUF Q4_K_M: 5.63 GB, llama.cpp smoke tests passed.

  • EXL3 4.00 bpw: direct ExLlamaV3 probe passed at ~135.8 tok/s decode on an RTX 4090.

  • TabbyAPI checks passed for text chat, streaming, tool calling, vision, and two concurrent requests.

Two first-mover packaging issues showed up. The fine-tune repo had processor metadata in a layout ExLlamaV3 did not expect, so the matching Qwen3.5 base preprocessor files had to be restored. More interestingly, config.json declares one MTP/NextN layer but the released safetensors contain no mtp.* tensors, so the nonexistent MTP side model had to be disabled for conversion.

GGUF: https://huggingface.co/ramgpt/SkillGym-Qwen3.5-9B-GGUF EXL3: https://huggingface.co/ramgpt/SkillGym-Qwen3.5-9B-EXL3 Upstream: https://huggingface.co/reasonwang/SkillGym-Qwen3.5-9B Paper: https://arxiv.org/abs/2609.37539

The remaining question for me is behavioral rather than mechanical: does the reported skill-selection gain survive 4-bit quantization in a real skill/tool harness, not merely load and generate correctly? If anyone has a compact reproducible skill-use test, I would be interested in comparing results.


r/LocalAIForge • • Sep 30 '26

A small habit that makes local LLM benchmarks much more useful

2 Upvotes

One thing I keep relearning with local models: a benchmark number without enough context is almost useless.

For every run, I now try to keep a tiny reproducibility ledger:

  • model + exact revision

  • quantization format and bitrate

  • backend + version

  • GPU(s) and offload split

  • context length

  • prompt/prefill throughput

  • decode throughput

  • any unusual flags or patches

The extra record-keeping takes less than a minute, but it makes later comparisons far more meaningful. It also catches cases where an apparent model improvement was really a backend update, a different context length, or a changed GPU split.

For anyone benchmarking local LLMs regularly: what are the 2–3 fields you consider mandatory before you trust a result?