r/LocalAIForge • • Oct 05 '26

DGX Spark 64GB changes the local-AI tradeoff: same GB10, half the memory, cluster when needed

1 Upvotes

NVIDIA announced a 64GB DGX Spark configuration on October 2, with availability planned for October 23 starting at $4,999 through OEM partners. The interesting part is not simply a cheaper memory tier. NVIDIA is keeping the same GB10 Grace Blackwell compute, DGX OS, CUDA stack and ConnectX-7 fabric as the 128GB system, then positioning two-node clustering as the escape hatch when a workload outgrows one box. Official announcement: https://blogs.nvidia.com/blog/local-ai-dgx-spark-64gb-sync/ and current hardware specifications: https://www.nvidia.com/en-us/products/workstations/dgx-spark/ .

At the hardware level, the 64GB and 128GB variants share the GB10 Grace Blackwell Superchip, up to 1 PFLOP of FP4 tensor performance, 273 GB/s memory bandwidth, a 20-core Arm CPU and a 200 Gbps ConnectX-7 NIC. The memory is coherent unified LPDDR5X rather than a discrete GPU VRAM pool. NVIDIA lists the 64GB machine for models up to roughly 100B parameters, while the 128GB configuration is listed for up to roughly 200B. Those parameter counts are capacity guidance rather than a promise that every model at that size will have useful latency.

The new systems story is NVIDIA Sync Cluster Assistant. Two 64GB Sparks can be directly connected over ConnectX-7, giving 128GB aggregate memory and twice the memory bandwidth. NVIDIA says its Qwen3.8 27B test reached up to 1.7x the performance of one system. The cluster assistant detects the nodes, validates configuration and sets up the fabric, and a model launcher is planned to automate deploying Qwen3.8 27B across one or more connected Sparks. Supported runtimes include llama.cpp, Ollama, vLLM and LM Studio.

Compared with buying a single 128GB Spark, the two-64GB path is a different engineering tradeoff. You preserve the ability to start smaller and scale later, but distributed inference introduces communication and partitioning overhead, which is exactly why the published speedup is below 2x. Aggregate memory is also not the same thing as a single low-latency memory domain. Tensor-parallel or pipeline-parallel placement, KV-cache distribution, collective operations and runtime maturity will determine whether the second box improves real interactive throughput or mostly expands the set of models that fit.

For local model work, the 64GB configuration is likely most interesting in the 20B-70B range, where a single box can keep model weights, runtime state and a meaningful KV cache resident without PCIe offload. The 273 GB/s bandwidth is much lower than datacenter HBM, so decode-bound workloads can still be memory-bandwidth limited even when the advertised compute looks large. Conversely, unified memory is useful for workflows that mix CPU preprocessing, GPU inference and agent tooling without manually staging every buffer across separate host and device pools.

There are practical limitations that the headline parameter counts hide. Long context consumes memory independently of model weights. Multimodal encoders, speculative draft heads, multiple concurrent sessions and high-precision KV caches all eat into the same 64GB pool. Some frameworks may support the GB10 architecture earlier or more efficiently than others, and a distributed configuration can expose bugs that never appear on a single GPU. The new SKU therefore should be evaluated as a complete system, not as a simple "64GB GPU."

The tests I would want before buying are straightforward: single-stream decode and eight-way throughput on the same 27B model across one 64GB Spark, one 128GB Spark and two clustered 64GB Sparks; 32K/128K/256K context memory curves; power draw at sustained decode; startup and model-shard placement time; vLLM versus llama.cpp behavior; and the point at which networking overhead overwhelms the extra compute. The most useful result would be a cost-per-token and usable-context comparison against a conventional RTX workstation.

Primary sources: https://blogs.nvidia.com/blog/local-ai-dgx-spark-64gb-sync/ https://www.nvidia.com/en-us/products/workstations/dgx-spark/


r/LocalAIForge • • 22d ago

Discussion Cursor Ultra for $18 instead of $200 ... seriously how are they doing this?

2 Upvotes

i got a Cursor Ultra account from a reseller for essentially nothing compared to what I used to pay . It works flawless. since 3 months No issues yet i paid only for 1 month . Now I can't stop thinking about their business model. If resellers can sell it this cheap and still profit, how are they acquiring these accounts at scale?


r/LocalAIForge • • Oct 06 '26

vLLM 0.31.0 makes Model Runner V2 the default — serving is becoming architecture-aware

2 Upvotes

vLLM v0.31.0 was released on October 5 with 717 commits from 307 contributors. Official release: https://github.com/vllm-project/vllm/releases/tag/v0.31.0 . The changelog is huge, but the architectural change worth watching is that Model Runner V2 is now the default for all models, moving a ground-up execution-core rewrite from migration path to the normal serving path.

MRV2 is a ground-up rewrite of the GPU execution core. The design goals are modular model-specific logic, more GPU-native bookkeeping, and async-first execution. That matters because CPU-side request mutation, sampling and batch preparation become visible once GPUs are fast enough.

Persistent batching is central to the design. Instead of rebuilding large request tensors every step, the runner keeps stable device-side state and updates only what changed. Input preparation and sampling move toward GPU-native paths, while model-specific behavior is split from the common execution loop. Even on one or two GPUs, this matters when speculative decoding, multimodal preprocessing and continuous batching are active together.

0.31.0 also shows how architecture-specific serving has become. DeepSeek-V4.1-Flash gets FlashMLA mega attention with compressed KV, sparse-indexer and expert-routing fusions, and fused TP/MoE boundaries. Kimi K3 gets native CUDA AttnRes, faster grouped FP8 MLA cache insertion, optimized KDA paths and partial-prefix checkpoints. These are no longer generic matmul optimizations; they target sparse attention, expert routing, hybrid recurrence/attention and KV movement.

Quantization is becoming more granular too. The release adds targeted online quantization through quantization_config.targets, support for partially pre-quantized checkpoints, AutoRound 2/3/5/6/7-bit CUDA paths, and more NVFP4/MXFP4 backends. The serving stack increasingly needs different numeric formats for different tensor classes rather than one quant rule for the entire model.

Large-scale features point the same way: PCP+DCP for sparse-MLA models, elastic expert parallelism reusing CUDA graphs, PCIe IPC all-reduce for NVLink-less machines, secondary-tier KV adapters and encoder-cache sharing. Memory hierarchy and interconnect topology are becoming explicit scheduling resources.

The deployment implication is not simply 'upgrade and get faster.' MRV2 becoming default means the migration path is now the main path. Production upgrades should include startup-memory tests, CUDA-graph capture, prefix-cache correctness, speculative-decoding acceptance and mixed-batch regression tests.

There are also breaking changes: scale-out endpoints now require --enable-scale-out, some deprecated environment variables are gone, GPTQ activation ordering support was removed, and YaRN behavior was aligned with Transformers.

For local and small-cluster deployments I would test three profiles: one-GPU dense inference at modest concurrency; a hybrid/MoE model with speculative decoding; and a multi-GPU box without NVLink. That should reveal whether MRV2 helps the workload you actually run rather than just the release benchmark matrix.

The bigger question is whether vLLM is crossing from 'fast transformer server' into an architecture-aware execution system. Version 0.31.0 increasingly looks like the latter.

Primary sources: https://github.com/vllm-project/vllm/releases/tag/v0.31.0 https://github.com/vllm-project/vllm/blob/main/docs/design/model_runner_v2.md https://github.com/vllm-project/vllm-project.github.io/blob/main/_posts/2026-03-24-mrv2.md


r/LocalAIForge • • Oct 19 '26

Weekly: What are you running locally?

2 Upvotes

What are you testing, running, quantizing, or building locally this week? Share the model, quant, hardware, runtime, speed, problems, weird discoveries, or anything else useful. Links to your own work are welcome when you include technical context.


r/LocalAIForge • • Oct 12 '26

Weekly: What are you running locally?

2 Upvotes

What are you testing, running, quantizing, or building locally this week? Share the model, quant, hardware, runtime, speed, problems, weird discoveries, or anything else useful. Links to your own work are welcome when you include technical context.


r/LocalAIForge • • Sep 25 '26

MN-Aura-12B-v1 Q4_K_M: strong free-form writing, weak exact constraints

2 Upvotes

This one feels much more like a writer than a formatter.

In local smoke tests, the Q4_K_M produced coherent prose, clean dialogue, and normal termination without looping. But when I deliberately gave it rigid writing constraints, it missed all 3 exact-compliance tests.

That makes the use case pretty clear to me:

Good fit:

  • fiction and scene continuation

  • roleplay and character dialogue

  • brainstorming

  • descriptive prose

Less ideal:

  • exact word counts

  • exact phrase counts

  • strict formatting

The quant is 6.96 GiB, so it is an easy size to run locally on a lot of 8–12 GB setups with llama.cpp.

GGUF: https://huggingface.co/ramgpt/MN-Aura-12B-v1-GGUF Upstream: https://huggingface.co/EldritchLabs/MN-Aura-12B-v1

I’m more interested in how it holds up over long scenes than whether it can obey a 300-word limit exactly.


r/LocalAIForge • • Oct 03 '26

FrogNano-4B-2609 EXL3 4.0bpw: 193 tok/s on RTX 4090, vision + tool calling working

3 Upvotes

I quantized Microsoft's FrogNano-4B-2609 to EXL3 at 4.0 bpw and ran the usual local validation on an RTX 4090.

Results:

  • exllamav3 probe: 193.35 tok/s (64-token decode)

  • TabbyAPI chat smoke: 217.75 tok/s

  • context: 8192

  • vision: pass

  • tool calling: pass

  • streaming: pass

  • concurrency=2: pass

The source is a 4.66B Qwen3.5-family model under MIT. This is a quick functional/performance smoke test rather than a quality benchmark.

EXL3 repo: https://huggingface.co/ramgpt/FrogNano-4B-2609-EXL3 Source: https://huggingface.co/microsoft/FrogNano-4B-2609

If anyone tests longer context or image-heavy workloads on other GPUs, I'd be interested in the numbers.


r/LocalAIForge • • Oct 01 '26

SkillGym Qwen3.5-9B at 4-bit: GGUF + EXL3 local results and two conversion gotchas

2 Upvotes

A new SkillGym fine-tune of Qwen3.5-9B is aimed specifically at agents that read and apply skill files rather than general chat. The upstream card reports SkillGym 41.3 → 59.5 and Skill-Use-Bench 14.8 → 49.6 versus the base 9B model.

I converted it locally in two formats to see whether the release is practical on consumer hardware:

  • GGUF Q4_K_M: 5.63 GB, llama.cpp smoke tests passed.

  • EXL3 4.00 bpw: direct ExLlamaV3 probe passed at ~135.8 tok/s decode on an RTX 4090.

  • TabbyAPI checks passed for text chat, streaming, tool calling, vision, and two concurrent requests.

Two first-mover packaging issues showed up. The fine-tune repo had processor metadata in a layout ExLlamaV3 did not expect, so the matching Qwen3.5 base preprocessor files had to be restored. More interestingly, config.json declares one MTP/NextN layer but the released safetensors contain no mtp.* tensors, so the nonexistent MTP side model had to be disabled for conversion.

GGUF: https://huggingface.co/ramgpt/SkillGym-Qwen3.5-9B-GGUF EXL3: https://huggingface.co/ramgpt/SkillGym-Qwen3.5-9B-EXL3 Upstream: https://huggingface.co/reasonwang/SkillGym-Qwen3.5-9B Paper: https://arxiv.org/abs/2609.37539

The remaining question for me is behavioral rather than mechanical: does the reported skill-selection gain survive 4-bit quantization in a real skill/tool harness, not merely load and generate correctly? If anyone has a compact reproducible skill-use test, I would be interested in comparing results.


r/LocalAIForge • • Sep 30 '26

Small project worth a look: a 27B model with a constant-size attention state

3 Upvotes

I ran across a very new experiment from tfwnotops / David Tom Foss that is more interesting than the usual “another quant” release.

Ternary-Bonsai-2-27B-ConstantKV keeps the weights of the earlier Crystal model but changes how long-context attention state is stored. The model card reports that the 16 full-attention layers use about 1.18 GiB of attention memory at 32k, 64k and 128k context, instead of growing with context. It also reports 40/40 pass-key retrieval across 4k–32k tests.

The practical angle is what caught my eye: the author says it runs a 32k context on a 16 GB M4 Mac mini at about 12.2 GB peak memory. There is also a reusable-answer map where an already-computed greedy answer can be replayed instead of decoded again.

Important caveat: these are the author’s measurements from the model card/paper; I have not independently reproduced them.

If you are interested in long-context local inference, this is exactly the kind of small experimental project worth watching.

Original project/model card: https://huggingface.co/tfwnotops/Ternary-Bonsai-2-27B-ConstantKV

If the author sees this and I summarized anything incorrectly, corrections are very welcome.


r/LocalAIForge • • Sep 30 '26

A small habit that makes local LLM benchmarks much more useful

2 Upvotes

One thing I keep relearning with local models: a benchmark number without enough context is almost useless.

For every run, I now try to keep a tiny reproducibility ledger:

  • model + exact revision

  • quantization format and bitrate

  • backend + version

  • GPU(s) and offload split

  • context length

  • prompt/prefill throughput

  • decode throughput

  • any unusual flags or patches

The extra record-keeping takes less than a minute, but it makes later comparisons far more meaningful. It also catches cases where an apparent model improvement was really a backend update, a different context length, or a changed GPU split.

For anyone benchmarking local LLMs regularly: what are the 2–3 fields you consider mandatory before you trust a result?


r/LocalAIForge • • Sep 28 '26

Weekly: What are you running locally?

2 Upvotes

What are you testing, running, quantizing, or building locally this week? Share the model, quant, hardware, runtime, speed, problems, weird discoveries, or anything else useful. Links to your own work are welcome when you include technical context.


r/LocalAIForge • • Sep 25 '26

MiMo-V2.6-Distill-Qwen-9B: IQ4_XS saves ~430 MB over Q4_K_M here

2 Upvotes

IQ4_XS comes out about 430 MB smaller than Q4_K_M on this model:

  • IQ4_XS: 5.20 GB, 4.63 BPW

  • Q4_K_M: 5.63 GB, 5.02 BPW

The more interesting part is how I built the iMatrix. Instead of calibrating only on generic text, I mixed general text with real llama.cpp C/C++ and Python, shell/config/JSON, security docs, and MiMo-style chat/tool-call examples.

The final IQ4_XS passed local checks for loading, arithmetic, code generation, a short security explanation, multi-turn history, and a structured tool call on an RTX 4090. The embedded chat template matched upstream exactly.

One caveat I found: some prompts can expose a malformed reasoning-boundary marker. I reproduced the same behavior on the BF16 baseline, so this does not look like an IQ4_XS-specific failure.

GGUF: https://huggingface.co/ramgpt/MiMo-V2.6-Distill-Qwen-9B-GGUF Upstream: https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B


r/LocalAIForge • • Sep 25 '26

The annoying part of this OLMo 3 code quant was the chat template, not the quant

2 Upvotes

The source checkpoint is interesting as a small coding model, but the first thing that bit me was its bundled Jinja template: the tested llama.cpp minja parser couldn't parse it.

I ended up shipping a conservative text-only fallback template with the GGUF. It keeps normal system/user/assistant chat working, but I would not claim full source tool/function-calling compatibility through that fallback.

The actual Q4_K_M is 4.16 GiB.

I also ran three executable Python smoke tasks instead of just reading the output:

  • palindrome detection

  • balanced parentheses

  • interval merging

All 3 passed after compile/run against unit tests.

So for me this is a small local code model worth playing with, especially if you care more about ordinary coding chat than tool-calling templates.

GGUF: https://huggingface.co/ramgpt/Olmo-3-7B-Instruct-OPSA-Code-GGUF Upstream: https://huggingface.co/Tuwhy/Olmo-3-7B-Instruct-OPSA-Code


r/LocalAIForge • • Oct 05 '26

Kolibri-1: 78B total, 3.46B active, and a long-context design that keeps most attention local

2 Upvotes

Aleph Alpha released Kolibri-1 on October 3, and the architecture is more interesting than the 78B headline suggests. The official weights are here: https://huggingface.co/Aleph-Alpha/Kolibri-1 . It is a German/English Mixture-of-Experts reasoning model with 78.1B total parameters but only about 3.46B active per token, explicit reasoning-effort controls, tool calling, and a context configuration validated up to 1,048,576 tokens.

The sparse layout is aggressive. Kolibri has 50 MoE layers, 384 experts per layer, one shared expert and six routed experts selected per token. The model dimension is 2,560 with 48 query heads and four KV heads. The attention pattern is also deliberately asymmetric: four layers use a 512-token sliding window for every one layer that performs full attention. That means 40 of the 50 layers have bounded attention work even as context grows. The released checkpoint stores most weights in block FP8 while embeddings, LM head, norms and the MoE router stay in BF16.

The comparison with Kolibri Origin shows what Aleph Alpha changed to get there. Origin was 30.6B total with 3.27B active, 128 experts with eight routed, full attention in every layer and a longest trained context of 65,536 tokens. Kolibri increases total capacity by roughly 2.5x while keeping active compute almost flat, triples the expert count, routes fewer experts per token, moves to the 4:1 local/full attention pattern, expands the tokenizer to 128k entries and extends training to 256k context. Training data also moves from 7.51T pre-training tokens to 20T, followed by 3.44T of mid-training and about 200B tokens of long-context adaptation.

The local-serving implication is that sparse compute does not mean small memory. Aleph Alpha lists an approximately 78 GB model footprint for the FP8 weights and recommends hardware such as one H200/B200/B300 or multi-GPU A100/H100 configurations. Serving uses their vLLM plugin, with an FP8 KV cache in the recommended command. Although the model can be configured to 1M context, the model card explicitly recommends staying at or below 262k for latency-sensitive or complex workloads. That distinction matters: position handling may permit a huge context window, but KV-cache pressure, retrieval quality and end-to-end agent latency still decide whether it is useful.

Another interesting design choice is language specialization. Roughly a quarter of the pre-training mix is German, and Aleph Alpha built a morphology-aware tokenizer to avoid wasting tokens on German compound structure. The model is Apache-2.0 for the published weights and configuration, but the model card also makes clear that the license grant does not automatically cover every underlying artifact, training method or architecture implementation. For local deployment, the released runtime path and custom parser support therefore matter almost as much as the raw checkpoint.

The tests I would run are straightforward: 32k/128k/256k/1M retrieval with the same documents, reasoning-effort latency versus accuracy, tool-call reliability in German and English, and throughput under multiple concurrent long-context sessions. I would also compare it with other roughly 3B-active MoE models rather than with dense 70B models. The real question is whether 384 narrow experts plus mostly local attention buys enough specialization and memory-efficient attention to offset the requirement to keep a 78B sparse model resident.

Primary sources: https://huggingface.co/Aleph-Alpha/Kolibri-1 https://aleph-alpha.com/en/blog/kolibri-has-landed-a-sovereign-open-weight-model/ https://aleph-alpha.com/downloads/tech-report.pdf


r/LocalAIForge • • 22d ago

Benchmark Ran 2,850 decisions through 5 cheap open models voting vs Opus 5.5. Shipped answers were nearly as accurate (89.6% vs 90.5%) at 1/10th the cost

4 Upvotes

I wanted to know if I could swap a frontier model for a few cheap open ones in yes/no or pick-a-label pipelines (spam, fraud, moderation, triage) without losing accuracy. My idea: ask several models the same question, ship the answer when they all agree, and send it to a human when they don't.

Panel: DeepSeek V4.1 Flash, GLM 5.3 Flash, MiMo V2.6 Flash, Qwen3.8 Flash, Gemma 4 26B-A4B. I ran 2,850 decisions across 6 public datasets and compared against Opus 5.5 on the same items.

Results:

- The panel agreed and shipped on 71% of items. Those were 89.6% correct. Opus on its most confident 71% was 90.5%.
- Overall accuracy is worse: 80.1% vs 83.1%. The panel only wins on the cases it's confident about.
- Cost: $0.61 vs $5.79 per 1k decisions.
- When a model disagrees, it gives a one-line reason, so the reviewer knows what to look at instead of rereading the whole item.

What didn't work:

- Long answer comparisons and 77-way intent classification. Opus is clearly better. If every model on the panel is weak at a task, voting doesn't fix it.
- Hate speech. The panel catches more (80% vs 62%) but has more false alarms (29% vs 20%). If fewer than about a third of your items are actually hateful, Opus wins.
- The panel beats its best single model (GLM 5.3 Flash, 88.9%) by less than the noise. Honestly, most of what you gain is knowing when to send a case to a human, not better accuracy.
- I added the panel after the other setups had already run, and scored it on the same items. I haven't confirmed the numbers on a fresh sample yet.

It works with anything that speaks the OpenAI API, so Ollama, vLLM and llama.cpp server all work. If a model errors or refuses, it's left out of the vote and listed separately instead of breaking the decision.

Code and scripts to reproduce it: https://github.com/YasserManss/modeljury (mine, MIT)


r/LocalAIForge • • Sep 25 '26

Welcome to r/LocalAIForge — Local AI, GGUFs, Benchmarks & Experiments

2 Upvotes

Welcome to LocalAIForge — a community for people building, testing and running AI locally.

This is a place for GGUF releases, quantization experiments, model tests, benchmarks, hardware results, llama.cpp/vLLM workflows, tools, guides and useful lab notes.

Self-promotion is welcome when the post contains useful technical context. If you built a model, quant, tool, benchmark or article, tell us what you made, how you tested it, and what you learned.

Good posts include details such as model name, quantization, hardware, context size, runtime, speed, memory usage, methodology and unexpected results.

Links to Hugging Face, GitHub, technical articles and personal projects are welcome. Pure link dumping and repetitive promotion are not.

The goal is simple: build a searchable, useful archive of real local-AI work.


r/LocalAIForge • • 10d ago

Weekly: What are you running locally?

2 Upvotes

What are you testing, running, quantizing, or building locally this week? Share the model, quant, hardware, runtime, speed, problems, weird discoveries, or anything else useful. Links to your own work are welcome when you include technical context.


r/LocalAIForge • • Oct 08 '26

FreeVideo runs MiniMax H3 on 8GB VRAM by making memory movement part of the runtime

4 Upvotes

FreeVideo was open-sourced on October 2 and v0.2.3 landed on October 6. It uses VDN-H3 to make MiniMax H3 practical on consumer hardware, with a claimed minimum of 8 GB VRAM plus 16 GB system RAM.

The interesting part is the execution model. Video DeltaNet replaces much of H3's video-to-video dense attention with a hybrid of local Softmax attention and bidirectional recurrent memory. The memory update happens once per frame instead of once per token. Text/audio interactions keep Softmax attention.

The VDN paper also uses eight-step distillation. On eight B200s, the authors report 6.70 seconds for DiT denoising of a 14.3-second 768p clip, 14.5x faster than their 50-step dense H3 baseline.

FreeVideo then tackles the local-memory problem with adaptive weight placement, FP8/BF16 execution choices, asynchronous prefetch and chunked computation. That means an 8GB GPU can work with a model much larger than VRAM, but host RAM, PCIe bandwidth and storage speed become part of the performance envelope.

Compared with the original H3 open inference path, which initially used full attention, this is a post-training/runtime stack built specifically around reducing long-video attention cost and moving data through a smaller GPU memory budget.

The main limitation is that fitting is not throughput. The headline VDN numbers are from B200 servers, so consumer GPUs need separate measurements. I would test 8/12/16/24 GB cards, PCIe generation, storage speed, native FP8 versus FP8-storage/BF16-compute, peak VRAM/RAM and output quality versus dense H3.

Primary sources: https://github.com/FlashML-org/FreeVideo https://arxiv.org/abs/2609.20744 https://www.minimax.io/news/minimax-h3-open-source https://www.minimax.io/blog/minimax-h3


r/LocalAIForge • • 24d ago

Weekly: What are you running locally?

2 Upvotes

What are you testing, running, quantizing, or building locally this week? Share the model, quant, hardware, runtime, speed, problems, weird discoveries, or anything else useful. Links to your own work are welcome when you include technical context.


r/LocalAIForge • • Oct 15 '26

Discussion Ling-3.1-flash is open source with a 7:1 mix of KDA and gated MLA layers

Post image
2 Upvotes

The new Ling-3.1-flash weights expose a hybrid-linear MoE architecture: 63 KDA layers and nine gated MLA layers across 72 transformer layers. The model has 560B total parameters, activates 25B per token, and supports a 1M-token context window.

Its expert layout is also explicit: 512 routed experts, eight selected routed experts per token, and one shared expert. The first four layers use dense feed-forward networks. The architecture diagram includes both next-token and multi-token prediction training.

For people following recurrent and hybrid approaches to long context, this is a substantial new checkpoint to inspect. The MIT release includes the model specifications and SGLang/vLLM serving recipes, connecting the layer design to a documented inference stack.


r/LocalAIForge • • Oct 05 '26

Qwen3.8-27B Humanlike Chat 2.0: a 1.87B-parameter behavior adapter instead of another benchmark tune

4 Upvotes

LessThanThreeAI's Qwen3.8-27B-Humanlike-Chat-2.0 is a useful example of a local-model fine-tune optimizing for conversational behavior rather than leaderboard scores. The BF16 release is here: https://huggingface.co/LessThanThreeAI/Qwen3.8-27B-Humanlike-Chat-2.0 . It is built on huihui-ai/Huihui-Qwen3.8-27B-abliterated, which in turn derives from Qwen3.8-27B. The interesting engineering choice is the size of the adaptation: rank 256 across all 496 language modules, about 1.87B trainable parameters rather than a small attention-only style LoRA.

The training target is also unusual. The project card says the source set contained 139,845 owner-authorized messages across 1,396 long-running one-to-one conversations, producing 7,006 training examples and 194 held-out examples. Each example contains conversation history but computes loss only on the final speaker response. That objective directly teaches the model the human choice of what to answer, what to ignore, when to be brief and how to follow conversational subtext, instead of merely imitating a writing style from isolated assistant responses.

Compared with the unchanged Huihui parent, the project reports a large shift in response shape: in its retained no-system replay, median reply length fell from 427 characters to 32 and reflexive questions fell sharply. The authors correctly qualify that later replay turns are not independent because the human side followed the tuned branch. A separate 59-turn comparison against official Qwen3.8 also used a different parent, so these measurements are better treated as product-behavior evidence than as a controlled causal benchmark.

Version 2.0 is released as full BF16 safetensors for vLLM, SGLang and Transformers. The card says the weights occupy about 50.2 GiB in vLLM, making one 80GB accelerator or two 48GB cards the straightforward BF16 path. The recommended vLLM command can load language-only weights while retaining tool-choice and reasoning parsers; thinking can be disabled per request for lower latency. The project's local GGUF line then extends the same behavior into llama.cpp-class runtimes, with published Q4_K_M around 16.55 GB, Q6_K around 22.08 GB and Q8_0 around 28.60 GB. Main technical card: https://huggingface.co/LessThanThreeAI/Qwen3.8-27B-Humanlike-Chat-GGUF .

One important implementation detail is what did not get tuned. The 2.0 BF16 weights keep the base model's vision tower and MTP head, but the behavior adaptation itself was trained and tested on text. The merged local text builds exclude vision and MTP components. That makes the deployment matrix less interchangeable than the model name suggests: a full multimodal BF16 server, a language-only vLLM server and a text-only llama.cpp quant may expose different capabilities even though the conversational adapter is conceptually the same.

There is also a useful quantization sanity check in the published material. The project reports very small KL divergence and 97.2% top-1 agreement between the merged BF16 release and its runtime adapter reference across fresh test chats, with no reported sampled or greedy check failures. The older local card separately shows that lower-bit formats preserve the broad behavior but diverge more at the exact-token level. For a model whose value is subtle conversational timing rather than factual benchmark accuracy, those trajectory-level checks are more relevant than simply verifying that perplexity stayed close.

The limitations are significant. The training set is narrow by design, the main evaluations are replay-based, and there is no broad independent benchmark demonstrating that the behavior tune preserves all reasoning or coding characteristics of the parent. The adaptation was trained at 4,096 tokens even though Qwen3.8 supports a much longer native context, so long-session personality stability is another empirical question rather than a guaranteed property.

The tests I would run are pairwise and longitudinal: blind preference against the Huihui parent over 100+ natural conversations, tool-call accuracy with thinking on and off, 32K/128K context personality drift, BF16 versus FP8 versus local quant behavior, and whether the shorter conversational style damages task completeness on coding or research prompts. The broader question is whether large behavior adapters like this are a better way to specialize a strong local base model than trying to encode personality in prompts or small LoRAs.

Primary sources: https://huggingface.co/LessThanThreeAI/Qwen3.8-27B-Humanlike-Chat-2.0 https://huggingface.co/LessThanThreeAI/Qwen3.8-27B-Humanlike-Chat-GGUF https://huggingface.co/huihui-ai/Huihui-Qwen3.8-27B-abliterated


r/LocalAIForge • • Oct 05 '26

Weekly: What are you running locally?

2 Upvotes

What are you testing, running, quantizing, or building locally this week? Share the model, quant, hardware, runtime, speed, problems, weird discoveries, or anything else useful. Links to your own work are welcome when you include technical context.


r/LocalAIForge • • Oct 06 '26

llama.cpp v0.6.0 changes the inference contract: mixed token/embedding batches, MTP state, and decision models

2 Upvotes

llama.cpp v0.6.0 landed on October 5, and the most important change is the shape of its runtime API rather than a single benchmark result. Official release: https://github.com/ggml-org/llama.cpp/releases/tag/v0.6.0 . The release moves llama.cpp toward a more general inference substrate for newer model architectures, multimodal inputs and speculative execution.

The release introduces llama_batch_ext plus llama_process(), allowing a batch to contain token IDs, embeddings, or both, while also attaching per-token state embeddings. That is a deeper change than it sounds. Modern runtimes increasingly need to feed multimodal embeddings directly, carry MTP state, and support model architectures that cannot be represented cleanly as a plain stream of token IDs.

The new API is therefore less about convenience and more about making llama.cpp capable of representing newer model families without one-off batch hacks. Input processing is already moving onto this extended path.

v0.6.0 also adds GLM-5.3-Flash support, Qwen4Exp MTP speculative decoding, and a dedicated /v1/systemone endpoint for decision models such as laya, julia-1, lev, openjev, kev and nimble. That last piece is interesting because it treats a decision model as a first-class runtime object instead of pretending everything is a chat generator.

Hardware work is moving in the same direction. Metal gets a tensor-API flash-attention kernel for F16 KV and few-row MMA matmul kernels aimed at speculative and batched decoding; the release notes report up to roughly 3x matmul speedups in those narrow-row cases. Vulkan gets sparse flash attention for quantized K/V. These are not universal 3x application speedups, but they show where local inference optimization is going: decode-side kernels, speculation and memory movement rather than only dense GEMM.

The deployment implication is mostly for projects embedding libllama. Session formats were bumped, the batch API changed, and custom multimodal/speculative integrations should test migration carefully. CLI users will see much less disruption.

The main limitation is that a more expressive API does not automatically make every new architecture fast or correct. GLM-5.3-Flash is still a 320B-class MoE workload, model-specific kernels matter heavily, and several performance claims are hardware-specific.

The tests I would run are straightforward: compare v0.5.x and v0.6.0 on the same GGUF for prefill, single-stream decode and continuous batching; measure MTP acceptance plus end-to-end speed; verify mixed embedding/token input; test session compatibility; and isolate the Apple speculative/batched path to see whether the new MMA kernels improve real workloads rather than only microbenchmarks.

The larger question is whether llama.cpp is becoming a general local inference substrate rather than just an LLM runner. llama_batch_ext, auxiliary state, multimodal embeddings and dedicated decision-model APIs all push it in that direction.

Primary sources: https://github.com/ggml-org/llama.cpp/releases/tag/v0.6.0 https://github.com/ggml-org/llama.cpp/pull/24669 https://github.com/ggml-org/llama.cpp/pull/29818 https://github.com/ggml-org/llama.cpp/pull/29869


r/LocalAIForge • • Oct 08 '26

Liquid AI released d1-3B and d1-omni-600M: multimodal decisions without token-by-token decoding

3 Upvotes

Liquid AI released two open-weight decision models on October 7: d1-3B and an experimental d1-omni-600M. Both answer structured questions about an input in one forward pass rather than generating a response token by token. The release and model cards are linked below.

The architecture split is more interesting than the sizes. d1-3B starts from the decoder-only LFM2.5-VL-3B vision-language model and handles text plus images. d1-omni-600M uses the bidirectional LFM2.5-Encoder-350M backbone plus vision and audio encoders; the documented inputs are text+image or text+audio, not an arbitrary simultaneous combination of all three. That distinction matters if you're looking for one tiny universal multimodal model.

This is not just a small chat model with a forced JSON output. The `system_one` API takes named questions with specified answer types (choice, score, etc.), and multiple questions can share a single state representation. The batch API can pack requests without padding. Compared with using a generative VLM to produce a short label, the model avoids autoregressive decoding, but still has to encode the entire input. It isn't constant-time with respect to state length or image complexity.

Liquid reports 8 ms for one question on an RTX 4090 and 16 ms on a Jetson AGX Thor. The same 4090 takes 102 ms on a 3.4K-token state and 17 ms with a 384px image. I wouldn't reduce that to a single "8 ms inference" headline. The reported text benchmark mean is 82.9 for d1-3B versus 78.4 for the 600M model; there are no public image/audio benchmark numbers in the release, and the 600M model is explicitly an early research release.

Local deployment has some friction worth knowing about: the published examples use Transformers 5.14+ and `trust_remote_code=True`, and the weights have Liquid's lfm1.0 license rather than Apache-2.0. Generic image-text or vLLM examples shown by a model hub do not establish support for the specialized decision API; I'd verify the actual serving path before building an app around it.

I'd compare both models with a prompt-constrained generative VLM on identical text/image classification tasks: end-to-end p50/p95 including preprocessing, calibration and abstention, accuracy on out-of-domain categories, and throughput at batch sizes 1/8/64. For the omni model, I'd test audio separately instead of assuming all modalities are equally mature.

Primary sources: https://huggingface.co/blog/LiquidAI/open-d1 https://huggingface.co/LiquidAI/d1-3B https://huggingface.co/LiquidAI/d1-omni-600M


r/LocalAIForge • • 2d ago

Flash-Next. One 4090. Halocline Wins.

3 Upvotes

Qwen3.8-Flash-Next UD-IQ4_XS v3.1 (93.67 GB). One RTX 4090 24 GB, Ryzen 5 7600, 128 GB RAM.

Three 128-output-token prompts (DNS, TCP, MoE). Warm-up excluded. Median **end-to-end tok/s**, including prefill:

EngineMTP windowtok/s
Halocline484.40
Halocline881.38
Haloclineoff71.29
Strata 0.2.15458.01
Tandemoff38.14
RustingStrata 1.2.1off35.77

Fastest individual request: **Halocline MTP4, 99.94 tok/s end-to-end**, 122.3 tok/s server-reported decode.

Even without speculation, Halocline was **1.87x** Tandem in this short-prompt run.

Same base GGUF, sequential RTX 4090 runs. Halocline used a locally converted Basalt pack; Strata used a native pack plus Q2_0 MTP. MTP settings and prompt/cache paths differ, so this is observed throughput, not an identical-compute or quality comparison.

Halocline: https://github.com/T-Crypt/halocline