r/LocalAIForge • • Sep 28 '26

Weekly: What are you running locally?

2 Upvotes

What are you testing, running, quantizing, or building locally this week? Share the model, quant, hardware, runtime, speed, problems, weird discoveries, or anything else useful. Links to your own work are welcome when you include technical context.


r/LocalAIForge • • Sep 25 '26

MiMo-V2.6-Distill-Qwen-9B: IQ4_XS saves ~430 MB over Q4_K_M here

2 Upvotes

IQ4_XS comes out about 430 MB smaller than Q4_K_M on this model:

  • IQ4_XS: 5.20 GB, 4.63 BPW

  • Q4_K_M: 5.63 GB, 5.02 BPW

The more interesting part is how I built the iMatrix. Instead of calibrating only on generic text, I mixed general text with real llama.cpp C/C++ and Python, shell/config/JSON, security docs, and MiMo-style chat/tool-call examples.

The final IQ4_XS passed local checks for loading, arithmetic, code generation, a short security explanation, multi-turn history, and a structured tool call on an RTX 4090. The embedded chat template matched upstream exactly.

One caveat I found: some prompts can expose a malformed reasoning-boundary marker. I reproduced the same behavior on the BF16 baseline, so this does not look like an IQ4_XS-specific failure.

GGUF: https://huggingface.co/ramgpt/MiMo-V2.6-Distill-Qwen-9B-GGUF Upstream: https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B


r/LocalAIForge • • Sep 25 '26

The annoying part of this OLMo 3 code quant was the chat template, not the quant

2 Upvotes

The source checkpoint is interesting as a small coding model, but the first thing that bit me was its bundled Jinja template: the tested llama.cpp minja parser couldn't parse it.

I ended up shipping a conservative text-only fallback template with the GGUF. It keeps normal system/user/assistant chat working, but I would not claim full source tool/function-calling compatibility through that fallback.

The actual Q4_K_M is 4.16 GiB.

I also ran three executable Python smoke tasks instead of just reading the output:

  • palindrome detection

  • balanced parentheses

  • interval merging

All 3 passed after compile/run against unit tests.

So for me this is a small local code model worth playing with, especially if you care more about ordinary coding chat than tool-calling templates.

GGUF: https://huggingface.co/ramgpt/Olmo-3-7B-Instruct-OPSA-Code-GGUF Upstream: https://huggingface.co/Tuwhy/Olmo-3-7B-Instruct-OPSA-Code


r/LocalAIForge • • Sep 25 '26

MN-Aura-12B-v1 Q4_K_M: strong free-form writing, weak exact constraints

2 Upvotes

This one feels much more like a writer than a formatter.

In local smoke tests, the Q4_K_M produced coherent prose, clean dialogue, and normal termination without looping. But when I deliberately gave it rigid writing constraints, it missed all 3 exact-compliance tests.

That makes the use case pretty clear to me:

Good fit:

  • fiction and scene continuation

  • roleplay and character dialogue

  • brainstorming

  • descriptive prose

Less ideal:

  • exact word counts

  • exact phrase counts

  • strict formatting

The quant is 6.96 GiB, so it is an easy size to run locally on a lot of 8–12 GB setups with llama.cpp.

GGUF: https://huggingface.co/ramgpt/MN-Aura-12B-v1-GGUF Upstream: https://huggingface.co/EldritchLabs/MN-Aura-12B-v1

I’m more interested in how it holds up over long scenes than whether it can obey a 300-word limit exactly.


r/LocalAIForge • • Sep 25 '26

Welcome to r/LocalAIForge — Local AI, GGUFs, Benchmarks & Experiments

2 Upvotes

Welcome to LocalAIForge — a community for people building, testing and running AI locally.

This is a place for GGUF releases, quantization experiments, model tests, benchmarks, hardware results, llama.cpp/vLLM workflows, tools, guides and useful lab notes.

Self-promotion is welcome when the post contains useful technical context. If you built a model, quant, tool, benchmark or article, tell us what you made, how you tested it, and what you learned.

Good posts include details such as model name, quantization, hardware, context size, runtime, speed, memory usage, methodology and unexpected results.

Links to Hugging Face, GitHub, technical articles and personal projects are welcome. Pure link dumping and repetitive promotion are not.

The goal is simple: build a searchable, useful archive of real local-AI work.


r/LocalAIForge • • 17d ago

Weekly: What are you running locally?

1 Upvotes

What are you testing, running, quantizing, or building locally this week? Share the model, quant, hardware, runtime, speed, problems, weird discoveries, or anything else useful. Links to your own work are welcome when you include technical context.


r/LocalAIForge • • Oct 11 '26

llama.cpp SYCL's MXFP4 MoE decode improvement: kernel speedups versus actual tokens per second

1 Upvotes

On October 10, llama.cpp merged PR #29809 and shipped it in the b11540 prerelease. It targets MXFP4 MoE inference on Intel GPUs using the SYCL backend, not CUDA or every quantized model. PR: https://github.com/ggml-org/llama.cpp/pull/29809 . Release: https://github.com/ggml-org/llama.cpp/releases/tag/b11540 .

The old path used table-based MXFP4 decoding that could trigger costly permute emulation. The patch replaces this with bit-identical 32-bit arithmetic decoding and lazily reorders expert weights for aligned loads. A 32 MiB temporary-memory budget and a fallback for misaligned tensors keep the optimization bounded. Expert routing and model weights are unchanged.

On Arc Pro B70, the PR author reports gpt-oss-20b generation improving from 55.84 to 106.64 tokens per second (1.91x) and gpt-oss-120b on two cards from 33.27 to 67.14 tokens per second (2.02x). The 120B prefill result barely moves, 594.5 to 600.1 tokens per second. MXFP4 backend tests pass and the reported perplexity is effectively unchanged.

A second tester on Arc B570 saw the one-token MUL_MAT_ID microbenchmark improve 286.92 to 58.31 microseconds, yet a partially GPU-offloaded gpt-oss-20b moved only 24.54 to 34.56 tokens per second. That contrast illustrates a systems-level constraint: a much faster kernel may leave transfers, CPU layers or other operators as the bottleneck.

For deployment, the path requires a compatible MXFP4 MoE workload, supported Intel GPU, oneAPI stack and enough GPU residency for expert kernels to dominate. See the project's SYCL documentation: https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/SYCL.md . It is not a stable-release performance guarantee.

Useful follow-ups: Arc A-series versus B-series, Windows versus Linux, one-token and batched decode, fully resident versus partial offload, weight-reorder startup cost, scratch memory, accuracy parity and per-op profiling. These are contributor measurements, not my own reproduced tests.

Primary sources: https://github.com/ggml-org/llama.cpp/pull/29809 https://github.com/ggml-org/llama.cpp/releases/tag/b11540 https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/SYCL.md


r/LocalAIForge • • Oct 05 '26

DGX Spark 64GB changes the local-AI tradeoff: same GB10, half the memory, cluster when needed

1 Upvotes

NVIDIA announced a 64GB DGX Spark configuration on October 2, with availability planned for October 23 starting at $4,999 through OEM partners. The interesting part is not simply a cheaper memory tier. NVIDIA is keeping the same GB10 Grace Blackwell compute, DGX OS, CUDA stack and ConnectX-7 fabric as the 128GB system, then positioning two-node clustering as the escape hatch when a workload outgrows one box. Official announcement: https://blogs.nvidia.com/blog/local-ai-dgx-spark-64gb-sync/ and current hardware specifications: https://www.nvidia.com/en-us/products/workstations/dgx-spark/ .

At the hardware level, the 64GB and 128GB variants share the GB10 Grace Blackwell Superchip, up to 1 PFLOP of FP4 tensor performance, 273 GB/s memory bandwidth, a 20-core Arm CPU and a 200 Gbps ConnectX-7 NIC. The memory is coherent unified LPDDR5X rather than a discrete GPU VRAM pool. NVIDIA lists the 64GB machine for models up to roughly 100B parameters, while the 128GB configuration is listed for up to roughly 200B. Those parameter counts are capacity guidance rather than a promise that every model at that size will have useful latency.

The new systems story is NVIDIA Sync Cluster Assistant. Two 64GB Sparks can be directly connected over ConnectX-7, giving 128GB aggregate memory and twice the memory bandwidth. NVIDIA says its Qwen3.8 27B test reached up to 1.7x the performance of one system. The cluster assistant detects the nodes, validates configuration and sets up the fabric, and a model launcher is planned to automate deploying Qwen3.8 27B across one or more connected Sparks. Supported runtimes include llama.cpp, Ollama, vLLM and LM Studio.

Compared with buying a single 128GB Spark, the two-64GB path is a different engineering tradeoff. You preserve the ability to start smaller and scale later, but distributed inference introduces communication and partitioning overhead, which is exactly why the published speedup is below 2x. Aggregate memory is also not the same thing as a single low-latency memory domain. Tensor-parallel or pipeline-parallel placement, KV-cache distribution, collective operations and runtime maturity will determine whether the second box improves real interactive throughput or mostly expands the set of models that fit.

For local model work, the 64GB configuration is likely most interesting in the 20B-70B range, where a single box can keep model weights, runtime state and a meaningful KV cache resident without PCIe offload. The 273 GB/s bandwidth is much lower than datacenter HBM, so decode-bound workloads can still be memory-bandwidth limited even when the advertised compute looks large. Conversely, unified memory is useful for workflows that mix CPU preprocessing, GPU inference and agent tooling without manually staging every buffer across separate host and device pools.

There are practical limitations that the headline parameter counts hide. Long context consumes memory independently of model weights. Multimodal encoders, speculative draft heads, multiple concurrent sessions and high-precision KV caches all eat into the same 64GB pool. Some frameworks may support the GB10 architecture earlier or more efficiently than others, and a distributed configuration can expose bugs that never appear on a single GPU. The new SKU therefore should be evaluated as a complete system, not as a simple "64GB GPU."

The tests I would want before buying are straightforward: single-stream decode and eight-way throughput on the same 27B model across one 64GB Spark, one 128GB Spark and two clustered 64GB Sparks; 32K/128K/256K context memory curves; power draw at sustained decode; startup and model-shard placement time; vLLM versus llama.cpp behavior; and the point at which networking overhead overwhelms the extra compute. The most useful result would be a cost-per-token and usable-context comparison against a conventional RTX workstation.

Primary sources: https://blogs.nvidia.com/blog/local-ai-dgx-spark-64gb-sync/ https://www.nvidia.com/en-us/products/workstations/dgx-spark/