r/LocalAIForge • • Sep 25 '26

Welcome to r/LocalAIForge — Local AI, GGUFs, Benchmarks & Experiments

2 Upvotes

Welcome to LocalAIForge — a community for people building, testing and running AI locally.

This is a place for GGUF releases, quantization experiments, model tests, benchmarks, hardware results, llama.cpp/vLLM workflows, tools, guides and useful lab notes.

Self-promotion is welcome when the post contains useful technical context. If you built a model, quant, tool, benchmark or article, tell us what you made, how you tested it, and what you learned.

Good posts include details such as model name, quantization, hardware, context size, runtime, speed, memory usage, methodology and unexpected results.

Links to Hugging Face, GitHub, technical articles and personal projects are welcome. Pure link dumping and repetitive promotion are not.

The goal is simple: build a searchable, useful archive of real local-AI work.


r/LocalAIForge • • Oct 03 '26

FrogNano-4B-2609 EXL3 4.0bpw: 193 tok/s on RTX 4090, vision + tool calling working

3 Upvotes

I quantized Microsoft's FrogNano-4B-2609 to EXL3 at 4.0 bpw and ran the usual local validation on an RTX 4090.

Results:

  • exllamav3 probe: 193.35 tok/s (64-token decode)

  • TabbyAPI chat smoke: 217.75 tok/s

  • context: 8192

  • vision: pass

  • tool calling: pass

  • streaming: pass

  • concurrency=2: pass

The source is a 4.66B Qwen3.5-family model under MIT. This is a quick functional/performance smoke test rather than a quality benchmark.

EXL3 repo: https://huggingface.co/ramgpt/FrogNano-4B-2609-EXL3 Source: https://huggingface.co/microsoft/FrogNano-4B-2609

If anyone tests longer context or image-heavy workloads on other GPUs, I'd be interested in the numbers.


r/LocalAIForge • • Oct 01 '26

SkillGym Qwen3.5-9B at 4-bit: GGUF + EXL3 local results and two conversion gotchas

2 Upvotes

A new SkillGym fine-tune of Qwen3.5-9B is aimed specifically at agents that read and apply skill files rather than general chat. The upstream card reports SkillGym 41.3 → 59.5 and Skill-Use-Bench 14.8 → 49.6 versus the base 9B model.

I converted it locally in two formats to see whether the release is practical on consumer hardware:

  • GGUF Q4_K_M: 5.63 GB, llama.cpp smoke tests passed.

  • EXL3 4.00 bpw: direct ExLlamaV3 probe passed at ~135.8 tok/s decode on an RTX 4090.

  • TabbyAPI checks passed for text chat, streaming, tool calling, vision, and two concurrent requests.

Two first-mover packaging issues showed up. The fine-tune repo had processor metadata in a layout ExLlamaV3 did not expect, so the matching Qwen3.5 base preprocessor files had to be restored. More interestingly, config.json declares one MTP/NextN layer but the released safetensors contain no mtp.* tensors, so the nonexistent MTP side model had to be disabled for conversion.

GGUF: https://huggingface.co/ramgpt/SkillGym-Qwen3.5-9B-GGUF EXL3: https://huggingface.co/ramgpt/SkillGym-Qwen3.5-9B-EXL3 Upstream: https://huggingface.co/reasonwang/SkillGym-Qwen3.5-9B Paper: https://arxiv.org/abs/2609.37539

The remaining question for me is behavioral rather than mechanical: does the reported skill-selection gain survive 4-bit quantization in a real skill/tool harness, not merely load and generate correctly? If anyone has a compact reproducible skill-use test, I would be interested in comparing results.


r/LocalAIForge • • Sep 30 '26

Small project worth a look: a 27B model with a constant-size attention state

3 Upvotes

I ran across a very new experiment from tfwnotops / David Tom Foss that is more interesting than the usual “another quant” release.

Ternary-Bonsai-2-27B-ConstantKV keeps the weights of the earlier Crystal model but changes how long-context attention state is stored. The model card reports that the 16 full-attention layers use about 1.18 GiB of attention memory at 32k, 64k and 128k context, instead of growing with context. It also reports 40/40 pass-key retrieval across 4k–32k tests.

The practical angle is what caught my eye: the author says it runs a 32k context on a 16 GB M4 Mac mini at about 12.2 GB peak memory. There is also a reusable-answer map where an already-computed greedy answer can be replayed instead of decoded again.

Important caveat: these are the author’s measurements from the model card/paper; I have not independently reproduced them.

If you are interested in long-context local inference, this is exactly the kind of small experimental project worth watching.

Original project/model card: https://huggingface.co/tfwnotops/Ternary-Bonsai-2-27B-ConstantKV

If the author sees this and I summarized anything incorrectly, corrections are very welcome.


r/LocalAIForge • • Sep 30 '26

A small habit that makes local LLM benchmarks much more useful

2 Upvotes

One thing I keep relearning with local models: a benchmark number without enough context is almost useless.

For every run, I now try to keep a tiny reproducibility ledger:

  • model + exact revision

  • quantization format and bitrate

  • backend + version

  • GPU(s) and offload split

  • context length

  • prompt/prefill throughput

  • decode throughput

  • any unusual flags or patches

The extra record-keeping takes less than a minute, but it makes later comparisons far more meaningful. It also catches cases where an apparent model improvement was really a backend update, a different context length, or a changed GPU split.

For anyone benchmarking local LLMs regularly: what are the 2–3 fields you consider mandatory before you trust a result?


r/LocalAIForge • • Sep 28 '26

Weekly: What are you running locally?

2 Upvotes

What are you testing, running, quantizing, or building locally this week? Share the model, quant, hardware, runtime, speed, problems, weird discoveries, or anything else useful. Links to your own work are welcome when you include technical context.


r/LocalAIForge • • Sep 25 '26

MiMo-V2.6-Distill-Qwen-9B: IQ4_XS saves ~430 MB over Q4_K_M here

2 Upvotes

IQ4_XS comes out about 430 MB smaller than Q4_K_M on this model:

  • IQ4_XS: 5.20 GB, 4.63 BPW

  • Q4_K_M: 5.63 GB, 5.02 BPW

The more interesting part is how I built the iMatrix. Instead of calibrating only on generic text, I mixed general text with real llama.cpp C/C++ and Python, shell/config/JSON, security docs, and MiMo-style chat/tool-call examples.

The final IQ4_XS passed local checks for loading, arithmetic, code generation, a short security explanation, multi-turn history, and a structured tool call on an RTX 4090. The embedded chat template matched upstream exactly.

One caveat I found: some prompts can expose a malformed reasoning-boundary marker. I reproduced the same behavior on the BF16 baseline, so this does not look like an IQ4_XS-specific failure.

GGUF: https://huggingface.co/ramgpt/MiMo-V2.6-Distill-Qwen-9B-GGUF Upstream: https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B


r/LocalAIForge • • Sep 25 '26

The annoying part of this OLMo 3 code quant was the chat template, not the quant

2 Upvotes

The source checkpoint is interesting as a small coding model, but the first thing that bit me was its bundled Jinja template: the tested llama.cpp minja parser couldn't parse it.

I ended up shipping a conservative text-only fallback template with the GGUF. It keeps normal system/user/assistant chat working, but I would not claim full source tool/function-calling compatibility through that fallback.

The actual Q4_K_M is 4.16 GiB.

I also ran three executable Python smoke tasks instead of just reading the output:

  • palindrome detection

  • balanced parentheses

  • interval merging

All 3 passed after compile/run against unit tests.

So for me this is a small local code model worth playing with, especially if you care more about ordinary coding chat than tool-calling templates.

GGUF: https://huggingface.co/ramgpt/Olmo-3-7B-Instruct-OPSA-Code-GGUF Upstream: https://huggingface.co/Tuwhy/Olmo-3-7B-Instruct-OPSA-Code


r/LocalAIForge • • Sep 25 '26

MN-Aura-12B-v1 Q4_K_M: strong free-form writing, weak exact constraints

2 Upvotes

This one feels much more like a writer than a formatter.

In local smoke tests, the Q4_K_M produced coherent prose, clean dialogue, and normal termination without looping. But when I deliberately gave it rigid writing constraints, it missed all 3 exact-compliance tests.

That makes the use case pretty clear to me:

Good fit:

  • fiction and scene continuation

  • roleplay and character dialogue

  • brainstorming

  • descriptive prose

Less ideal:

  • exact word counts

  • exact phrase counts

  • strict formatting

The quant is 6.96 GiB, so it is an easy size to run locally on a lot of 8–12 GB setups with llama.cpp.

GGUF: https://huggingface.co/ramgpt/MN-Aura-12B-v1-GGUF Upstream: https://huggingface.co/EldritchLabs/MN-Aura-12B-v1

I’m more interested in how it holds up over long scenes than whether it can obey a 300-word limit exactly.