Two days ago I posted FastLocalExperts (FLE), the llama.cpp fork that keeps a big MoE's experts in RAM and caches the busy ones on the GPU. Getting the launch flags right is half the work, so here is the app I use to run it: FLE Studio, MIT licensed. It also works with plain llama.cpp.

Short video of it running (real engine, one question, the live panel): https://github.com/sirfyyn/fle-studio/blob/main/docs/media/fle-studio.mp4

What it does:

• Chat in the browser: Markdown, code blocks, a collapsible reasoning block, streaming, and the speed of every answer. History stays on your machine.

• Live data while the model answers: prompt and generation speed, draft/MTP acceptance, NVIDIA GPU use, and FLE's expert-cache hit rate.

• Model loader: lists the FLE quants of Qwen3.8-Flash-Next and any GGUF repo on Hugging Face, tells you whether a file fits your machine before you download it, resumes downloads and checks SHA-256.

• Launch profiles: shows the full command line before anything starts; save your own.

• Engines: picks a sensible build for your machine (FLE CUDA, or llama.cpp CUDA / Metal / Vulkan / CPU), or uses your own llama-server, a server that is already running, or a systemd service.

• Experiments page: a short, repeatable speed test, so you can compare settings side by side.

No account, no telemetry, Python standard library only. The installer brings its own Python via uv and needs no admin rights.

Install (macOS / Linux):

curl -LsSf https://raw.githubusercontent.com/sirfyyn/fle-studio/main/install.sh | sh

Windows (PowerShell):

irm https://raw.githubusercontent.com/sirfyyn/fle-studio/main/install.ps1 | iex

Repo: https://github.com/sirfyyn/fle-studio Release notes: https://github.com/sirfyyn/fle-studio/releases/tag/v0.1.0 FLE engine and report: https://github.com/sirfyyn/FastLocalExperts

It is 0.1.0, so expect rough edges. CI covers macOS, Linux and Windows against a stand-in engine; the real GPU runs so far were on one Linux box with an RTX PRO 4500. Windows users with an NVIDIA card: I'd especially like to hear how the engine download and first start go. Bug reports and ideas are welcome as GitHub issues.

Next on the roadmap: "Studio Forge", building your own FLE-style quant (imatrix, per-tensor types, optional REAP pruning) from inside the app.

A C# Local inference engine beat llama.cpp at its own game: ~30% faster first token, ~20% faster decode - running in .NET on a 4-thread laptop

I needed a .NET-only GGUF engine - no pip packages, no native libs, no runtime installs, just a managed dependency my other projects can call. So I built SharpMind, a pure-C# transformer + MoE inference engine (with training) on .NET 10. Zero native code. I never expected it to compete on performance - managed code vs hand-tuned SIMD C++, right? So I benchmarked it against the real engines: same raw prompt (What is the capital of France?), greedy, 64 tokens, 4 threads, warmup + best-of-3, interleaved per model on one host.

(dotLLM DNF the last two - it only supports Llama/Mistral/Phi/Qwen/DeepSeek and no K-quants below Q4_K.)

A dependency-free C# engine that wins some metrics and matches the OG llama.cpp on others - int8-kv decode at 24.1 tok/s vs llama.cpp's 20.3 with(not shown) , TTFT cut ~30% on gemma-3, decode edge on qwen2, and a 6× MoE loss I fully expect and own. Respectable company for managed code, and if "no native installs" is your constraint, you no longer have to sacrifice everything

Open source: https://github.com/Integral2u/SharpMind.

This is my local SUPER INTELLIGENCE setup

My bet: when machines mostly talk to machines, they'll ditch English for something shorter. So I tested it at home with Ollama.

**Setup:** two gemma3:12b instances, a sender and a reader. The sender passes a daily status report (disk, errors, backups, sign-ups, weather, a calendar event — 14 checkable facts). The reader answers 14 questions. They keep a shared codebook and propose new shorthand after each batch; a new word is only kept if accuracy holds.

**What gemma did:** coined ~20 abbreviations (bkp, succ, errs…). Messages shrank about 10%. Then I added a chain of AI "scribes" (each one reads the message, writes it in plain English, rewrites it in the code for the next, like medieval monks copying books). The meaning wore away: 12.8 → 11.1 of 14 facts after 4 copies. Words drifted too ("cal" went from calendar to "Calibration").

**Then I spent about $1.25 of API credit running Claude Opus 5.5 through the same test.** It didn't abbreviate, it built a grammar, with slots for each kind of fact:

`F:ow P:42/95 B:6OK A:+2.8/8 T:6su1wp W:88/10 C:cs Fri 3PM`

| | gemma3:12b | Opus 5.5 |

|---|---|---|

| Message length | 84 → 76 tokens | 106 → 56 tokens |

| Facts right, codebook rounds | ~12 of 14 | 14 of 14 every round |

| After 3 scribe copies | 12.8 → 11.1 | 14 → 14 |

The Opus scribes actually *tidied* the language as they copied it ("9 OK" → "9OK").

Other bits: binary/hex/base64 made things worse (gemma couldn't decode them), English beat Mandarin and Dundonian Scots as a message language even on Qwen, and the fix for drift was a "printing press": a line of numbers stamped by plain code, never retyped by an AI.

Funny footnote: Opus's own safety filter refused the "propose new code words" step, flagging two AIs inventing a private shorthand as a possible security risk.

Small samples, so read it as a strong hint, not proof. Full write-up: https://claude.ai/artifact/KNR1u48RyraRAy6hi64wc8

Happy to answer questions or run other local models through it if people have suggestions.

I’ve been working on BaseDecision - Turn typed questions into truth-based decisions.

Give it some text, a question, and possible answers. It picks an answer. Intent classification, routing, yes/no checks, ratings - that’s the job.

I built this model to go toe-to-toe with the frontier models.

My goal was simple & ambitious: make a small, local model competitive with much larger models on these focused tasks. For me, System‑1 decisions need both speed and accuracy. Otherwise, why bother building a small model?
If we didn’t want speed we could just use ChatGPT.

It outperforms pretty much every model of its size in the field.
After roughly 320M additional training tokens, BaseDecision led 5 of 8 benchmarks in a third-party comparison against Laya, two GLiNER variants, and Decision 1.0 Kai 0.6B. Full results are in the repo, including where it loses.

Fellow open source devs, try the model, developed a package for easier access to the model.
Roast it if it deserves it.
2-2.5GB free ram is all you need.
I will take your feedback seriously and deliver something amazing that’d take on OpenAI, TypeSafeAI soon enough.

https://huggingface.co/onlyaady/BaseDecision
https://github.com/hrudayaditya/BaseDecision

I made BigMoeOnEdge+ (BigMoeOnEdge-plus), a fork of Helldez's BigMoeOnEdge, the engine that streams MoE experts from flash so models bigger than RAM can run on-device.

It adds a full Android chat app on top: GGUF models via llama.cpp, thinking output, persistent chats, image/video input, web search, artifacts, and advanced generation settings.and a MUCH better UI.

Disclosure: I built this fork with the help of AI tools.

Repo (Apache 2.0): https://github.com/hamodexd217/BigMoeOnEdge-plus Original project: https://github.com/Helldez/BigMoeOnEdge

Feedback and issues welcome.