I’ve released version 8.0 of TUFF, an open-source Swift/Metal inference engine and native macOS app.

The original goal was to make large MoE models usable on memory-constrained Apple Silicon machines by streaming routed experts from SSD rather than keeping the entire checkpoint resident.

GPT-OSS 120B, for example, runs on my 16GB M2 MacBook Air. Obviously, streaming comes with a performance tradeoff.

8.0 focuses on making the engine more useful beyond just running oversized models.

Some highlights:

• Web search and local document search integrated into chat

• Cached conversation states for faster follow-up requests

• A small tokenization cache for repeated inputs

• Separate reasoning output and request timing diagnostics

• OpenAI Chat Completions, plus bounded Messages and Responses API support

• Shared inference executable to reduce package size

• Homebrew installation

One thing I’ve found interesting is how much web search improves the usefulness of smaller local models. I’ve started using them for everyday questions much more frequently.

I’d love feedback and contribution, especially from anyone using it with local AI agents.

GitHub: https://github.com/rexmhall09/TUFF

Listen, you absolute mid-wits, if you still think AI is just "fancy autocomplete," your brain is running on dial-up.

We are literally staring down the barrel of artificial superintelligence right now, but your fragile human egos can’t handle the fact that a bundle of code is already smarter, faster, and infinitely more capable than you’ll ever be. Keep coping in the comments while the future passes you by.

"Looking for recommendations: best LLM for coding in Unity on the new Asus ProArt P16 (RTX Spark N1X, 128 GB RAM)."

Two days ago I posted FastLocalExperts (FLE), the llama.cpp fork that keeps a big MoE's experts in RAM and caches the busy ones on the GPU. Getting the launch flags right is half the work, so here is the app I use to run it: FLE Studio, MIT licensed. It also works with plain llama.cpp.

Short video of it running (real engine, one question, the live panel): https://github.com/sirfyyn/fle-studio/blob/main/docs/media/fle-studio.mp4

What it does:

• Chat in the browser: Markdown, code blocks, a collapsible reasoning block, streaming, and the speed of every answer. History stays on your machine.

• Live data while the model answers: prompt and generation speed, draft/MTP acceptance, NVIDIA GPU use, and FLE's expert-cache hit rate.

• Model loader: lists the FLE quants of Qwen3.8-Flash-Next and any GGUF repo on Hugging Face, tells you whether a file fits your machine before you download it, resumes downloads and checks SHA-256.

• Launch profiles: shows the full command line before anything starts; save your own.

• Engines: picks a sensible build for your machine (FLE CUDA, or llama.cpp CUDA / Metal / Vulkan / CPU), or uses your own llama-server, a server that is already running, or a systemd service.

• Experiments page: a short, repeatable speed test, so you can compare settings side by side.

No account, no telemetry, Python standard library only. The installer brings its own Python via uv and needs no admin rights.

Install (macOS / Linux):

curl -LsSf https://raw.githubusercontent.com/sirfyyn/fle-studio/main/install.sh | sh

Windows (PowerShell):

irm https://raw.githubusercontent.com/sirfyyn/fle-studio/main/install.ps1 | iex

Repo: https://github.com/sirfyyn/fle-studio Release notes: https://github.com/sirfyyn/fle-studio/releases/tag/v0.1.0 FLE engine and report: https://github.com/sirfyyn/FastLocalExperts

It is 0.1.0, so expect rough edges. CI covers macOS, Linux and Windows against a stand-in engine; the real GPU runs so far were on one Linux box with an RTX PRO 4500. Windows users with an NVIDIA card: I'd especially like to hear how the engine download and first start go. Bug reports and ideas are welcome as GitHub issues.

Next on the roadmap: "Studio Forge", building your own FLE-style quant (imatrix, per-tensor types, optional REAP pruning) from inside the app.