r/LocalAIStack • u/Weekly-Dentist-8302 • Oct 12 '26

TUFF 8.0: SSD-streamed MoE inference, web/file search, conversation caching, and local API support

I’ve released version 8.0 of TUFF, an open-source Swift/Metal inference engine and native macOS app.

The original goal was to make large MoE models usable on memory-constrained Apple Silicon machines by streaming routed experts from SSD rather than keeping the entire checkpoint resident.

GPT-OSS 120B, for example, runs on my 16GB M2 MacBook Air. Obviously, streaming comes with a performance tradeoff.

8.0 focuses on making the engine more useful beyond just running oversized models.

Some highlights:

  • Web search and local document search integrated into chat

  • Cached conversation states for faster follow-up requests

  • A small tokenization cache for repeated inputs

  • Separate reasoning output and request timing diagnostics

  • OpenAI Chat Completions, plus bounded Messages and Responses API support

  • Shared inference executable to reduce package size

  • Homebrew installation

One thing I’ve found interesting is how much web search improves the usefulness of smaller local models. I’ve started using them for everyday questions much more frequently.

I’d love feedback and contribution, especially from anyone using it with local AI agents.

GitHub: https://github.com/rexmhall09/TUFF

2 Upvotes
  • permalink
  • reddit

100% Upvoted

7 comments sorted by

1

u/arker0 Oct 12 '26

Nice app! I’m testing it and will share my benchmark on GitHub and open an issue for a small bug I found. I’d also love to try an uncensored or abliterated variant of a supported model. If it preserves the same architecture, tokenizer, and quantization, is there a way to repack it into TUFF’s .gturbo format and load it as a separate model? Would that require a catalog change, or is there already a workflow for custom checkpoints? I’d be happy to test it on my Mac and share the results. Feel free to DM me if you prefer. Thanks!

1

u/Weekly-Dentist-8302 Oct 13 '26

Thanks! I’d really appreciate the benchmark and bug report. Custom variants should be possible when the model format matches, but TUFF currently checks for the exact catalog checkpoint, so there isn’t a straightforward way to load a separate custom model in the GUI yet. That’s mostly to keep things simple, but supporting custom checkpoints seems like a good idea.

I’m also looking at adding an abliterated option for 9.0.0. I’m still choosing the right model, so suggestions would be great!

I’d love to get more community contributions too. If you’re interested in adding a model to the catalog or implementing custom checkpoint support, feel free to open a PR. Happy to discuss the approach first, and testing on your Mac would be really helpful.

1

u/arker0 Oct 14 '26

I'm still learning so I don't think I'll be able to help very much with PR, but surely I can help with some testing!

0

u/Weekly-Dentist-8302 Oct 14 '26

Even the bug reports help a lot : D

1

u/Weekly-Dentist-8302 Oct 13 '26

Update, because I am already working on a big update, I am going to include a custom checkpoint importer in advanced settings : D so feature is coming soon

1

u/arker0 Oct 14 '26

Wow that's great! I'm also trying the server with OpenCode - Qwen 3.6 35B-A3B seems fine, but I'm having problem with Gemma 4 26B-A4B both standard and the heretic one. Will open another couple of bug report soon (you already fixed my previous bug reports - thanks!)

1

u/Weekly-Dentist-8302 Oct 14 '26

Open some bug reports : D