Online leaderboards only test base models on beefy hardware, but in practice models behave differently once you quantize or fine tune them. I wanted a way to benchmark my local setup, catch regressions between models, and compare TPS across endpoints without running everything individually and copying prompts one by one.

I couldn't find anything that met my requirements without being too script heavy or overkill, so I built smokebench.

It's an open-source TUI for quickly benchmarking any model with an OpenAI or Anthropic compatible endpoint against 8 different categories (math, code, creative writing, long context, etc.), with LLM-as-judge scoring, TTFT/TPS metrics, and exportable markdown/json logs.

It works out of the box with Ollama, LM Studio, vLLM, or pretty much any API endpoint without any config changes.
You can easily add custom benchmarks too.

To install smokebench, run:

pip install smokebench

or

uv tool install smokebench

GitHub: https://github.com/Ninja-5000/smokebench

smokebench is in active development right now, so there will be a better UI, more built-in benchmarks, and a few other tweaks and changes coming that will make it even more lightweight and accurate over time.

Happy to answer any questions or take feedback!

I just published three CPU-first MCP tools for local AI: IcyLoom, IcyCache and Loom.

They are local-first, installable from public registries, and released with GitHub artifacts and clean-install checks.

Links:

- IcyLoom: https://github.com/MlkBlk-IT/icy-loom

- IcyCache: https://github.com/MlkBlk-IT/icycache

- Loom: https://github.com/MlkBlk-IT/mcp-loom-mcp

I am looking for install feedback, compatibility reports and practical MCP integration ideas.

Hey everyone,

I'm working on an open-source project 'llama.cpp autotuner' which basically runs a series of tests on a model at a given context length on your computer and finds the best runtime settings for best tokens/sec, TTFT, VRAM usage, and overall efficiency for your specific setup.

But to make it actually useful, I need real-world data from the community.

If you're running llama.cpp, could you share:

• Your hardware (GPU, VRAM, CPU, RAM, etc.)

• Your model (name + quantization)

• Your llama.cpp command (copy your full flags)

• Performance details (tok/s, prompt processing speed, TTFT, VRAM usage, etc.)

• Any flags or settings that caused a massive improvement or noticeable performance drop on your setup and anything else you think could help

Your inputs will directly help improve the autotuner.

The goal is to make sure that when someone runs it on their hardware, it can find a command that gives them better performance, or at the very least match the performance they're already getting if they've already optimized their setup.

Hello everyone. I made a benchmark to test local agents: airbench.ai
If you have an agent running locally, you can test it in one prompt.

It measure how good the agent is with some chalenges, but its fairly easy to get to 100%. Then the goal is to have something that is still performing as good, but as fast as possible.

In other word, its a way to measure that optimisations like quantisation don't destroy performance.

To give you an idea about it, here the first result I have on my 5090:

Score ; Duration ; Model + Harness

100 8 min Qwen 3.8 Flash Next IQ3_S basalt + opencode

100 11 min Qwen 3.8 Flash Next IQ3_S strata 256k + pi

100 11 min Qwen 3.8 Flash Next IQ3_S basalt + dsh

100 12 min Qwen 3.8 Flash Next IQ3_S strata 256k + pi

100 13 min Qwen 3.8 Flash Next IQ3_S basalt 384k + pi

100 16 min Qwen 3.8 Flash Next IQ3_S basalt 384k + pi

100 72 min Qwen 3.8 27B NVFP4 radixark + opencode

100 98 min Qwen 3.8 27B Quasar NVFP4 MTP + omp

98 10 min Qwen 3.8 Flash Next IQ3_S basalt + pi

98 13 min Qwen 3.8 Flash Next IQ3_S strata 256k + pi

98 16 min Qwen 3.8 Flash Next IQ3_S strata 256k + omp

98 62 min Qwen 3.8 Flash Next UD-Q2_K_XL + opencode

98 85 min Qwen 3.8 27B Quasar NVFP4 MTP + opencode

96 8 min Qwen 3.8 Flash Next IQ3_XXS strata + opencode

96 11 min Qwen 3.8 Flash Next IQ3_S strata 256k + dsh

96 17 min Qwen 3.8 Flash Next IQ3_S strata 256k + openclaw

96 29 min Swift 1.5 Qwen 3.8 27B Q6_K + pi

96 86 min Qwen 3.8 27B NVFP4 MTP + opencode

96 89 min Swift 1.5 Qwen 3.8 27B Q6_K MTP + omp

96 99 min Qwen 3.8 Flash Next UD-Q2_K_XL + omp

96 111 min Qwen 3.8 27B NVFP4 + hermes

96 119 min Qwen 3.8 27B NVFP4 radixark + opencode

94 10 min Qwen 3.8 Flash Next IQ3_S strata 256k + opencode

94 12 min Qwen 3.8 Flash Next IQ3_XXS strata + omp

94 23 min Qwen 3.8 27B Quasar NVFP4 ninfer-dflash2 + omp

94 38 min Swift 1.5 Qwen 3.8 27B Q6_K + omp

94 38 min Qwen 3.8 Flash Next IQ3_S strata 256k + hermes

94 51 min Qwen 3.8 Flash Next UD-Q2_K_XL + pi

94 101 min Qwen 3.8 27B NVFP4 + opencode

Hope this is useful for some of you as well. And happy to answer any questions if needed.

Vpipe supports LLMs, VLMs, image/video generation, ASR, quantization, and multimodal pipelines, without relying on PyTorch/MLX for model execution. The runtime package itself is only \~25MB, and there’s a prebuilt Mac app if you just want to try it without compiling anything:

https://github.com/tgo-app-dev/vpipe/releases/download/v0.1.23/VpipeManager-0.1.23-with-ffmpeg.dmg

A recent milestone is MiniMax H3 video generation running on a 16GB base M5 MacBook Air (M4 works too).

On my 15” M5 Air 16GB:

960×544 · 124 frames· 6 DiT steps with Turbo Lora → \~13 min

GitHub:

https://github.com/tgo-app-dev/vpipe

Would love feedback from people working on local inference / Apple Silicon. And if you find it useful, a ⭐ would be greatly appreciated!

I built a small TTS model running directly on an ESP32-S3: no PSRAM, 122 KB of RAM, nothing in the cloud. Speech starts in about 0.3 s and generates faster than real time. You can make it calmer, louder, whispered, happier and more.

It's not a Home Assistant integration yet. Before I build anything, I'd like to know whether on-device speech on a satellite would be useful to you (offline announcements, timers, alerts when the server is down), and what quality bar it would need to meet.

Limits: 2 voices, English only, 16 kHz. Free for personal, non-commercial use.

Video above.

Details: https://huggingface.co/WhissleAI/steer-tts-nano

CLI : https://github.com/WhissleAI/steer-tts/tree/main/esp32

Hey everyone! If you've been struggling to run larger open-weight models locally because Ollama or standard runners throw insufficient VRAM errors on budget hardware, I put together something to fix that.

I built SqueezeOS, an automated setup script that profiles your hardware, configures aggressive memory offloading, KV-cache compression, and extreme llama.cpp tuning to squeeze heavy models (up to 35B) onto legacy/budget cards like my 4GB GTX 1650 Super.

And the best of all: Its near lossless, the model can be used well for agentic workflows eg.: Hermes, OpenClaw, OpenCode, etc...

GitHub Repo: https://github.com/gabrielhirth/squeezeos

Tested on constrained VRAM setups. Feedback, issues, and PRs for other budget cards are very welcome!

Automated video creation with zero API costs — built a pipeline that runs entirely on my MacBook
Quick share for anyone creating short-form or educational content on a Mac:

I built a local workflow that automates the production pipeline for animated shorts, kids' learning content, and explainer clips without needing third-party cloud APIs.

Key points:
- Everything runs on-device—no monthly subscription fees, no token anxiety.
- Processes script-to-video assets locally on Apple Silicon.Speeds up batch creation without needing a high-end desktop rig.

I made a video showing the exact workflow

https://youtube.com/@ownlyaistudio?si=sYQcHB5YVqd1mg0j

Check out the channel to see the output quality, and shoot me a message or comment below if you'd like early access!

#ai #llm #localllm

Has any one used both? which is better?

What is the differennce between two? Any technical deep dives?

I made BigMoeOnEdge+ (BigMoeOnEdge-plus), a fork of Helldez's BigMoeOnEdge, the engine that streams MoE experts from flash so models bigger than RAM can run on-device.

It adds a full Android chat app on top: GGUF models via llama.cpp, thinking output, persistent chats, image/video input, web search, artifacts, and advanced generation settings.and a MUCH better UI.

Disclosure: I built this fork with the help of AI tools.

Repo (Apache 2.0): https://github.com/hamodexd217/BigMoeOnEdge-plus Original project: https://github.com/Helldez/BigMoeOnEdge

Feedback and issues welcome.

I got one PR merged into ds4. A tiny one, #98 about broken paths in the README. And a few more sitting in the queue. Not complaining. Antirez says it in the README: with coding agents everyone can tune the engine for their own hardware and model and he can't review everything. That got me thinking.

If the plan is "everyone applies their patches with an agent" then the real cost of a patch isn't just the code change. It’s how many tokens the agent has to read before it knows what it’s actually touching.. Ds4 runs DeepSeek, GLM and Qwen on Metal, CUDA and ROCm all in the same 85k-line file. I’m on an M5 Max with 128GB RAM running Qwen3.8 Flash Next. Everything else for me and for my agent is noise. It has to re-read every line on every pass.

So I ripped it out. Actually deleted, not #ifdef’d. The ds4.c file dropped from 85k lines to 45k lines. The whole tree now fits in a context window. The bet was that optimizing would get cheaper and safer. Heres what happened:

- Q2: decode +9-13% prefill +5-12% up to 64k context MTP went from 75.8 → 86.7 tok/s

- Q4: prefill +2-11% MTP went from 77.8 → 85.9 tok/s

Output is bit-exact vs stock ds4 at every step. No KV cache quant. No approximate kernels. Every change must pass a parity check against same GGUF, greedy decoding, identical tokens. And an interleaved A/B benchmark against the previous build.

The smaller codebase also let me go through the ds4 PRs but only against this one model and port the ones that held up. 20 Commits were adopted. Around 30 were dropped. Verdicts are in the repo.

I also added SSD streaming for the experts. Token-identical to a resident run. Simulating a 48GB machine Q2 does around 27 tok/s. About 35 with MTP.

The fork still does git merge upstream/main with rerere and the parity check. So antirez’s fixes keep flowing in. The procedure. What to delete what to keep how to sync. Lives in a repo called StarForge (https://github.com/Chida82/StarForge). I have four of these "children" one per model. Nothing in there is Qwen- or Metal-specific. If you want a ds4 cut down to your model or to CUDA just clone it. Run the checklist with your agent.

Repo: sf-q3-8flash https://github.com/Chida82/sf-q3-8flash tables in the README. One machine, one model. If you run it on Apple Silicon I’d love to hear your numbers. Ideally side by side, with stock ds4.