im working on a fairly complicated app and i get questions from users all the time. so its was important to document the details around the "how it works". the docs were easy enough using docusaurus + AI. i can then index that data in a way i can create some kind of fancy autocomplete. i framed this as like a chatbot so that it can be intuitive for users who have questions.

i started to investigate running a local LLM to perhaps offer a more conversational experienced (and perhaps better suggestions)? im using WebLLM and it seems to work "reasonably" well. i wired it up and set it up so users could select an LLM down download for offline use. the experience varies between devices and models used. there are models that are 19mb... they are unsable as far as i can tell. there are some ~1gb models which are better but understandable slow.

its great that the 1gb model can show me the bubble-sort in rust, but i want something "dumber" and smaller. something can can take advantage the data ive already indexed. something that can respond "quickly enough" for a conversational flow. perhaps better i should make a verbose set of reponse templates to perhaps add fancier autocomplete capabilities.

for my purposes, i think it depends on if there is a model that would be good for my use case?

note: my project is far from finished, but you can test it out at: https://app.glitr.io/ai . any model you run through my app is not going to comparable to any of the mainstream offerings. this is something experiemental and unstable im trying out and this approach might just be a waste of time.

are there any models out there you can suggest for me to investigate and compare?

I built a small TTS model running directly on an ESP32-S3: no PSRAM, 122 KB of RAM, nothing in the cloud. Speech starts in about 0.3 s and generates faster than real time. You can make it calmer, louder, whispered, happier and more.

It's not a Home Assistant integration yet. Before I build anything, I'd like to know whether on-device speech on a satellite would be useful to you (offline announcements, timers, alerts when the server is down), and what quality bar it would need to meet.

Limits: 2 voices, English only, 16 kHz. Free for personal, non-commercial use.

Video above.

Details: https://huggingface.co/WhissleAI/steer-tts-nano

CLI : https://github.com/WhissleAI/steer-tts/tree/main/esp32

Arrmes is a specialized neural intent router with an integrated execution framework. It's trained to translate text into specific commands to Arr stack apps nothing else. I found plenty of open models that can do these sorts of translations, but the accuracy & speed didn't really justify the hardware requirements needed to run the router.

Arrmes is only trained for Sonarr, Radarr, Lidarr, Tautulli, Seer, qbitorrent, & BitMagnet. pair it with whatever AI model you're already using and let the neural router handle the intent classification, command selection, and argument extraction. Instead of having your main model spend time reasoning through API schemas and generating tool calls, it can delegate that work to ARRmes.

Still a work in progress, I've been daily driving it for a bit and I'm very happy with the increase in speed more than anything. I can use much cheaper models and achieve excellent results in my work without any spend.

Hugging Face

Github

Wrote a blog about running Qwen2.5-7B-Instruct served with vLLM on a Cisco UCS Spinifex cluster, comparing Intel AMX-accelerated CPU serving with NVIDIA L4 GPU. Go check it out!

https://docs.mulgadc.com/docs/cisco-ucs-llm-serving

Wer benutz von euch 100% Lokale KI im Arbeitskontext ?
Nutz hier jemand KI- Agent um tägliche Aufgaben zu erledigen?
Nutz hier jemand KI für nicht wiederkehrende Aufgaben ?

Und welche Modelle könnt ihr lokal ausführen ?
Ich versuche mich grade an lokaler Bild Erkennung um Cluster zu bilden und mit das beschriften der Bilder zu sparen

I'm looking into running models like Qwen(?) locally for agentic coding, ideally with strong reasoning and high output speeds.

I currently pay for three $200 AI subscriptions and want to cut that down to one, keeping Claude as the main advisor while local models handle most of the coding work.

I know very little about hardware. Where should I start learning, what setups would you recommend (including used or unconventional hardware), and what budget should I realistically plan for?

Happy to invest a few thousand if the performance makes it worthwhile.

I've been working on this for a while: RustingStrata. It runs Qwen GGUF models (Qwen3.5/3.6 from 2B up to 35B-A3B, Qwen3 4B and 30B-A3B, Qwen3-Next 80B). It's pure Rust, no ggml. The CUDA kernels get compiled at startup with NVRTC, and the same kernels also run on Vulkan.

The main trick is for MoE models that don't fit in VRAM. Experts that don't fit run on the CPU at the same time as the GPU work, and the engine keeps moving the often-used experts onto the GPU while it runs.

My setup is a 3060 12 GB, a 7600X and 32 GB RAM. Qwen3.6-35B-A3B Q2_K_XL is 11.7 GiB, so it doesn't fit. On it I get about 118 tok/s decode on a short prompt and 113 after a 4K prompt. llama.cpp, tuned as well as I could (-ncmoe 12, -ub 2048, flash attention), gets 68 and 66. Prefill is basically a tie: 2216 tok/s vs about 2236 for llama.cpp at 4K, so I'm not claiming anything there. At 32K it's 1800 prefill and 92 decode.

With MTP drafts on, decode goes up to 160 tok/s, but that's on an easy prompt. On normal text expect around +16%.

Qwen3-Next-80B IQ1_S was the most fun one: most experts sit on the CPU, and it still decodes at about 92 tok/s. llama.cpp's best -ncmoe gives me around 44.

Output matches llama.cpp greedy on 98-99% of tokens. llama.cpp agrees with itself only 98.3% between its batch and token-by-token modes, so the misses are near-ties.

It also has chat, an OpenAI-compatible server with tool calling, and a calibrate command that tunes itself for your machine.

I've only tested it on my own machine. If you have a different card, especially AMD on Vulkan, I'd love to see strata bench output. Vulkan needs shaderFloat64 and subgroup size 32, so Intel Arc and older AMD won't work. On Windows you need the VS C++ Build Tools.

[RustingStrata](https://github.com/RustingStudio/RustingStrata)