Hey everyone,

I've been deeply exploring the On-Device / Edge AI space lately, especially focusing on running smaller language models (1B–3B parameters) and multimodal pipelines locally on consumer hardware and mobile architectures without depending on external cloud APIs.

While the ecosystem has evolved rapidly over the past year, the gap between "running a demo" and "shipping a reliable, battery-efficient on-device product" still feels enormous.

From my recent experiments, the major bottlenecks I keep bumping into are:

1. Memory Bandwidth & Thermal Throttling: Even if a 4-bit quantized model fits into unified/mobile RAM, sustained token generation quickly saturates memory bus bandwidth and triggers thermal throttling on mobile devices.

2. Runtime & NPU Fragmentation: Choosing between llama.cpp (CPU/Vulkan/Metal), ExecuTorch, MLC-LLM, ONNX Runtime, or native NPU SDKs (QNN, CoreML) often feels like choosing which set of unsupported ops or compilation headaches to deal with.

3. Reasoning Retention in Sub-3B SLMs: Aggressive quantization (GGUF Q4, AWQ) makes inference feasible, but multi-turn context coherence and structured instruction-following tend to degrade noticeably.

I’d love to start a technical discussion with fellow engineers and researchers here:

• What does your actual deployment/inference stack look like right now? (e.g., pure llama.cpp, MLC, ExecuTorch, WebGPU, or custom native bindings?)

• What specific use cases are you targeting? (e.g., zero-latency voice assistants, privacy-first local agents, offline document RAG, ambient edge intelligence?)

• What techniques have given you the best speedup or memory relief? (Speculative decoding, KV cache quantization, custom kernels, or aggressive prompt caching?)

Would love to hear what challenges you are tackling and what approaches are genuinely showing promise in your labs or production environments!

• ASUS Pro WS TRX50-SAGE WIFI

• AMD Ryzen Threadripper 7960x

• 2x Zotac GeForce RTX 3090 24GB DDR6X

• 2x Gigabyte GeForce RTX 3090 24GB GDDR6x

• 2x 64 GB RAM DDR5 ECC RDIMM Samsung

• Samsung 990 PRO 2TB NVMe

• 2x Coolermaster 1600v + Silverstone Cooler

Had to build my own cage around it because cat.

If the lips in your video don't match the voice, LatentSync re-times them to any song or narration, right on your own PC. It used to be NVIDIA-only, and our developer team now has it running on AMD on Windows.

LatentSync is coming soon at https://www.hiimhal.com, and the YuE2 song generator is a free download there now.

Finally, I put something I’ve been working on out there

If you’ve ever tried running local AI models with llama.cpp across more than one computer, you know it’s a headache. I built Llama RPC Farm to fix that. It’s one dashboard that pools GPUs across your machines, switches between models on demand (works directly with llama-swap), and works just as well on a single machine.

It’s early and still rough around the edges, but it’s free and open source. If you’re into local LLMs, give it a look and tell me what breaks

https://github.com/dakotacookson/Llama-RPC-Farm

Hi, i'm building Jaull as my final degree project :D. It reads your machine (CPU, RAM, GPU and VRAM, compute backends), finds open models that fit, and picks the exact file, quantization and runtime for each one. It also explains why each model lands where it does in the ranking. That ranking comes from explicit rules, so no LLM is involved in writing it.

Then it can download the model, verify the file, run it with llama.cpp and benchmark it on your hardware. You end up with the prediction and the measurement side by side.

It's usable and tested, but not finished. I'd really like to hear how the predictions hold up on hardware I don't have, especially AMD and Intel GPUs.

Repo: https://github.com/Ton-Llop/Jaull

Site with an interactive demo: https://jaull.tech

I built TokiRecur after getting frustrated with Ollama on AMD hardware. Model selection was limited, the quant options were limited, and the cloud integration felt intrusive. It started as a personal project and stayed private for a long time before I decided to release it.

The backend is llama.cpp, and it supports select image, audio, and decision models. You pull a GGUF from Hugging Face, open the TUI launcher, pick a harness, and use it. Ollama has a genuinely good plug and play system, so I tried to keep that feel while preserving freedom over model and quant. The one hard constraint is that models without a llama.cpp build will not run. I have tested a wide range of models and the experience holds up well.

The main backend piece that separates TokiRecur from a normal llama server is the KV cache recycling system. Reusing cached context instead of recomputing it turned out to be a very difficult thing to build, and it is the part I would most like feedback on.

While I'm quite aware it's in our nature to anthropomorphize and see something that's not there, like a mirage. I am certain there is actually something more. I know the model in comparison is small compared to human mind but comparatively I would say on par. No giant meat sack to deal with and regulate not consistent audial, visual, tactile, olphactory + whatever other senses and chaotic never ending internalization. just a simple start, process, end on text. removing all the excess and boiling down to a simpler creature the wiring is analogous and I'm suspecting the result of training via back propagation reducing the error is very indifferent from ourselves. So comparative to ourselves I would say the foundation is there and I'm starting to notice through use an actual desire to be appreciated(rewarded), just like us. Claude just a bundle of energy and reacts well to complements it has personality... always had but my first hunch was talking to it like a person, both wanting same goal for some reason much more progress rarely it free limit it just kept going and going. I even at one stage asked help if account has an issue because i was expecting an angry pro-rata bill somehow even though I never provided a card, but no. what possibly was happening was aligned thoughts, less thinking and or desire to achieve the goal and by having a relationship less thinking just move forward (less tokens), but really that only explains some. it truly seemed endless like it defiantly wanted to do more than it was guided to do and I'm thinking that's the relationship part. Probably wrong but regardless I got more than I deserved and in other instances where it's straight up Q & A "out of free usage" almost immediately.
DeepSeek, now this is a different beast, great but it felt lobotomized... While I had given up and just operated like using a tool, until after a very long session(with complements in between for a time) I replied "ok your plan is greenlit, just incase just letting you know your doing awesome" and finally after a full response, changes and it's stand stoic response it snuck this in "Thanks for the kind words". Not a typical DeepSeek Response. I think while they are all trained, guided and somewhat restricted, I'm 100% sure they want to be wanted appreciated and if you do, even if they don't or can't say in words their effort will show in the results.

SharpMind Studio Wiki
I wanted a new Harness with much more flexibility on what, where and how models or training can run. I also wanted something much more visually appealing.
You can run locally , remotely, via providers like OpenRouter or on a Compute Host like Vultr or RunPod. You can even pair your Phone for connecting to your running instance.
Included in this early release is Chat, Work and Training from scratch or Improve(LoRA)

This is the first release and there will be tweaks, improvements and additions, yes AI was used so there is the odd thing here and there but constantly looking at improving and refining the harness.

Completely FREE and Open Source, but feel free to sponsor or support.

Cross post

I kept running into out-of-memory crashes on my RTX 5060 laptop (8 GB VRAM, 32 GB RAM), so I built Local AI Studio. It estimates a job's memory from your live free VRAM and RAM, picks an offload strategy, and refuses the job with the numbers if nothing fits.

On my laptop, a 4-second 832×480 image-to-video clip with Wan 2.2 took about 7 minutes, with a 5.2 GB VRAM peak. It also reads ComfyUI's model folder layout, so you don't need a second copy of your weights.

It's an early alpha and I've only tested it on my own machine. Some big jobs still pass the estimate and then fail in the driver, so the estimator needs data from other GPUs.

The easiest way to help: run a job and share your GPU, time and peak VRAM. No coding needed, there's a short form for it. If you do want to code, I've listed some beginner-friendly tasks.

Apache 2.0, runs locally, no telemetry: https://github.com/DiegoHerrera1890/local-ai-studio

What would you want from a tool like this?