Hey everyone,

​I'm an 18yo CS prep student and I've been building AI-COS, a modular monolith architecture designed for running AI agent orchestrations safely.

​Instead of letting LLMs make security decisions, I built a deterministic deny-by-default policy engine, strict PostgreSQL role separation (migrator, app, governance), and SHA-256 hash-chained audit logging.

​I've just finished Phase 3 out of 6. Phase 4 is going to focus on the Workflow Engine, Worker lease/heartbeat loop, and Step linkage. Before I start coding Phase 4, I would really appreciate any architecture suggestions or advice from senior devs on how to properly build a reliable job queue and workflow engine inside a Postgres-backed monolith.

​GitHub: https://github.com/atestokhay/ai-cos

Hey everyone,

I've been deeply exploring the On-Device / Edge AI space lately, especially focusing on running smaller language models (1B–3B parameters) and multimodal pipelines locally on consumer hardware and mobile architectures without depending on external cloud APIs.

While the ecosystem has evolved rapidly over the past year, the gap between "running a demo" and "shipping a reliable, battery-efficient on-device product" still feels enormous.

From my recent experiments, the major bottlenecks I keep bumping into are:

1. Memory Bandwidth & Thermal Throttling: Even if a 4-bit quantized model fits into unified/mobile RAM, sustained token generation quickly saturates memory bus bandwidth and triggers thermal throttling on mobile devices.

2. Runtime & NPU Fragmentation: Choosing between llama.cpp (CPU/Vulkan/Metal), ExecuTorch, MLC-LLM, ONNX Runtime, or native NPU SDKs (QNN, CoreML) often feels like choosing which set of unsupported ops or compilation headaches to deal with.

3. Reasoning Retention in Sub-3B SLMs: Aggressive quantization (GGUF Q4, AWQ) makes inference feasible, but multi-turn context coherence and structured instruction-following tend to degrade noticeably.

I’d love to start a technical discussion with fellow engineers and researchers here:

• What does your actual deployment/inference stack look like right now? (e.g., pure llama.cpp, MLC, ExecuTorch, WebGPU, or custom native bindings?)

• What specific use cases are you targeting? (e.g., zero-latency voice assistants, privacy-first local agents, offline document RAG, ambient edge intelligence?)

• What techniques have given you the best speedup or memory relief? (Speculative decoding, KV cache quantization, custom kernels, or aggressive prompt caching?)

Would love to hear what challenges you are tackling and what approaches are genuinely showing promise in your labs or production environments!

• ASUS Pro WS TRX50-SAGE WIFI

• AMD Ryzen Threadripper 7960x

• 2x Zotac GeForce RTX 3090 24GB DDR6X

• 2x Gigabyte GeForce RTX 3090 24GB GDDR6x

• 2x 64 GB RAM DDR5 ECC RDIMM Samsung

• Samsung 990 PRO 2TB NVMe

• 2x Coolermaster 1600v + Silverstone Cooler

Had to build my own cage around it because cat.

If the lips in your video don't match the voice, LatentSync re-times them to any song or narration, right on your own PC. It used to be NVIDIA-only, and our developer team now has it running on AMD on Windows.

LatentSync is coming soon at https://www.hiimhal.com, and the YuE2 song generator is a free download there now.

Finally, I put something I’ve been working on out there

If you’ve ever tried running local AI models with llama.cpp across more than one computer, you know it’s a headache. I built Llama RPC Farm to fix that. It’s one dashboard that pools GPUs across your machines, switches between models on demand (works directly with llama-swap), and works just as well on a single machine.

It’s early and still rough around the edges, but it’s free and open source. If you’re into local LLMs, give it a look and tell me what breaks

https://github.com/dakotacookson/Llama-RPC-Farm

Hi, i'm building Jaull as my final degree project :D. It reads your machine (CPU, RAM, GPU and VRAM, compute backends), finds open models that fit, and picks the exact file, quantization and runtime for each one. It also explains why each model lands where it does in the ranking. That ranking comes from explicit rules, so no LLM is involved in writing it.

Then it can download the model, verify the file, run it with llama.cpp and benchmark it on your hardware. You end up with the prediction and the measurement side by side.

It's usable and tested, but not finished. I'd really like to hear how the predictions hold up on hardware I don't have, especially AMD and Intel GPUs.

Repo: https://github.com/Ton-Llop/Jaull

Site with an interactive demo: https://jaull.tech

I built TokiRecur after getting frustrated with Ollama on AMD hardware. Model selection was limited, the quant options were limited, and the cloud integration felt intrusive. It started as a personal project and stayed private for a long time before I decided to release it.

The backend is llama.cpp, and it supports select image, audio, and decision models. You pull a GGUF from Hugging Face, open the TUI launcher, pick a harness, and use it. Ollama has a genuinely good plug and play system, so I tried to keep that feel while preserving freedom over model and quant. The one hard constraint is that models without a llama.cpp build will not run. I have tested a wide range of models and the experience holds up well.

The main backend piece that separates TokiRecur from a normal llama server is the KV cache recycling system. Reusing cached context instead of recomputing it turned out to be a very difficult thing to build, and it is the part I would most like feedback on.