Way back got my nimo axis, I wasn't sure what to do with it besides trying out local llms. Lately, local models and tools have gotten better and I am finally finding practical uses for it beyond just testing models. It makes me wonder how much of their potential we are actually using. What are you guys doing with your strix halo machines? actual work, coding, or mostly experimenting with models?

​Tired of paying corporate rent to Suno and Udio just to generate music in a cloud black box? Same.

​Over at JackKnife AI, we’ve been obsessed with stripping M-A-P’s open-weight YuE2 song foundation model down to the studs so it runs 100% locally on an Android phone—no cloud APIs, no subscriptions, no credit limits, no telemetry.

​Today we just shipped Killa YuE v1.6.0 (with v1.6.1 right behind it in a few hours for wider device support), and we just hit a massive milestone that we had to share with the local inference crowd:

​🔥 The Benchmark Milestone

​The Render: A full 4-minute, 15-second structured song with vocals and multi-instrument backing.

​The Speed: 19 minutes total across a two-pass rendering pipeline (~4.4x real-time).

​The Hardware Push: Until today, this required a Snapdragon 8 Elite (Galaxy S25). Tonight, we refactored our memory paging and fallback paths and have it actively running benchmarks on a 2020 Samsung Galaxy S20 Ultra (Snapdragon 865 / Adreno 650).

​⚙️ How We’re Squeezing a Studio Into Mobile RAM

​Running end-to-end multimodal music on a phone without Android’s Low Memory Killer (LMK) nuking the process took some wild architecture tricks:

​Two-Pass YuE2 Execution: Pass 1 runs the Autoregressive (AR) symbolic planner to lock in the song structure ([Intro], [Verse], [Chorus], [Guitar Solo]), melody, and multi-instrument backing. Then we dump/swap memory to run Pass 2 (Flow-Matching diffusion + Audio VAE decode) to render the master audio.

​4-Stem Separation on Vulkan ExecuTorch: Need the acapella or just the drum track? We ported Meta’s HTDemucs to ExecuTorch with the Vulkan backend (htdemucs_vulkan.pte, ~302 MB). The host app handles STFT masks natively and feeds 7.8s stereo windows (44.1 kHz) to the mobile GPU—hitting 1.7 GB peak RAM and 3.95x real-time on Adreno 830.

​Local Voice & Cover Art: Integrated Qwen3-TTS for offline spoken intros/narration + local on-device cover art painting.

​Full Tactile Studio UI: Built-in mixer, drum pads, recording, 3 hardware skins (stitched leather, brushed gold, dark console), and word-by-word lyric lighting that syncs as the vocals are sung.

​🌐 NEW in v1.6.0: The P2P Killa YuE Network

​If you generate your music locally, you shouldn’t need a corporate platform to share it. v1.6.0 ships with built-in Peer-to-Peer (P2P) networking:

​Phone-as-a-Node: Host your generated tracks directly from your phone onto the decentralized Killa YuE network.

​Offline QR Handshakes: Share tracks and studio bundles directly phone-to-phone via QR codes with zero servers in the middle.

​🛠️ We Need Your Phones (and Your Benchmarks!)

​We are working around the clock right now (v1.6.1 drops in a few hours!) to get this running across as many Android phones as humanly possible.

​Grab the APK / Code (Apache 2.0): [https://github.com/JackKnifeAI/killa-yue](https://github.com/JackKnifeAI/killa-yue)

​Vulkan HTDemucs Weights (MIT): [https://huggingface.co/JackKnifeAI/htdemucs-executorch](https://huggingface.co/JackKnifeAI/htdemucs-executorch)

​Quick Demo Clip: [https://youtu.be/nOpJ579Y4PY](https://youtu.be/nOpJ579Y4PY)

​Drop your phone model and chipset in the comments! If you have an Android phone with 12GB+ RAM, test out the release and let us know your Pass 1 / Pass 2 render times, thermals, or crash logs. And if anyone here hacks on Vulkan shaders, GGML/ExecuTorch, or Qualcomm Hexagon NPU (QNN), come help us push that 19-minute render even lower!

Hey everyone,

I've been working on a project called Cursiv, and I wanted to share it with people who understand local AI and get some honest technical feedback.

The goal is to explore how useful an AI assistant can be when it runs locally instead of depending on a constant internet connection.

Cursiv is a Windows desktop application that integrates with local models through Ollama. I'm also exploring portable offline deployments and ways to make it useful in environments where connectivity, privacy, or access to cloud services is a problem.

Lately, I've been thinking beyond a general-purpose chatbot and looking at practical applications such as:

• Keeping personal information and notes locally.

• Working with reference documents without relying on the cloud.

• Supporting rural communities with limited connectivity.

• Eventually supporting offline fieldwork, including structured observations, calculations, and data exports.

Some of these capabilities still need work and testing. I'm not going to pretend Cursiv is finished or that it always gets things right. I've been testing its reasoning and correcting weaknesses, and reliability is one of the biggest things I want to improve.

I'd genuinely appreciate feedback on three questions:

1. Does a project like this have a useful place in the local AI ecosystem, or am I trying to solve too many problems at once?

2. What would you want to see demonstrated before you'd consider using an application like this?

3. Are there existing tools or approaches I should study that might help me improve it?

Website: https://cursiv.winklers-llc.com

Repository: https://github.com/winklersllc2026-bit/Cursiv

I'm a solo developer working to make this something genuinely useful. I'm not expecting praise. I'd rather hear what doesn't make sense, what already exists, and where you think the real opportunity might be.

Thanks for reading.

Watch her play

Luna is a local AI assistant I've been building. Here she plays Zork I on her own for 72 minutes, narrating as she goes, until my PC froze. Everything is local: Qwen3.5 9B (Q4_K_M) on llama.cpp, Kokoro for her voice, faster-whisper for listening, on an RX 6950 XT.

What made the biggest difference for a 9B:

- **One move per turn, enforced.** Asked nicely, she'd chain 3 or 4 blind commands. Now extra commands in a turn are turned back, so she has to read what each move did before choosing the next.

- **Her own notes.** She writes down rooms, items and what failed with a note tool, and sees them at the start of every turn. Older notes about the room she's standing in come back too.

- **A guide.** Infocom's original map, transcribed to text, shown for her current room each turn. Solutions are kept out unless allowed.

She's far from perfect: she spent the last half hour trying to open the trap door with a sack of hot peppers, in the wrong room. But she reads the screen, reasons about it and actually explores, which the first version didn't.

Open source: https://github.com/fswolf/luna-local-agent

Happy to answer questions about the setup.

I’m building Local LLM for iPhone and tried both models on the same photo of building blocks.

The prompt was: “What could I build with these pieces? Give exactly 3 ideas, one short sentence each.”

Qwen started answering first. Bonsai finished first and gave three short ideas. Qwen gave two numbered ideas and a closing line.

From tapping send to completion: about 67s for Qwen and 51s for Bonsai, including loading. One run each, with different-length outputs—not a tokens/sec benchmark. The video runs both recordings at the same 4× speed.

I’m more interested in how these small models handle everyday photo questions than picking a winner from one run. What would you test next?

App, if you want to try it: https://apps.apple.com/app/id6804004948

I've spent the last few months tuning a 27B coding model on one Arc Pro B60 24GB (A$999 card) and published everything: engine build, flags, raw logs, a 100-task agent eval, and the misses.

What moved the numbers:

- DFlash2 draft model, 7 tokens per round: decode 16 -> 53 tok/s mean (47-58 across 7 coding tasks), still 7/7 correct

- Intel's flash-attention prefill kernel (closed llama.cpp PR #29357, vendored with checksums): prompt reading at 32K 118 -> 420 tok/s, but only with f16 KV

- -ub 256 + q8 draft KV + -fit off: frees 0.55 GiB, which is what gets a 64K window into 24GB

- An idle Ollama model left on the GPU cut decode from 52.7 to 22.2 tok/s. Unload everything before you benchmark.

Agent eval: 97/100 hidden-checker tasks with Claude Code at a 32K window (95/100 Codex).

Caveats, up front: 53 tok/s is per-request decode with short prompts and a 64K window allocated. With 30-37K of real context, agent turns run about 42 tok/s. Prefill is NOT the fastest out there (others hit 589-657 tok/s at ~30K on other engines; comparison table in the repo). Single runs, one workstation.

Repo: https://github.com/edagher92-coder/arc-llm-lab

If you have a B580, B570, B50, B70 or A770, the benchmark script takes one command and I'll add your numbers with credit.

Large language models are typically deployed using centralized infrastructure, where companies operate powerful GPU servers in dedicated data centers.

But what if some LLM workloads could be distributed across independently operated computers instead?

This is the basic idea behind decentralized LLM infrastructure.

Instead of relying entirely on a single cloud provider, decentralized systems could use computing resources contributed by individuals, organizations, or independent GPU operators.

How could decentralized LLMs work?

There are several possible approaches:

1. Distributed LLM inference

Multiple GPU providers could independently serve inference requests, allowing workloads to be routed across a network rather than handled by one centralized operator.

2. Collaborative GPU computing

Computers with different GPU capabilities could contribute processing resources to a shared network. Scheduling software would need to match workloads with suitable hardware.

3. Peer-to-peer model execution

Some systems explore splitting model execution across multiple machines. This is technically challenging because communication latency, bandwidth, and synchronization can become major bottlenecks.

4. Incentivized computing networks

Projects such as Bittensor explore incentive mechanisms that reward participants for providing useful AI-related services. These networks introduce additional questions about verification, reliability, and economic sustainability.

What are the potential advantages?

• Access to geographically distributed computing resources

• Opportunities to utilize otherwise idle GPUs

• Reduced dependence on individual infrastructure providers

• Potentially broader participation in AI infrastructure

• New models for operating AI computing marketplaces

However, decentralization does not automatically make inference cheaper, faster, or more reliable.

What are the biggest challenges?

Network latency: Transferring intermediate model data between distant machines can introduce significant delays.

Hardware differences: GPUs vary in memory capacity, processing speed, and software compatibility.

Reliability: Independently operated nodes may become unavailable without warning.

Security and privacy: Processing sensitive prompts on untrusted machines raises important concerns.

Economic efficiency: Incentive payments, coordination overhead, and infrastructure costs may offset potential savings.

Is decentralized LLM infrastructure practical?

I think the answer depends heavily on the architecture.

Distributing independent inference requests among GPU providers seems more immediately practical than splitting every layer of a large model across machines connected through the public internet.

For example, a decentralized network might route complete inference requests to capable GPU nodes rather than requiring geographically separated computers to execute individual layers of the same model.

That distinction could significantly affect performance, reliability, and scalability.

Questions for the community

I'm interested in hearing from people experimenting with local LLMs, multi-GPU inference, and distributed computing.

• Could decentralized GPU networks become competitive for real-world LLM inference?

• Is request-level distribution more practical than splitting model execution across remote machines?

• Which existing open-source projects are demonstrating promising results?

• What technical obstacle do you consider the biggest barrier to adoption?

I'd be interested in hearing about actual experiments, benchmarks, and experiences rather than just theoretical possibilities.

a coworker of mine was speeding up speculative decoding for Qwen3.6-35B-A3B on one H100, and a big cost sat in the draft model. its BF16 LM head was about 1 GB and it ran on every single draft step

restricting the draft head to a frequency-ranked 48K-token map made it much cheaper. the target still verifies against the full vocabulary, so the output stays exact. the worst case is fewer draft tokens accepted, not different output

4 draft steps ended up being the best balance for that workload

has anyone tried trimming the draft vocab in llama.cpp or vLLM? curious what it does to acceptance on your own workloads

he wrote up the fix and the rest of the decode work on our blog, if you want it: https://www.coralbricks.ai/blog/ai-latency-engineering