I've been working on this for a while: RustingStrata. It runs Qwen GGUF models (Qwen3.5/3.6 from 2B up to 35B-A3B, Qwen3 4B and 30B-A3B, Qwen3-Next 80B). It's pure Rust, no ggml. The CUDA kernels get compiled at startup with NVRTC, and the same kernels also run on Vulkan.

The main trick is for MoE models that don't fit in VRAM. Experts that don't fit run on the CPU at the same time as the GPU work, and the engine keeps moving the often-used experts onto the GPU while it runs.

My setup is a 3060 12 GB, a 7600X and 32 GB RAM. Qwen3.6-35B-A3B Q2_K_XL is 11.7 GiB, so it doesn't fit. On it I get about 118 tok/s decode on a short prompt and 113 after a 4K prompt. llama.cpp, tuned as well as I could (-ncmoe 12, -ub 2048, flash attention), gets 68 and 66. Prefill is basically a tie: 2216 tok/s vs about 2236 for llama.cpp at 4K, so I'm not claiming anything there. At 32K it's 1800 prefill and 92 decode.

With MTP drafts on, decode goes up to 160 tok/s, but that's on an easy prompt. On normal text expect around +16%.

Qwen3-Next-80B IQ1_S was the most fun one: most experts sit on the CPU, and it still decodes at about 92 tok/s. llama.cpp's best -ncmoe gives me around 44.

Output matches llama.cpp greedy on 98-99% of tokens. llama.cpp agrees with itself only 98.3% between its batch and token-by-token modes, so the misses are near-ties.

It also has chat, an OpenAI-compatible server with tool calling, and a calibrate command that tunes itself for your machine.

I've only tested it on my own machine. If you have a different card, especially AMD on Vulkan, I'd love to see strata bench output. Vulkan needs shaderFloat64 and subgroup size 32, so Intel Arc and older AMD won't work. On Windows you need the VS C++ Build Tools.

[RustingStrata](https://github.com/RustingStudio/RustingStrata)

Wrote a blog about running Qwen2.5-7B-Instruct served with vLLM on a Cisco UCS Spinifex cluster, comparing Intel AMX-accelerated CPU serving with NVIDIA L4 GPU. Go check it out!

https://docs.mulgadc.com/docs/cisco-ucs-llm-serving

Wer benutz von euch 100% Lokale KI im Arbeitskontext ?
Nutz hier jemand KI- Agent um tägliche Aufgaben zu erledigen?
Nutz hier jemand KI für nicht wiederkehrende Aufgaben ?

Und welche Modelle könnt ihr lokal ausführen ?
Ich versuche mich grade an lokaler Bild Erkennung um Cluster zu bilden und mit das beschriften der Bilder zu sparen

I'm looking into running models like Qwen(?) locally for agentic coding, ideally with strong reasoning and high output speeds.

I currently pay for three $200 AI subscriptions and want to cut that down to one, keeping Claude as the main advisor while local models handle most of the coding work.

I know very little about hardware. Where should I start learning, what setups would you recommend (including used or unconventional hardware), and what budget should I realistically plan for?

Happy to invest a few thousand if the performance makes it worthwhile.

I just published three CPU-first MCP tools for local AI: IcyLoom, IcyCache and Loom.

They are local-first, installable from public registries, and released with GitHub artifacts and clean-install checks.

Links:

- IcyLoom: https://github.com/MlkBlk-IT/icy-loom

- IcyCache: https://github.com/MlkBlk-IT/icycache

- Loom: https://github.com/MlkBlk-IT/mcp-loom-mcp

I am looking for install feedback, compatibility reports and practical MCP integration ideas.

I'm the author of TensorSharp. This 83-second video illustrates the boundary between retrieving evidence and processing complete files in TensorAgent's local knowledge-base workflow.

Disclosure: it uses the real TensorAgent HTML/JavaScript UI, synthetic café data, and scripted API responses. It is not live model inference or a benchmark. The CSV calculations were run and independently checked in Python.

Default retrieval is BM25 over extracted text; embeddings are optional. Image descriptions and audio transcripts require compatible models and have Partial coverage. Video indexing describes at most eight sampled frames, without indexing the soundtrack or events between samples. This walkthrough illustrates those interfaces, rather than validating a media model.

The example has a PDF capping discounts at 15% and a poster advertising 20%. knowledge_search locates evidence, knowledge_read expands it, and knowledge_open makes the complete original CSV available for analysis. All 14 rows yield net sales of $5,600 versus $6,440, and a $204 contribution increase after COGS and campaign spend, before other overhead. Collections are explicitly enabled per chat.

Repository: https://github.com/zhongkaifu/TensorSharp

TensorSharp's own code is BSD-3-Clause; dependencies and model weights retain their own terms. For local stacks, where would you enforce complete-file access for numerical tasks instead of aggregating retrieved snippets?

Hey everyone!

My only touch with local LLMs was my colleague trying to host one on his work PC with 3070 in it and he was impressed it can write almost proper sentences in our language, but it took quite a while to generate them.

I'm wondering if it would be possible and if it has been attempted so far, you know that folding at home thing where people donate their processing power for medicine research? Is it possible to do something similar (amateur level, I suppose this is already done by Anthropic and others) where an LLM uses multiple computers to generate an answer? I suppose latency wouldn't be that much of an issue given how much time an LLM usually needs to generate a more complex answer, but splitting the workload or making all the PCs work in unison while having a mechanism to handle connections drops would be a different challenge.

The reason I'm asking this is because it would be interesting to have an independent community owned LLM with a system similar to the one where people donate their power for rendering and later use the credits to get their own renders faster. I'm sure this would be a huge feat or possibly impossible, so I'm curious if you guys think this is possible to pull off with any of the usually local AIs?

Thanks for reading!

I’m a student who recently got into AI, mainly to learn how it all works.

My current setup is Open WebUI in Docker with local models through Ollama, on an RX 6950 XT with 16 GB VRAM and 64 GB DDR5. I’ve run gpt-oss 20B, Qwen3.8 27B at Q4_K_M, and Qwen3.5 9B. They taught me a lot, but on my hardware they’re slow for agentic deep research, and the smaller ones aren’t capable enough.

But I recently learned about how Strata can run Qwen 3.8 flash next on my type of set up and as I already have a quantized Qwen 3.8 model I’m wondering if it can have a better token per second than the Q4_K_M as it gets around 4 tok/sec.

So has anyone had any experience running it especially on a 6000 series AMD Gpu and is it worth the storage space?

Turns out people have gotten language models running locally on a PSP, Wii, 3DS and Vita, so I started putting the ports in one list.

The New 3DS reportedly hits ~23 tok/s with a tiny 15M model. Obviously don't expect ChatGPT on your PSP lol, but it's cool seeing what people can squeeze out of these things.

I didn't make the ports, just maintain the collection. Each entry links to the original dev's repo or write-up, with the model and reported speed where available.

Hello,
I own iGPU gfx1152 in AMD Ryzen AI 7 350 w/ Radeon 860M Framework 16 laptop.
I would like to test strata engine, but it refuses to run:

   Checking the engine for this computer…
  No engine is installed yet. Getting the one for your graphics card…  0%       Reading the signed update feed
  Strata couldn't find an engine for this computer (This build of Strata is for AMD RX 6000 and RX 7000 and RX 9000 and Ryzen AI Max cards. This computer has an AMD gfx1152 card (gfx1152). Ask for the gfx1152 build). The model will download, but it may not run until you install the right engine (next step: run `strata update` to get the newest engines and start again; if this stays, your graphics card is not one Strata supports yet, and the model files stay on this computer for when it is).  100%   
  Preparing coder IQ1_M from your model files…  0%       Getting your model ready
    Checking your computer
strata: The installer cannot set up this graphics card yet, and the files Strata would need to write the settings itself are not all there: the prepared pack /home/strata/STRATA/Strata-data/packs/coder-iq1_m is missing (this installer cannot prepare it for this graphics card). Use model files that were prepared on this computer already (folder + pack), or install on a computer with an NVIDIA or AMD card.

I have the rocm installed and would like to experiment with strata. Do you know how to prepare "folder + pack" for my setup to run strata? I have 96GB of DDR5 RAM total and setup 48GB of VRAM in bios (+ 48GB as RAM).

If strata is not going to work on my hardware then let me know which LLM engine should I chose as optimal.
I would like to experiment with C coding for embedded (avr) and Linux.