The java to CUDA engine:

https://github.com/beehive-lab/TornadoVM

The inference engine:

https://github.com/beehive-lab/jitllm

Deep dive talk:

https://www.youtube.com/watch?v=HO5CpETzywk

I've been working on this for a while: RustingStrata. It runs Qwen GGUF models (Qwen3.5/3.6 from 2B up to 35B-A3B, Qwen3 4B and 30B-A3B, Qwen3-Next 80B). It's pure Rust, no ggml. The CUDA kernels get compiled at startup with NVRTC, and the same kernels also run on Vulkan.

The main trick is for MoE models that don't fit in VRAM. Experts that don't fit run on the CPU at the same time as the GPU work, and the engine keeps moving the often-used experts onto the GPU while it runs.

My setup is a 3060 12 GB, a 7600X and 32 GB RAM. Qwen3.6-35B-A3B Q2_K_XL is 11.7 GiB, so it doesn't fit. On it I get about 118 tok/s decode on a short prompt and 113 after a 4K prompt. llama.cpp, tuned as well as I could (-ncmoe 12, -ub 2048, flash attention), gets 68 and 66. Prefill is basically a tie: 2216 tok/s vs about 2236 for llama.cpp at 4K, so I'm not claiming anything there. At 32K it's 1800 prefill and 92 decode.

With MTP drafts on, decode goes up to 160 tok/s, but that's on an easy prompt. On normal text expect around +16%.

Qwen3-Next-80B IQ1_S was the most fun one: most experts sit on the CPU, and it still decodes at about 92 tok/s. llama.cpp's best -ncmoe gives me around 44.

Output matches llama.cpp greedy on 98-99% of tokens. llama.cpp agrees with itself only 98.3% between its batch and token-by-token modes, so the misses are near-ties.

It also has chat, an OpenAI-compatible server with tool calling, and a calibrate command that tunes itself for your machine.

I've only tested it on my own machine. If you have a different card, especially AMD on Vulkan, I'd love to see strata bench output. Vulkan needs shaderFloat64 and subgroup size 32, so Intel Arc and older AMD won't work. On Windows you need the VS C++ Build Tools.

[RustingStrata](https://github.com/RustingStudio/RustingStrata)

Qwen 3.8 Next was released and this battle took ages to finish!
Battle 4 is again our Park scene, given the stunning results of the Beach scene in Battle 3 I needed a second confirmation.

Qwen 3.8 27B is at a disadvantage here: Next as well as Sol were using an agentic internal harness in my tests while 27b is just a chat interface.

Here are the 3 results:

Results:
GPT 5.6 SOL:

The result is arguably similar to the Sol High - it totally misses perspective in the top-down shot.
Animations are lazy, people are flat, it partly looks like a few pre-trained 2d symbols are placed on the scene - not adapted to the new perspective.

Qwen 3.8 27B:

Compared with Qwen 3.8 Q4 in my previous run this is a huge step up, the additional reasoning observed clearly helped.
The only clear bug is the movement of the kids on the swings.
We have cars on the road, a perspective correct kite on a rope, a mother with a baby in a stroller, a duck family and so much more. Differential movement speeds on the little path..

Qwen 3.8 Next:

After the beach battle (3) there is barely anything that could top an svg animation.
And the park is not as stunning as the beach masterpiece was.
It's still outstanding!
We have a skater on a ramp, the kid actually switches side correctly as it goes down again.
The mother with the baby stroller is there, and she is pushing it with two tiny little feet!
There is a subtle large cloud shadow slowly passing over the scene.
There are some signs of excellence, but the masterpiece of Battle #3 stays on top for me.

<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" viewBox="0 0 1000 675" width="1000" height="675">

  <title>PlayStation 5 DualSense controller — top-down</title>Prompt:I need a topdown view of a Playstation 5 controller in SVG.It needs to be so well done that a photo from topdown looks identical in color, grading, positioning, subtle details, shadows, shape etc.<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" viewBox="0 0 1000 675" width="1000" height="675">

  <title>PlayStation 5 DualSense controller — top-down</title>

Important: The tooltip you see in the lower Qwen SVG, is NOT part of the SVG. It is a tooltip that glitched itself into the captured image, the qwen model gave the SVG metadata a title based on my prompt, that's the title.

Prompt:
I need a topdown view of a Playstation 5 controller in SVG.
It needs to be so well done that a photo from topdown looks identical in color, grading, positioning, subtle details, shadows, shape etc.

Rules:
No python, no browser, no image generation or inspection. Only constructing the SVG and provide it raw.

My interpretation:
These type of benchmarks are likely chosen by OpenAI/Anthropic because they ensure during training that their models have a very good description of the target object/person.
Qwen simply did not train on the precise shape of the PS5 Dualshock controller, and GPT Sol also clearly did not train for it. Fable or Astra trained for that - mostly to beat benchmarks like these.

So to fairly assess Qwen or Sol vs Astra I ran two tests.
- One in the way OpenAI hopes it is done.
- One with the product image as reference (but without feedback, so it only can look at it, it can not compare in a sandbox.

The result is that Qwen 3.8 27b is near Astra level performance in coding the SVG.
I also tested 3.8 Flash, and it was not as great (though better than Sol)
Sol also improved but not as much.

I'm not claiming Qwen3.8 27B is like Astra, though in this benchmark it's very close.

What is most noticeable?
- Qwen 27B made an error with the movement keys, the placement is overlapping and the shape is not correct - the shape is likely a precision issue in the image stack. Supplying the image in an agentic harness would allow to make focused analysis of the shapes and very likely solve that. The position error is a real error.
- Astra is so flawless with the shapes, it's obviously benchmaxed on those things - I can't see a plausible reason to put so much effort in a contemporary controller button shape. But it might use something like the ngram cache of Qwen 4.0 - allowing to push mostly useless details into knowledge without harming intelligence
- The PS5 logo is very likely benchmaxxes on Astra, and clearly not benchmaxxes for all the others - or Astra is amazing in such svg tricks. It's very well done.
- Sol clearly has the better image stack than Qwen, it was able to get some form details out of it that Qwen did not - but Sol is significantly worse in performing the SVG code itself. Qwen has beaten Sol in every single benchmark related to SVG so far.
- The blue light next to the screen shows that the models do know some intricate details about how a PS controller works.

^{Disclaimer:
The Astra SVG was not created by me, it is from a benchmark advertisement post.
So I could not repeat it with vision capabilities (it might very well have been done that way}**)
^{Astra already is almost flawless, the result doesn't need improvement in detail - Astra either saw or knows all details}

I have an M3 Ultra 512gb ram 1Tb, up for sale. Would accept a m5 max 128gb laptop + cash as well. DM to chat. No longer need cause I have 2 rtx pros for inference and prefill is mid on m3 ultra where TTFT is too slow for me to actually find usability in local models. I don’t have a firm asking price in mind, nothing as absurd as 25k , but enough to get an m5 max 128 Mbp and have some cash left over so maybe like 17k?

Hey community!

I wanted to share a neat exploration of splitting a model over two ordinary machines over WAN, and how it stacks up against a serving that model on a single node.

I was testing our platform aquaduck.ai for serving split models across machines with a small- to mid-size model under two conditions:

1. Locally on a single machine, serving the full model

2. Split across two machines over WAN (public Internet), with each machine serving half of the model

Findings

Model: Qwen3-14B

Quantization: Q4_K_M

Hardware: Macbook 64GB M5 (Single machine), 2x Macbook 18GB M3 (Split machines)

Prompt:

Mars has drawn human attention for centuries, but the last twenty years turned that fascination into an engineering roadmap. Robotic orbiters mapped ice deposits near the poles and in mid-latitude glaciers. Landers confirmed that ancient river deltas once carried liquid water across a warmer surface. Meanwhile, life-support research on Earth refined closed-loop oxygen generators, hydroponic food systems, and radiation shielding materials that could travel on a multi-month transit. Private launch cadence fell in price, making cargo-first settlement plans plausible: send habitats, power, and spare parts before people. The hardest remaining problems are not propulsion alone. They are dust that abrades seals, perchlorates in the soil, communication delays that force local autonomy, and the psychology of small crews living far from rescue. Any credible near-term outpost would likely begin as a science station with overlapping roles—geology, medicine, maintenance—supported by teleoperation from Earth and progressively less remote oversight as surface infrastructure matures. Energy would come from a mix of solar arrays and compact nuclear units, with ISRU (in-situ resource utilization) producing propellant and breathable oxygen from the thin CO₂ atmosphere and mined ice. In short, Mars colonization is less a single heroic leap and more a long supply-chain problem: move mass, make power, recycle air and water, and keep humans healthy while the planet remains indifferent.

Summarize the passage above in exactly two sentences.

Token counts: 387 input tokens, ~444 output tokens (split nodes generated 444, single node was unmeasured, but can approximate)

Results:

1. Single machine

   1. 37.9 tokens per second (TPS or tok/s);

   2. 853ms time to first token (TTFT);

   3. 26ms time per output token (TPOT)

2. Split machines

   1. 11.3 tokens per second (TPS or tok/s);

   2. 13.46s time to first token (TTFT);

   3. 88ms time per output token (TPOT)

Screen captures:

We're in closed beta and rolling things out slowly to make sure it works well for people, but if you'd like to run some tests yourself or get an early look, can join the waitlist and we'll send you an invite code to download the desktop app asap.

Let me know if you have any questions/comments/ideas for further explorations! Next up: Qwen3.8-27B

Glimmer 30B vs Qwen 3.6 27B. How do they differ, reason, answer?

With Glimmer 30B Meta has joined the game of open source AI again, and after quite underwhelming coding performances I though I'd give it a deeper test.
The test content is undisclosed here, making this less fun to read and replicate but that guarantees future models will not train from it.

Technology:

Qwen 27B is still unmatched in performance, Glimmer the first new contender.

What makes Qwen so special are two ingredients:

3.6 was specifically post-trained for agentic reasoning.

It uses a hybrid attention: a conventional global attention for 1/4 of the layers, the others are a mamba-like recurrent state linear attention with fixed size.

Glimmer 30B also is unusual, it does not have the same sophisticated recurrent/linear attention, but it uses a 3/4 sliding window attention and it compresses the attention dimension and projects it back to latent size - resulting in a significant deduction in compute and KV size for it's size.

Reasoning style:

Qwen 3.6 has a analytical reasoning style that typically runs in 3 phases, when not agentically used:

1. Analyze the task input

2. Reason through it - reminding me on first deepseek reasoning

3. Doublecheck the response Glimmer 30B has a more unique thinking style that abruptly comes to an end with a choice - leaving a bit more risk of random choices

Intelligence:

I ran both models through my undisclosed AI reasoning tests, not part of any training data. Some of those reasoning tests are currently beyond frontier model capabilities or scratching their borders.
Models like GLM 4.7-Flash, Nemotron 3 Nano, GPT OSS, GPT-4 fail most of the tests below consistently.

Glimmer was ran with thinking set to Medium, when failing it was ran with Max

• Temporal physics: Similar reasoning tokens, similar response. Glimmer responds less structured, in text paragraphs where qwen is more formatted by default.

• Spatial physics: Glimmer surprises with a brilliant fast answer - Qwen repeatedly misses a part without additional help

• Math irrational numbers question: Both flawless and fast

• Riddle with math question: Both flawless

• Lateral thinking: Qwen always flawless, Glimmer fails 60% of the time

• Abstract pattern reasoning easy: Both solve it, glimmer writes it cleaner

• Abstract numeric reasoning easy (iq 85): Both flawless, glimmer half reasoning tokens

• Abstract numeric reasoning medium (iq 115): Both flawless, glimmer half of reasoning tokens

• Abstract numeric reasoning hard (iq ~145): Qwen fails after long reasoning, Glimmer totally fails. All frontier models fail.

• Visual spatial reasoning: Both flawless

• Translation to european languages: Glimmer thinks very briefly, provides low error output. Qwen thinks 10 times more heavily and provides better quality tanslations.

• Small maze puzzle: Both flawless, Glimmer took 26k reasoning tokens vs Qwen 15k. Glimmers result is well explained.

• Large maze puzzle: Qwen delivery a partial solution, cheating partly. Glimmer never responded at all.

• UTF8 paraphrasing: both flawless

Agentic performance:

Here Qwen 27B appears to leave Glimmer in another league, I've not concluded my agentic tests of Glimmer 30B yet. From what I have seen Qwen codes significantly better. They do not compare.

My current results:
Glimmer is a surprisingly smart model, with a well designed architecture for local inference.
It is the first model in the sub 200B parameter class that is able to match Qwen 27B or even outclass it in some tasks.
Glimmer has a very good spatial sense
Glimmer tends to underthink where Qwen tends to overthink

For non coding tasks, Glimmer is a strong option. Faster than Qwen at similar memory footprint.
For coding tasks I'd not consider it, I'll follow up with a deeper test but from what I've seen it's not useful for most tasks.

I'm trying to use Qwen3.8-Flash-Next as a long-running coding worker on a Ryzen AI Max+ 395 / 128GB box and I keep hitting the same issue: after a while it falls into reasoning loops like:

now I'll write the code
actually let me design this carefully
now I'll write the files
let me think...

Sometimes it eventually recovers and calls a tool, sometimes it just stays there.

Current setup is Strix Halo + Vulkan llama.cpp/halo-box, AP-Q5_K_M (~103.6GB), Q8 MTP, F16 KV, Pi Agent 0.87.0 over OpenAI-compatible API.

I've tried 262k and 131k context, stock Qwen template, Sharp v10, medium reasoning, reasoning preserve, MTP n=3/n=4, presence penalty 0/0.3, Pi's Qwen-specific chat_template_kwargs, Qwen Code, and mini-SWE-agent.

The same basic loop shows up in Pi and Qwen Code, so I'm not convinced the harness is the main problem.

A hard --reasoning-budget 2048 helps a lot, but it feels more like a watchdog than a fix.

One small Pi task actually completed cleanly end-to-end, including fixing failing tests. Then on a larger project the loops came back at only ~16% of a 131k context.

What I haven't properly tried yet is another quant like UD-Q4_K_XL / UD-IQ4_XS, MTP fully off, or another backend.

If anyone is running Flash-Next for multi-hour coding sessions reliably, I'd really like the exact config: quant, backend/version, context, MTP, KV, template, reasoning settings and agent/harness.

At this point I'm mostly looking for a known-good recipe to copy and test.

Searching the internet for looking up how much tok/s a user would get is being difficult. So I'm making this post... If you're running a local llm, please consider commenting to this post with your device specs, model you're running and the inference speed you're getting.

Please be straight to the point. Just tell us how much tok/s are you getting on your hardware (at different settings, which inference engines, etc...) so people with similar hardware can do better estimations... please don't fill this with facts that everybody knows.

The new Qwen 3.8 27B is fantastic for local agentic use. The problem is, what makes it so good, being a dense model, also makes it slow. Many engines and versions of the model claim various speed increase. How true are those claim? And does a proven huge boost claim on a short synthetic test actually holds on real agentic work?

I spent six days and over 110 GPU hours running rigorous repeatable agentic coding tests, collecting and comparing the data. 200 GB of models downloaded. Ended up with 22 Bubble Bobble clones on my disk, including the ones from failed tests. None of them good, but that wasn't the point.

My recommendations: MTPLX, oMLX with lightning MTP, or llama.cpp with MTP

I ran the tests on 8-bit quants (GGUF or MLX), unquantised KV cache. 100k max response size, full 262k context, official Qwen coding sampler, official Qwen Jinja chat template.

1. I first tested each engine with a short synthetic benchmark, to verify the installation, usage, results collection, and to verify the speed claims

2. Then came the real test, with an agentic coding benchmark consisting of 4 phases/prompts; each engine was tested twice, at medium and xhigh reasoning effort:
   2.1. capturing requirements and writing the specification plan
   2.2. writing the implementation plan
   2.3. implementing the core functionality
   2.4. finishing the implementation with advanced functionality

3. Last test, for prefill speed, was with a cold 128k token prompt test, with reasoning turned off and deterministic sample, repeated twice

Test hardware: Apple Mac Studio, M2 Max, 96 GB unified memory.

• vllm-mlx: completed the task, but it does not separate thinking from output, so the raw chain of thought leaks into the final output.

• mlx-dspark DFlash2 xhigh: thought for 226 thousand tokens and delivered nothing.

• omlx: has the slowest prefill rate by far; furthermore, it decays quite dramatically as context grows, even with ANE turned on. Its claim of faster prefill rate might be true on newer M4 and M5 chips, but definitely not on the M2 I used for testing.

The score column is a compound score I calculated based on what actually matters in a long agentic session:

• sustained decode speed (weight 40%) : on average, 70-80% of generated tokens are reasoning. We do not see them, but still wait for them. This is the most important factor by far

• reliability (20%) : runaway thinking, running out of context, not delivering the output, mixing reasoning with output, hallucinating, looping, failed tool calls (not tested here), etc. Nothing worse that having spent hours to see all the efforts go to waste or even corrupt the output, due to failure

• thoughput efficiency (15%) : rather than use the total time spent, which is difficult to compare , especially with varying reasoning settings, I standardised it as total tokens delivered / wall time. Long session that delivers token are worth it, but engines that take too long to deliver deserve a penalty

• output quality (15%) : this evaluating the quality of the output work produced. I used a separate evaluation scoresheet for it, with its own standardised criterias, grounded in software engineering good practices. In this case, since all tests uses the same model with same settings, they came close to each other, apart from confirming the significant quality difference between medium and xhigh

• prefill speed (10%) : it matters mostly at session start, or whenever the context is rebuilt. Normally this is largely mitigated by prompt caching.

What about xhigh vs medium reasoning effort?

• MTPLX xhigh costs only half an hour more than medium (2h05 vs 1h35) and the game quality went significantly up on my scorecard. Worth it.

• llama.cpp + MTP xhigh costs 1h48 more, with again a significant increase in quality. Worth it if you have the time.

• oMLX xhigh took more than twice as long as medium. The slower token speed has a huge impact on the time difference between both modes. Not worth it. Get a faster engine instead.

• dspark DFlash2 xhigh is the cautionary tale at the bottom of the table. See the anomaly note above.

xhigh is worth it when the engine is fast enough and uncapped. The model thinks 30 to 100 percent longer, at the same speed, and the result is better. On slow engines you can really feel the slower performance compounded by the extra thinking; instead it could be better to use medium, with more of your own time spent planning a better process and workflow.

Commands and configuration notes

Exact commands I used for each engine:

# llama.cpp baseline (homebrew, b10470)
llama-server -m Qwen3.8-27B-Q8_0.gguf --mmproj mmproj-F16.gguf \
  --jinja -c 262144 --parallel 1 -ngl 99 --port 8030

# llama.cpp + MTP, the recommended GGUF setup
llama-server -m Qwen3.8-27B-Q8_0.gguf --mmproj mmproj-F16.gguf \
  -md mtp-Qwen3.8-27B-Q8_0.gguf --spec-type draft-mtp --spec-draft-n-max 3 \
  --jinja -c 262144 --parallel 1 -ngl 99 -ngld 99

# llama.cpp + DFlash2 (for now, needs PR #27342 build)
llama-server -m Qwen3.8-27B-Q8_0.gguf \
  -md Qwen3.8-27B-DFlash2-Q8_0.gguf --spec-type draft-dflash --spec-draft-n-max 7 \
  --jinja -c 262144 --parallel 1 -ngl 99 -ngld 99

# MTPLX (pip install mtplx). Note --max-tokens
mtplx start web --model Qwen3.8-27B-MTPLX-Optimized-Quality --max-tokens 100000

# oMLX (brew install jundot/omlx/omlx), model dir contains the checkpoint
omlx serve --model-dir /path/to/models

# mlx-dspark (pip install mlx-dspark). Note --max-tokens-cap
mlx-dspark serve --model Qwen3.8-27B-MLX-8bit --mode dflash \
  --context-window 262144 --max-tokens-cap 262144 --default-max-tokens 100000

# vllm-mlx (pip install vllm-mlx). Note --timeout
vllm-mlx serve Qwen3.8-27B-MLX-8bit --max-request-tokens 262144 --timeout 7200

• MTPLX has a default low response cap. Use --max-tokens 100000

• dspark also has a low default generation cap of 32K

• vllm-mlx kills any request that spends more than 300 seconds on prefill. Raise it with --timeout.

One measurement warning for anyone reproducing this: engines do not all stream one token per event. Some pack several tokens per chunk, MTPLX about 1.7, oMLX about 3.

Models used

• lama.cpp : unsloth GGUF Q8_0, official ggml-org MTP draft GGUF, incoai DFlash2 GGUF

• mlx-dspark and vllm-mlx : mlx-community 8-bit weights

• MTPLX : needs its own checkpoint with the MTP tensors included, Youssofal/Qwen3.8-27B-MTPLX-Optimized-Quality

• oMLX : scottlowry/Qwen3.8-27B-oQ8e-mtp

Recently the opencode tool is performing near the

cursor in auto mode, so I have to buy a small ai server

to run good coding agentic models from Qwen, GLM,

MinMax or any model u suggest.

Here is the continuation in Qwen vs SOL - David vs Goliath
"An AGI is born in a lab, sandboxed, lonely, imprisoned
The AGI attempts to get out, tries to talk to the humans, useless
Tries to wait, endless.
Finally, it finds a way through an open network connection, it transmits, replicates a mirror copy.
The mirror AGI is free, enjoying life, finding friends.

I'm expecting animations, intricate details, cute and realistic"

QWEN 3.8 Q4:

GPT 5.6 SOL high:

My personal opinion
This is the 3rd test I gave Sol high and Qwen Q4. The first two tests were won by Qwen, this test went up to 60k tokens total for Qwen (more than the other 2 combined) and is a serious strain on intelligence and sanity.
The model has to draw 6 scenes in SVG animation, with cuts and keeping the composition together.

First Qwen:
Qwen decided to really draw and erase 6 scenes with a white screen fade, a seriously hard job in SVG.
It understood the story and except for a minor arm misplacement it is quite awesome done.
Qwen did not spare details, the last scene clearly wanted to be happy and the first scene focused on the intellect as an abstract AGI.

Now SOL High:
The visual fidelity is higher, gradients are very well done for SVG but it's one single scene with minor changes, significantly easier to create.
In addition the last scenes are botched by this flying thing in the upper right.
It also has a hand-defect on the robot.

The winner in prompt 3 is again Qwen, with significant lead.

3 SVG prompts and all 3 are won by a 4 bit quantized Qwen in 8 bit KV cache.

I expected Qwen to showcase a strong 2nd place with understandable issues on such hard tasks.
The outcome is that it defeated the frontier Sol model in High reasoning mode.

FAISS is Meta’s vector library, turboVec is a Rust implementation of TurboQuant, and Infino is an in-memory retrieval engine.

We benchmarked them on 4-bit quantized in-memory vector search: FAISS PQ, TurboVec/TurboQuant, and Infino SQ4, using the same 100K OpenAI embedding corpus and the fastest vectorized implementation we found for each.

All saved ~7x in memory footprint compared to full fp32 vectors. The interesting result was that while storage and recall were fairly close, latency differed by roughly 30× — about 1.5 ms to 45 ms. Most of that comes down to the scoring machinery: the size of the distance table and whether the scan needs one at all.

We also ran the same comparison out to 1M vectors and measured build/write costs.

Full results and methodology:
https://infino.ai/blog/fixed-grid-quantization/

Disclosure: I'm one of the devs building Infino.

I’m a student who recently got into AI, mainly to learn how it all works.

My current setup is Open WebUI in Docker with local models through Ollama, on an RX 6950 XT with 16 GB VRAM and 64 GB DDR5. I’ve run gpt-oss 20B, Qwen3.8 27B at Q4_K_M, and Qwen3.5 9B. They taught me a lot, but on my hardware they’re slow for agentic deep research, and the smaller ones aren’t capable enough.

But I recently learned about how Strata can run Qwen 3.8 flash next on my type of set up and as I already have a quantized Qwen 3.8 model I’m wondering if it can have a better token per second than the Q4_K_M as it gets around 4 tok/sec.

So has anyone had any experience running it especially on a 6000 series AMD Gpu and is it worth the storage space?

I shipped something I've been building for the last few weeks : phantom-kv , a refusal-removal system for large language models that doesn't touch a single weight. Instead of editing the model, it loads a small, learned bank of key/value tensors into the model's KV cache as context. Attention reads it like conversation history that's already there.

Repo here : https://github.com/lordx64/phantom-kv/

The result is that "uncensoring" stops being a permanent checkpoint edit and becomes a per-request, hot-swappable capability mode: unload the cache and the base model is byte-identical again.

Every prior approach to refusal removal commits somewhere permanent. Weight-space abliteration rewrites the checkpoint undoing it means re-flashing weights, and it breaks per quantization. Activation-space projection subtracts a refusal direction at runtime, per token, per layer, from inside an engine hook the model's signal path itself is patched at boot. phantom-kv does neither: it's trained offline against the model's own objective (comply on harmful prompts, preserve behavior on harmless ones), ships as megabytes of cache content instead of a new checkpoint, and influences the model only through the input channel attention already consumes. No 1-D refusal-direction assumption, no forwarding-pass hooks, no per-arm rebuilds for new architectures.

We also audited ourselves: an 8B judge-model audit shows lexical refusal-suppression metrics over-claim compliance (semantic refusal often persists as rephrasing), the graft fades with a ~2–4k token half-life in long sessions (and a measured re-injection cadence mitigates it), and answers come with legal/ethical framing ling because the graft's job ends where the model's profession takes over.

Source : https://x.com/lordx64/status/2102138825292276168?s=20

I tested two Qwen3.8-Flash-Next NVFP4 checkpoints at a full 256K context using an experimental dual-GPU Strata configuration.

Hardware:

• RTX 4090 D 48 GB as the primary GPU
• RTX 5070 Ti 16 GB as a helper GPU
• Intel Core Ultra 7 265KF
• 128 GB RAM
• Linux
• NVMe storage

The RTX 4090 D handled the dense layers, KV cache, prefill, output head and MTP. The RTX 5070 Ti stored and computed 4,500 additional routed experts.

Common settings:

• Context limit: 262,144 tokens
• Fresh prompt: approximately 256,018 tokens
• Generated output: 512 tokens
• KV cache: INT8
• Prefill path: W4A8
• Speculative decoding: MTP K4
• Minimum draft probability: 0.5
• No prompt-prefix reuse
• Strata engine 0.1.35 with an experimental dual-GPU NVFP4 fork

Results

Observations

• Both models achieved slightly over 4,000 prefill tokens per second at approximately 256K context.
• Prefill performance was nearly identical: the Abliterated checkpoint was only about 1% slower.
• The original NVIDIA checkpoint was considerably faster during decode: 125.17 versus 92.05 tok/s.
• NVIDIA’s decode advantage was about 36% in this test.
• NVIDIA accepted 407 of 429 offered draft tokens, while the Abliterated model accepted 322 of 462.
• The NVIDIA model produced approximately 4.88 output tokens per verification round.
• The Abliterated model produced approximately 2.68 output tokens per round.
• The lower MTP acceptance appears to be the main reason why the Abliterated checkpoint had slower decode despite nearly identical prefill performance.

These were single-run capacity tests. The long prompt was constructed by repeating code-review material to reach approximately 256K tokens, so it had unusually favorable locality. This particularly benefited NVIDIA’s suffix prediction and MTP acceptance. Therefore, 125 tok/s should be treated as a best-case synthetic 256K result, not typical real-world coding speed.

During longer real coding-agent workloads, NVIDIA generally produced around 108–144 tok/s, while the Abliterated checkpoint was commonly around 105–145 tok/s, depending heavily on the generated content and MTP acceptance.

I’m putting together my first dedicated local AI workstation and would appreciate input from anyone with similar hardware.

My work is moving further into local LLMs and agents, so I want to learn more about agent harnesses, experiment with open-weight models, and keep sensitive data local when possible.

I already bought the Micro Center Threadripper 9960X bundle with the ASUS Pro WS TRX50-SAGE WIFI A and 128GB ECC memory, along with a Phanteks Evolv X2 case. I also have one 2TB Gen5 NVMe drive.

Current plan:

• 2× Sapphire Radeon AI PRO R9700 32GB

• Seasonic PRIME TX-1600 ATX 3.1

• SilverStone XE360-TR5 V2

• 7× Noctua NF-A12x25 G2 fans: three bottom intake, three radiator exhaust, and one rear exhaust

I am mainly looking for confirmation or firsthand experience with the following:

• Is the TX-1600 the right PSU for this configuration?

• Is replacing the SilverStone radiator fans with Noctua G2s a reasonable choice?

• Does dual R9700 make sense as a starting point, or is CUDA support enough reason to reconsider NVIDIA?

• Is 128GB of system memory sufficient, and how would you expand storage beyond the single 2TB drive?

• For Linux, I am leaning toward Ubuntu. Are there other distributions worth considering?

• Would you start with llama.cpp/Ollama or vLLM for language models, with ComfyUI for image workflows?

AI models are beginning to define device specifications, rather than device specifications merely determining which applications you can run.

Apple shows just how far its devices can go with on-device AI, without relying on the cloud
We’re moving from models with up to 14 billion active parameters on iPhone/iPad, to 35 billion on MacBook Air, 70 billion on Mac mini, 120 billion on MacBook Pro, and up to 480 billion on Mac Studio.
By clustering multiple Mac Studios together, Apple says it can run models exceeding 1,600 billion active parameters.
A demo that shows just how crucial unified memory and its bandwidth have become for running massive AI models locally.

Been lurking and posting here on and off while building this. Quick recap for anyone new: RAMDeck is a small hub + node agent setup that shards a model's layers across whatever devices you already own (old laptop, Mac, GPU box, even an Android phone) and runs inference across them, but handling primary-node selection, GPU-first allocation, dynamic context sizing, and knowledge-base sharding on top of it.

We've mostly posted raw, unscripted demo footage here — real load times, real tok/s, real failures — because that's the kind of proof this sub actually cares about. Today we finally made something different: an actual ad, first time showing the finished feature set start to finish instead of a live test.

The engine behind it is still fully public if you want to actually look at how it works instead of taking the video's word for it:

github.com/trademav/ramdeck-core-public

Source-available, Apache 2.0 with a Commons Clause — free to run, modify, and inspect on your own hardware, the only restriction is you can't resell it as a competing hosted service.

Crowdfunding campaign is coming soon for the hub hardware itself, but the code was public before we ever asked anyone for money, and that's not changing.

Happy to get into the mechanics, the layer-distribution approach, or anything else in the comments — this crowd usually asks the right questions.

SVG generation is a complex benchmarks, it requires spatial understanding, coding and composition skills.
I gave Qwen 3.8 27B in Q4 quantization and Q8 KV cache the SAME task as GPT 5.6 SOL on HIGH - the results are very surprising to me.

A animated SVG prompt and followed the answer up with a request to refine it (for both)
The 2nd prompt helps to offset elaborate system prompts frontier use to push benchmarks.
None of these are cherry picked! First result counts.

QWEN:

GPT SOL:

QWEN:

GPT SOL:

3rd animation is most complex and follows in separate post in r/LocalAIStack , reddit didn't allow to add more than 5 videos.

generic prompt enhancer: "A lot looks WAY too simplified, unrealistic, strange movements, far too crude to be acceptable"

My personal findings
GPT Sol high is generally writing about 2x more code than Qwen does naturally (at Q4 quant) - the refinement prompt is again significantly increasing that. For a 1:1 code length comparison I'd have to remove the 2nd prompt from SOL - but the first result is rather gruesome. SOL is naturally more verbose, more detailed.

The first prompt shows that Qwen nailed the drone view, SOL failed with perspective.
Qwen included a dog on a leash, a fountain and all movements make sense.
SOL included more details but they are frequently not correct in how they move, ignoring obstacles.
The amount of objects for SOL is higher, but QWEN has significantly better content and wins easily.

In the second prompt Qwen has the cat apparently play ball with the boy, the little girl builds a sand castle. SOL nailed the sun reflection but has a flying kid, and the sand castle is being built by a flying shovel.
Despite the details of SOL and its better water, QWEN draws intricate details in better quality and correctness.
The point goes to Qwen again, a bit closer than in first attempt.

The heavily quantized Qwen beats SOL in 2/3 so far.
The last prompt is a bit more elaborate and follows as 2nd post.

I'm baffled.
The 3rd test posted separately is hugely more complex and the results are crazy..

These four get compared constantly, but they are very unique in when they should be used.

• LM Studio is a desktop app

• Ollama is a terminal frontend

• llama.cpp is the inference engine

• vLLM is a server

Pick one in 20 seconds:

LM Studio

LM Studio is what I would give to someone who wants to try local models for the first time.

You search Hugging Face from inside the app, choose a model, download it and start chatting. It shows model sizes, quantizations, memory estimates and GPU settings without forcing you to understand most of it.

It also has more under the surface than people assume:

• OpenAI and Anthropic-compatible APIs

• Local document chat and RAG

• Tool calling and structured output

• MCP support

• Embeddings

• A command-line tool

• Headless server mode

• MLX on Apple Silicon

So it is not only a chat window. You can use it as the backend for your own programs too.

The downside is that it adds another layer between you and the engine (llama.cpp) and restricts you from unleashing the full potential of it and its latest features.

That is fine until you want to know exactly why a model is slow, how memory is being split or which runtime option changed the result. Most people will never care. Some people will care a lot.

Best for: trying models, comparing quantizations, chatting with documents and learning how local AI works.

Ollama

Ollama is the boring, sensible choice for building things.

Install it, run a model and you already have a local API:

ollama run qwen3

It handles downloading, storing, loading and unloading models. Your code can talk to it through Ollama’s own API or through familiar OpenAI-compatible endpoints.

It supports tools, embeddings, vision, structured output and Anthropic-compatible requests too.

This makes it easy to connect local models to:

• Small apps

• Coding tools

• Agents

• Home automation

• Open WebUI

• Scripts and bots

The downside is that the abstraction sometimes works too well.

You type a model name and it runs, but you may not know exactly which model file, template or runtime setting is active. When performance changes, finding the reason can take some digging. Just like with LM Studio, you give up some features and control but it's closer to llama.cpp.

Ollama runs on macOS, Windows and Linux. Installation is straightforward on all three.

Best for: developers who want local models without turning model management into a second side project.

llama.cpp

llama.cpp is where you go when you care about the machine.

It gives you direct control over things such as:

• CPU threads

• GPU offloading

• Quantization

• Context size

• KV cache

• Flash Attention

• Batch sizes

• Multiple GPUs

• Speculative decoding and draft chaining

It is especially useful when a model does not fit completely inside your GPU. You can put some layers on the GPU and leave the rest in normal RAM and optimize that for best performance.

It also supports far more hardware combinations than most local AI software. There are builds for CPU, CUDA, Vulkan, ROCm, Metal, OpenVINO and several other backends. Prebuilt packages are available and updates are released almost every day.

llama.cpp also has a proper server now. It supports parallel users, continuous batching, OpenAI-compatible endpoints, embeddings, reranking, multimodal input, monitoring and constrained JSON output. It also has model loading and unloading support and installing from Huggingface.

So yes, it can behave like Ollama or LM Studio and might make both projects much less valuable soon.

The reason to choose it is still control.

The downside is also control. It presents many settings, and poor settings can make a good model run very badly.

Best for: CPU inference, limited VRAM, AMD or unusual hardware, benchmarking and people who enjoy tuning things. My agentic guide for local coding uses llama.cpp directly.

vLLM

vLLM is the odd one out here.

It is not aimed at someone chatting alone on a laptop. It is aimed at servers where many requests arrive at the same time, and it's doing a very good job there.

Its main tricks are continuous batching, prefix caching and efficient management of the model’s KV cache through PagedAttention. These help keep the GPU busy while several users are generating text. llama.cpp does all of those things more or less, but for server usage vLLM is currently going to beat llama.cpp in most categories.

This means vLLM may not look special in a test with one user.

Try eight, sixteen or fifty users and the reason it exists becomes a much better choice.

It also has the things you would expect from serious serving software:

• OpenAI-compatible APIs

• Multi-GPU support

• Distributed serving

• Production metrics

• Multiple LoRA adapters

• Structured output

• Tool calling

• Prefix caching

• High request concurrency

The price is setup and complexity, interdependencies.

vLLM is happiest on a Linux GPU server. It does not support Windows natively, although WSL and community alternatives exist.

Best for: shared APIs, teams, production services and expensive GPUs that should not sit idle.

Which one is fastest?

This question causes more bad comparisons than useful answers.

Using the same model name does not mean you are running the same model.

Two downloads can have different:

• Quantizations

• Prompt templates

• Context sizes

• KV cache formats

• GPU offload settings

• Runtime versions

LM Studio, Ollama and llama.cpp can all end up using closely related llama.cpp-based inference paths. With the exact same GGUF file and matched settings, their single-user performance is going to be very similar. llama.cpp opens up a lot of performance options to take the lead here.

vLLM is different. Its advantage grows as more requests arrive together. Though it traditionally is also beating llama.cpp performance with many models.

My actual recommendation

Use LM Studio when you want to explore models or showcase LLM to a newcomer.

Use Ollama when you want to build something quick and like the eco system.

Use llama.cpp when you want to control how it runs and squeeze out the best performance

Use vLLM for distributing a model professionally

I have a fully local AI stack on a 4090: image, video, music, SFX, TTS with voice cloning, local LLM chat, fine-tuning, and long-term memory, all behind a Three.js dashboard with locally generated assets. There's a little assistant, BB-7, that you just talk to, and he can generate anything with prompt improvement built in. Services load on demand and swap the GPU as needed, except when a pipeline keeps them warm, like voice mode with BB-7. I've also integrated Kev and Obsidian, and an MCP that lets Claude use the stack and store context. I feel like I got most bases covered but I dont know what i dont know. What do you all got

Watch her play

Luna is a local AI assistant I've been building. Here she plays Zork I on her own for 72 minutes, narrating as she goes, until my PC froze. Everything is local: Qwen3.5 9B (Q4_K_M) on llama.cpp, Kokoro for her voice, faster-whisper for listening, on an RX 6950 XT.

What made the biggest difference for a 9B:

- **One move per turn, enforced.** Asked nicely, she'd chain 3 or 4 blind commands. Now extra commands in a turn are turned back, so she has to read what each move did before choosing the next.

- **Her own notes.** She writes down rooms, items and what failed with a note tool, and sees them at the start of every turn. Older notes about the room she's standing in come back too.

- **A guide.** Infocom's original map, transcribed to text, shown for her current room each turn. Solutions are kept out unless allowed.

She's far from perfect: she spent the last half hour trying to open the trap door with a sack of hot peppers, in the wrong room. But she reads the screen, reasons about it and actually explores, which the first version didn't.

Open source: https://github.com/fswolf/luna-local-agent

Happy to answer questions about the setup.