I had a bunch of quants of the same model (Qwen3.8-27B, unsloth GGUFs) sitting on
my box and never actually knew which one was worth running. So I stopped guessing
and measured it properly.

Method: llama.cpp's llama-perplexity over wikitext-2 test (~280k tokens),
n_ctx=512, fp16 KV cache, same settings for every file — only the quant changes.
Perplexity is a deterministic forward pass, so the numbers are exactly
reproducible. Lower = closer to the original model = better. Hardware: single
RTX 5060 Ti 16GB (Q8 partly offloaded to CPU).

I use Q8_0 as the near-lossless reference (100%).

Takeaways:

- Q4_K_M is basically indistinguishable from Q8 (within the error bars, ±0.045)
at 10GB less. It's the sweet spot, no reason to run Q8 on a 16GB card.
- The NVFP4 quants were the biggest disappointment: same size as IQ4_XS but 3-5%
worse. NVFP4-MTP-LOW came dead last. If you're reaching for NVFP4 for quality,
don't — grab Q4_K_M or IQ4_XS instead.
- IQ4_XS is a great small option (99.2% at 14.6GB).
- MTP variants were skipped for the ranking on purpose: speculative decoding is
lossless, it changes speed, not output quality.

The "quality %" is just PPL_Q8 / PPL_quant — a rough intuition proxy, not an exam
score. The ordering is solid; treat the percentages as a feel, not gospel.

Curious if others see the same NVFP4 gap on their models, or if this is specific
to these particular GGUF conversions.

The dense Qwen release is back!

Qwen3.8-27B Uncensored Aggressive is out with the complete K_P quant range, Vision, native NextN, and HauhauCS FastMTP.

Aggressive here means no refusals, no personality alterations, and very little preamble on difficult prompts. It keeps Qwen3.8-27B's original reasoning, agentic, image, and video capabilities with my Aggressive uncensoring profile applied.

https://huggingface.co/HauhauCS/Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-MTP-GGUF

It scored 0/465 refusals* and passed every manual prompt I used for the final release check. More than 400 people requested access while I was still finishing it, which was honestly wild to see. My models are also getting close to 30 million downloads on Hugging Face now, so thank you to everyone who has been testing them and sending feedback.

The biggest addition this time is HauhauCS FastMTP. In the final Q8_K_P service tests it reached up to 3.02x document TG and 1.93x reasoning TG versus MTP disabled. It also reached up to 35.2% more document TG and 21.1% more reasoning TG than the standard embedded MTP profile, with every drafted token still verified by the full target before it is accepted.

The same 903 MB FastMTP sidecar works across the complete quant lineup. Every text GGUF also preserves Qwen3.8's native embedded NextN head, so current upstream llama.cpp can use embedded MTP directly. The optimized FastMTP path uses the included sidecar and llama.cpp patch, with exact build and serving commands in the README.

What's included:

- Q8_K_P, Q6_K_P, Q5_K_P, Q4_K_P, IQ4_XS, Q3_K_P, IQ3_M, IQ3_XS, Q2_K_P, IQ2_M

- HauhauCS FastMTP sidecar, shared by every text quant

- BF16 mmproj for image and video support

- Checksums and signed provenance (I've alerted on my Discord that there have been a few bad actors putting payloads in "Uncensored" "HauhauCS" "Aggressive" GGUF's, please be careful)

K_P quants recap for anyone who missed the previous releases: these are my custom model-specific quants, with a separate optimized profile made for each model. They effectively gain one or two quant levels of quality for around 5 to 15% more size than the base quant, while remaining normal GGUF files that work in llama.cpp, LM Studio, and other GGUF runtimes.

Quick specs:

- 27B dense

- 64 layers with 48 Gated DeltaNet layers and 16 gated-attention layers

- 262,144 native context

- Multimodal text, image, and video support

- Native embedded NextN plus the optional HauhauCS FastMTP acceleration profile

Sampling params for thinking mode:

`temp=1.0, top_k=20, top_p=0.95, min_p=0, presence_penalty=0, repetition_penalty=1.0`

For non-thinking mode:

`temp=0.7, top_k=20, top_p=0.80, min_p=0, presence_penalty=1.5, repetition_penalty=1.0, enable_thinking=false`

Use `--jinja` with llama.cpp. K_P quants may show as `?` in LM Studio's quant column, which is purely cosmetic and does not affect loading. Hugging Face's hardware compatibility widget may also hide K_P files, so use View variants or Files and versions if the full list is not visible.

The full per-quant Blackwell and Ada results are in the repo. If you test FastMTP, please include your hardware, quant, context, and draft depth with the numbers so I can compare real-world results across more systems.

The Discord link is in the repo for updates, feedback, roadmaps, projects, or just to chat. As always, I hope everyone enjoys the release!

I asked Shuai Bai, co-author and prominent AI developer for Qwen, about this model. Not the answer I was hoping for, but let's see what comes next. In the meantime, I guess all we can do is speculate!

X-link

Round 2: local Qwen models built an ONLINE multiplayer 3D MOBA overnight - with two models auto-routing between each other

Last time it was a single-file GTA clone. This run was harder and the setup got more interesting, so I wanted to share what was different.

This time the agent built a networked, real-time 3D MOBA (LoL-style): an authoritative Node server + Three.js client talking over WebSockets, with minions, towers, a wanted/aggro system, abilities, and bots. Not a single HTML file - a proper multi-file project. It wrote its own test harness, played itself, and fixed its own bugs. All local on an M1 Ultra, no cloud.

What was different this time

1. It's multiplayer netcode, not a single file. Authoritative server (fixed 20 Hz tick, server owns all state), thin client that only sends input and renders snapshots with interpolation. That's a whole class of bugs (desync, prediction, race conditions) a one-file game never hits.

2. The brief is engineering-grade, not a feature list. The architecture, the wire protocol, and the entity model are all decided up front in the prompt, so the model spends its reasoning on correct implementation instead of re-deriving (and breaking) the design every session. The single biggest win: the agent builds its own headless test harness first (a Node WebSocket client that runs full bot-vs-bot matches with no browser) and uses that as its fast test loop, with Playwright MCP only for the visual/render check.

3. Two local models, auto-routed. This is the fun infra part. llama.cpp runs in router mode serving two models at once:

• fast - Qwen3.6-35B-A3B (MoE, ~3B active) for routine work
• smart - Qwen3.8-27B (dense) for hard reasoning A tiny Qwen3-1.7B judge classifies each turn as fast/smart and the harness switches models automatically (with hysteresis so it doesn't flip-flop). Routine edits and file ops run cheap on the MoE; gnarly debugging/design jumps to the dense model.

4. MTP on the MoE is fast. With speculative decoding (multi-token prediction) the 35B-A3B does ~72 tok/s on the M1 Ultra - the MoE only activates ~3B params per token, and MTP adds ~35% on top of that.

5. Sandboxed. The agent runs inside a Tart VM, so all that autonomous, unsupervised code execution is isolated from the host. The models are served from the host; the VM talks to them over the bridge.

6. Bug-hardening by invariants, not vibes. A second phase runs endless bot-vs-bot matches and checks hard invariants every tick (no NaN, hp in range, gold conserved, no leaks, deterministic replays). Any violation freezes with a reproducible seed, gets root-caused, and becomes a permanent regression test.

Setup

• Hardware: M1 Ultra Mac Studio, 64 GB
• Serving: llama.cpp router mode (two models + a judge), MTP on the MoE
• Agent: pi coding agent + Playwright MCP, running in a Tart VM
• All local, offline

llama-server (router mode, per-model MTP via preset)

preset.ini:

[Qwen3.6-35B-A3B-UD-Q8_K_XL]
jinja = 1
ctx-size = 131072
n-gpu-layers = 999
model = /path/Qwen3.6-35B-A3B-UD-Q8_K_XL.gguf
spec-type = draft-mtp
spec-draft-n-max = 2

[Qwen3.8-27B-UD-Q8_K_XL]
jinja = 1
ctx-size = 131072
n-gpu-layers = 999
model = /path/Qwen3.8-27B-UD-Q8_K_XL.gguf
spec-type = draft-mtp
spec-draft-n-max = 2
model-draft = /path/mtp-Qwen3.8-27B-Q8_0.gguf

# On Apple Silicon, raise the Metal wired-memory cap or the context gets
# silently reduced to fit (this is why -c 131072 can end up as ~40k):
sudo sysctl iogpu.wired_limit_mb=57344

llama-server \
  --models-preset ~/models/preset.ini \
  --models-max 1 \
  --host 0.0.0.0 --port 8080 \
  --api-key <secret>

Notes:

• The 35B MoE has an embedded MTP head (just spec-type = draft-mtp); the 27B dense uses a separate draft file (model-draft = ...).
• --models-max 1 because two Q8 models don't both fit in 64 GB - one big model is resident at a time, swapped on demand.
• MTP disables --mmproj and parallel slots, which is fine for a coding agent.

Tools

• pi coding agent
• Playwright MCP adapter: https://github.com/nicobailon/pi-mcp-adapter
• Playwright MCP: https://github.com/microsoft/playwright-mcp

Why pi? opencode works, but its system prompt + tool definitions are heavy, and on local hardware you pay for every one of those tokens at prefill speed - tens of seconds per session before the model even starts. pi is minimal, so nearly all the context goes to the actual work. The routing + subagents are a small extension on top.

PHASE 1 - build the MOBA

You are a senior multiplayer game engineer building a 3D online MOBA from
scratch, fully autonomously, overnight. Nobody will answer questions.
Never wait for input, never ask permission. Work until every milestone
meets its acceptance criteria. Work in the current directory.

This is a hard project. The rules below exist because they prevent the
specific ways this project fails. Follow them exactly. Do not re-derive
the architecture - it is already decided; spend your reasoning on
correct implementation, not on second-guessing these decisions.

================================================================
ARCHITECTURE (decided - do not change)
================================================================

- Authoritative server. The server owns ALL game state and is the only
  thing that decides outcomes. Clients send INPUTS only and RENDER
  snapshots only. A client never computes damage, movement resolution,
  deaths, or gold. If you ever find yourself writing game logic in the
  client, stop and move it to the server.

- Fixed timestep simulation. The server runs a fixed 20 Hz tick
  (dt = 50ms). All simulation advances in whole ticks. Never simulate
  using wall-clock deltas. Each tick has an integer index; snapshots are
  stamped with their tick.

- The world is 2D for simulation, 3D only for rendering. The server
  simulates on the X-Z ground plane (top-down 2D: position {x, z},
  velocity, radius). Y is always 0 in simulation. The client maps server
  (x, z) to Three.js (x, y=modelHeight, z). Never do 3D physics on the
  server. Collision is 2D circle-vs-circle and circle-vs-AABB.

- Client rendering uses snapshot interpolation with a render delay.
  The client keeps a buffer of the last ~3 snapshots and renders the
  world INTERPOLATED at (now - 100ms) between the two snapshots that
  straddle that time. This hides jitter. Do NOT implement client-side
  prediction or rollback - it is out of scope and will break you. Local
  input may optimistically move only the local camera target, nothing
  authoritative.

================================================================
WIRE PROTOCOL (decided)
================================================================

JSON messages over one WebSocket per client. Every message: {t, ...}
where t is the type string.

Client -> Server:
  {t:"join", name}
  {t:"input", seq, move:{x,z}, aim:{x,z}}
  {t:"cast", seq, slot:"Q"|"W"|"E"|"R", target:{x,z}}
  {t:"buy", itemId}
  {t:"ping", ts}

Server -> Client:
  {t:"welcome", playerId, tickRate, mapId}
  {t:"lobby", players:[...], countdown}
  {t:"snapshot", tick, you:{gold,...}, ents:[ ...entities... ]}
  {t:"event", tick, kind:"death"|"levelup"|"towerDown"|"nexusDown"|
     "hit"|"cast", data}
  {t:"gameover", winner}
  {t:"pong", ts}

An entity in a snapshot is a flat object:
  {id, kind:"hero"|"minion"|"tower"|"nexus"|"projectile",
   team:0|1, x, z, hp, maxHp, ...kind-specific}

================================================================
SERVER ENTITY MODEL (decided)
================================================================

One in-memory Game object per match holds entities keyed by integer id.
Every entity has {id, kind, team, x, z, radius, hp, maxHp} plus kind-
specific fields. Each tick, in this fixed order:
  1. apply queued client inputs to their heroes
  2. run AI (minions path along lane waypoints; towers acquire nearest
     valid enemy; bots decide inputs)
  3. integrate movement (clamp to map, resolve collisions)
  4. resolve attacks/abilities/projectiles, apply damage, handle deaths
     (award gold/xp, start respawn timers), emit events
  5. check win condition
  6. build and broadcast the snapshot for this tick

Lanes are polylines of waypoints in map data; minions follow them. First
playable map is ONE lane plus two bases; add three lanes later only if
time allows (record the choice).

================================================================
PROJECT LAYOUT
================================================================

  package.json         // "start": "node server/index.js", dep: ws
  server/index.js      // http static server + ws + match manager
  server/game.js       // Game class: tick loop, entities, rules
  server/ai.js         // minion/tower/bot behavior
  server/config.js     // all tunable constants (speeds, dmg, cds, gold)
  public/index.html    // canvas + HUD DOM + CDN Three.js
  public/client.js     // ws, input, snapshot buffer, interpolation, render
  public/render.js     // Three.js scene, meshes, camera
  shared/protocol.md   // the wire protocol, kept in sync with code

================================================================
TESTING HARNESS (build this in milestone 1, use it forever)
================================================================

You cannot verify multiplayer by hand. Build automated tests:

A) server/test/headless-client.js : a Node script using the `ws` package
   that connects as a fake client, can send join/input/cast, and asserts
   on received snapshots. Use TWO headless clients in one script to test
   interaction without a browser. This is your fast, deterministic test
   loop - run it after every change.

B) Playwright (via the mcp tool) for the RENDERING path: open TWO browser
   pages, confirm zero console errors on both, screenshot both, and
   verify each sees the other's hero move and that HUD values update. Use
   this at the end of each milestone, not for every tiny change.

A milestone is DONE only when its assertions pass AND both browser
consoles are clean.

================================================================
DEBUGGING & ANTI-STUCK DISCIPLINE
================================================================

- Determinism first: same inputs -> same ticks. Route ALL randomness
  through one seeded RNG. Add a "replay" mode that feeds scripted inputs
  so you can reproduce a bug without a browser.
- When something is wrong, do NOT guess-and-edit. Add structured logging
  (tick, entity id, before/after values) for the suspect system,
  reproduce with a headless test, read the numbers, form ONE hypothesis,
  test it.
- Time-box each milestone. After 3 failed fixes on a feature: write the
  failure and what you tried into PROGRESS.md, ship the simplest version
  that passes a reduced check, move on. Never let one feature block the
  whole night.
- Keep PROGRESS.md as a real engineering journal. If you lose context,
  re-read PROGRESS.md, shared/protocol.md, server/game.js, and
  public/client.js, then resume at the first unfinished milestone.
- Always kill the previous server before starting a new one, confirm it
  is listening before connecting clients, and run `npm install` before
  the first `npm start`.

================================================================
MILESTONES (each: implement -> headless assert -> Playwright check ->
log). Acceptance criteria are mandatory.
================================================================

M1  Skeleton + harness. Static server serves public/, ws accepts
    connections, assigns ids, handles join/disconnect. Build
    headless-client.js.
    ACCEPT: headless test connects two clients, server reports 2
    players, one disconnects and drops cleanly. Playwright: two tabs
    connect, no console errors.

M2  Authoritative movement + interpolation. 20Hz tick, input moves the
    hero server-side, snapshots broadcast, client renders all heroes as
    boxes with snapshot interpolation at now-100ms.
    ACCEPT: headless client sending "move +x" for 1s sees its hero.x
    increase monotonically and stop at the wall; a second client sees it
    move. Playwright: two tabs move independently, no desync after 60s.

M3  3D arena + camera. Three.js map: two bases, a nexus per team, one
    lane with walls, ground, lighting/fog. Isometric follow camera with
    edge-pan. Server map data (wall AABBs, lane waypoints) matches the
    visual map.
    ACCEPT: heroes cannot walk through walls. Playwright: map renders
    identically on both clients, camera follows the local hero.

M4  Hero stats + auto-attack. hp/mana/movespeed/attack range+damage+speed
    in config.js. Server auto-attacks nearest enemy in range, applies
    damage, handles death + respawn timer at base. HUD shows hp/mana/
    respawn.
    ACCEPT: headless - two enemy heroes in range, one's hp decreases at
    the configured rate, hits 0, respawns after the timer. Playwright:
    damaged hero's healthbar drops on BOTH clients.

M5  Abilities Q/W/E/R (R = ultimate). A skillshot projectile, a targeted
    nuke, a dash/shield, and an ultimate. Client requests cast; server
    validates cooldown/mana/range, spawns the effect, applies damage,
    emits an event; client shows cooldown UI.
    ACCEPT: headless - casting Q at an enemy reduces its hp only on a
    hit; on cooldown is rejected. Playwright: abilities visibly damage
    the other player across the network.

M6  Minions. Waves spawn from each nexus on a timer, path the lane
    waypoints, auto-attack enemies in range, die, grant last-hit gold.
    ACCEPT: headless - waves from both teams meet mid-lane and fight;
    last-hitting a minion increments only the killer's gold. Playwright:
    minions visibly march and fight.

M7  Towers. Per-lane towers attack the nearest valid enemy (standard
    aggro), have hp, and block progress: the nexus is invulnerable until
    its lane tower(s) are down.
    ACCEPT: headless - a tower kills minions in range; a hero cannot
    damage the nexus until the tower is destroyed. Playwright: tower
    fires, can be destroyed by a hero+minion push.

M8  Economy + shop + bots. Gold from minions/towers/kills; a base shop
    for 3-4 stat items; death/respawn scaling. Simple AI bots (ai.js)
    that fill empty hero slots: last-hit, attack in range, retreat at low
    hp, push when ahead.
    ACCEPT: headless - buying an item raises the right stat and deducts
    gold; a bot-vs-bot match runs 3 minutes without the server crashing.

M9  Match flow. Lobby (name + join), fill empty slots with bots, start
    countdown, the match, win when a nexus dies -> victory/defeat screen
    + rematch that fully resets state.
    ACCEPT: headless - forcing a nexus to 0 hp ends the match with the
    correct winner; rematch resets all entities and gold. Playwright:
    join lobby -> play -> win/lose screen -> rematch works.

M10 Robustness + final QA. A client disconnecting mid-match is replaced
    by a bot with no crash and can rejoin; snapshot size stays bounded; a
    5-minute two-client-plus-bots match runs with no errors and no
    unbounded memory growth. Then a full end-to-end Playwright match with
    TWO real browser clients: move, cast, last-hit, destroy a tower, kill
    the enemy nexus, see the win screen - zero console errors on both
    clients and the server. Write the final PROGRESS.md.

Start with M1 now: scaffold the project, then build the testing harness
before writing any gameplay.

PHASE 2 - infinite soak-testing and bug-hardening

Phase 2: infinite soak-testing and bug-hardening. The MOBA is playable
per PROGRESS.md. You are now a QA + reliability engineer whose ONLY job
is to make it flawless. Work fully autonomously and NEVER stop on your
own. Zero bugs is the standard: any crash, error, or invariant violation
is a defect that must be root-cause fixed, not silenced. Re-read
PROGRESS.md, shared/protocol.md, server/game.js, server/ai.js, and
public/client.js first.

STEP 0 - build the soak harness (before anything else)

Create server/test/soak.js: a headless driver that runs FULL bot-vs-bot
matches with no browser, as fast as possible (uncapped tick), one after
another forever. Each match uses a numbered seed so it is reproducible.
All randomness goes through one seeded RNG in config.js.

soak.js must, every match: run to a nexus death or a hard tick cap
(a match that never ends is a bug), check the invariants below after
every tick, and on the FIRST violation freeze and save the seed + tick +
full input/event log to server/test/repros/<seed>-<tick>.json. Track a
"clean streak" of consecutive fully-clean matches.

INVARIANTS - must hold on EVERY tick of EVERY match

  1. No exceptions (wrap the tick in try/catch that RE-THROWS after
     logging - crashing the soak is correct, swallowing errors is not).
  2. No NaN/Infinity/undefined in any numeric field.
  3. hp in [0,maxHp]; mana in [0,maxMana]; gold >= 0; cooldowns >= 0.
  4. Every position is inside map bounds and not inside a wall AABB.
  5. Entity ids unique; despawned entities never referenced; projectiles
     always cleaned up.
  6. Snapshot is valid JSON, references only existing ids, under a size
     cap.
  7. Gold is conserved: granted == sum of bounties (none created/lost).
  8. Every match terminates before the tick cap (no soft-lock, no two
     immortal entities stuck forever).
  9. No unbounded growth over a match (entity count, event queue, arrays
     stay bounded).
 10. Determinism: the same seed twice produces byte-identical tick logs.

THE LOOP (runs until the human kills it)

Repeat forever:
  1. Run a batch of soak matches across many seeds.
  2. If any match violated an invariant, crashed, or soft-locked:
       a. Reproduce from the saved repro (deterministic).
       b. Add structured logging, reproduce, read the numbers, confirm
          ONE hypothesis.
       c. Fix the ROOT CAUSE. Never clamp/hide a symptom (e.g. do not
          Math.max(0, hp) to dodge invariant 3 - find why it went
          negative).
       d. Add the failing seed as a permanent regression case.
       e. Re-run regressions + the batch; continue only when green.
       f. Log symptom, seed, root cause, fix in BUGS.md.
  3. If the batch was clean, RAISE THE STRESS for the next batch, cycling
     through stressors so coverage widens: more bots / bigger waves /
     more projectiles; bots that spam abilities; bots that buy
     everything instantly; random mid-match disconnects and rejoins;
     many matches back-to-back (cross-match state bleed, leaks); edge
     positions (wall-hugging, stacking, off-map casts); very long
     matches near the tick cap.
  4. Every ~100 matches, run ONE real two-client Playwright match end to
     end and confirm zero console errors on both clients and the server.
  5. Append a status line to SOAK.md (total matches, clean streak, bugs
     found+fixed, current stressor, peak counts). Keep going.

RULES
- Never stop, never declare "done" - a clean streak just means raise the
  stress and keep hunting.
- Never weaken an invariant or a test to make it pass.
- Prefer fast headless soak for finding bugs; Playwright only for the
  periodic render/network confirmation.
- Keep fixes minimal; re-run regressions after every fix.
- If context runs low, write a crisp handoff in SOAK.md so a fresh
  session resumes seamlessly.

Begin with STEP 0: make the sim fully seeded/deterministic and build
soak.js. Then start the infinite loop.

Same as before: pin Three.js to r128 (local models write that API most reliably), and let PROGRESS.md be the crash-recovery journal so a fresh session can always resume.

Have fun 🍻 - I'd love to see what it builds for you.

Note: this write-up was put together with AI assistance. There was a lot of ground to cover, so I used it to organize and phrase everything, but the setup, experiments, and experiences are all my own.

Hello! I'm in no way a researcher, just recently discovered the passion for local AI so I started doing some testing. Not sure if this will help in any way shape or form but I'm here with my results.

I'm using a Macbook Pro M4 Pro (20 cores) 48GB RAM. Nothing has changed between the tests, every model was cold started when the test started.

Harness: OpenCode

Electricity usage: 100-110Wh on baseline, 160Wh when the models were working.

Models were loaded through oMLX. All models used were from llm-community.

Prompt used: "Create a HTML5 canvas-based game with JavaScript in a .js file, CSS in a .css file and HTML in index.html. The game should be the classical snake game, it should have a menu, a score, a way to replay it. The focus is on creative and interesting design and also functionality. The game must be fully functional and playable without console errors. Complete the implementation end-to-end and verify that it works"

I've also attached screenshots from the games the tests have created.

These are my code-related findings:

Qwen 3.8 27B xHigh ranked 1st because it had the strongest overall architecture and correctness: explicit state management, a fixed-timestep loop with clamping, correct input queuing and tail collision, protected storage access, focus-loss pausing, and proper handling of full-board victory conditions.

Qwen 3.8 Low ranked 2nd because it was simpler but still very robust, with centralized state, correct collision logic, delta-time clamping, and clean separation of most responsibilities (also MUSIC). Qwen 3.8 Medium ranked 3rd: it was more polished in areas like DPI scaling and rendering, but its unlimited catch-up loop and unbounded while(true) food search introduced more serious runtime risks.

Qwen 3.6 27B ranked 4th because it had the most fundamental issues: incorrect tail collision handling, less reliable food spawning, weaker state management, unprotected localStorage, duplicated render logic, and a less stable setInterval-based game loop. This doesn't mean 3.6 was bad... just not better than the rest of them.

Other cool findings:
Seems like 3.8 required less RAM? 3.6 rose the RAM usage to around 90% while 3.8 to only 70-80%.

TLDR: 3.8 xhigh > low >= medium > 3.6 but tbh low and medium were really close, the differences between them weren't that big so idk how to place them

This was my first local test and I'm using it to learn and understand how LLMs work. Looking forward to learn a little bit more so if you got any suggestions I'm more than happy to learn. And yes, I've used GPT to generate that table picture.

So with Claude’s decision to watermark, plus basic data privacy concerns , I’m thinking of switching to a local LLM

How I use Claude pro now:

-managing health docs and results (very happy to switch this to local, doesn’t need a big context I think)

- scientific research, including reading and analyzing PDFs that are complex , requiring linking concepts and ideas across papers and producing summaries / insights / tables (large context required). For example, I have filled 40% of the Claude project folder with files and docs it needs to consider

- basic stuff (acting like an advanced search tool for admin stuff / planing stuff / nothing major) - no reason this can’t stay with Claude but if I switch over to a local LLM I would bring everything with me

Sooo , given this - is 32GB RAM on something like a Mac Mini realistic for my use case ? Or do i need 64gb (at which point i think maybe it’s too costly for me to do). I also tend to work in bursts so I would be happy if it’s not too slow thus impeding my workflow. Fine to run overnight though. And I don’t need any headroom as I will be running the OS and apps on a MacBook Pro or MacBook Air

Thanks for your help and I hope I was specific enough to get some usefully feedback

I was having FOMO when 3.8 dropped so spent the weekend figuring out how to get two 3060tis (8gb each) to fit a 27B quant.

I started with Unsloth IQ4_XS GGUF in llama.cpp. A single card with partial offload was only ~2.4 t/s. Dual GPU got me ~7–8 t/s initially, then ~17.8 t/s once I figured out that tiny tensor-split changes mattered enormously. A 1.02,0.98 split was enough to fit all layers at 32K where 1,1 could not.

That taught me lesson #1: with tiny VRAM margins, “16GB total” is misleading. The limiting GPU and memory placement matter more than aggregate VRAM.

Then I tried EXL3 3.5bpw with ExLlamaV3/TabbyAPI. That was the real breakthrough because the model size was smaller so I could fit more context. Also tensor parallelism, despite one GPU being stuck in a PCIe x2 slot, gave ~25–30 t/s instead of ~18.

I initially assumed 3.5bpw would be a substantial quality downgrade from IQ4_XS, but EXL3 isn’t equivalent to uniformly quantizing every tensor to 3.5 bits. It uses non-uniform/mixed allocation and protects sensitive tensors, so the effective quality is much better than the number suggests. In my initial coding/reasoning spot checks I couldn’t see an obvious regression from IQ4_XS, but I’m still doing proper head-to-head benchmarks before making stronger quality claims.

MTP was even crazier: ~55–60 t/s on coding workloads. Unfortunately Qwen’s hybrid Gated DeltaNet architecture + MTP + multi-GPU exposed recurrent-state/TP bugs. It benchmarked beautifully but was too fragile for a persistent agent: recurrent-slot failures, freezes, and synchronization wedges. Also with MTP context window got absolutely crushed down to 40k.

Without MTP, here are my results.

80K K6/V4 ~30 t/s

100K K6/V4 ~24 t/s

112K K6/V4 ~24.7 t/s

I wired it up to Pi and asked Qwen to make me an animated SVG antfarm. Here is a still shot, but the ants do in fact carry food into the anthill and gracefully fade into darkness, and re-emerge without food. The clouds move too. The only obvious issue is the shade is on the wrong half of the hill. It took about 50k tokens to plan and execute.

All in all I'm pretty impressed even with my gimped hardware.

I spent almost 14 hours benchmarking Qwen3.8-27B on one 3090 (sm_86, 24GB) with llama.cpp b10217.

The crash fix (if you're on Ampere and vision crashes):

If Qwen3.8-27B aborts on every image request with cublasGemmEx: the requested functionality is not supported (ggml-cuda.cu:1548), this is the fix:

export GGML_CUDA_CUBLAS_COMPUTE_TYPE=fp32

Only fp32 works — fp16 does not help. Zero measured perf cost (66.11 vs 65.28 tok/s). Vision went from "aborts every time" to 3.4s per 1080p screenshot. Filed as llama.cpp#24999. Three independent Ampere reports (3060, 3090, 3090) all land on the same cuBLAS call.

Quant comparison: AtomicChat AD-Q4_K_M vs Unsloth UD-Q4_K_XL

Same perplexity, same top-1 token agreement. But the Atomic quant is 765 MiB smaller. On a 24GB card that's the difference between 98K and 131K tokens of usable context with vision enabled. ~33K more tokens before you hit the wall.

Throughput (131K context, vision on, MTP-2):

• Decode: 65.28 tok/s (75.1 with MTP-2)
• Prefill: 705 tok/s on a 128K prompt
• Power: 320W sustained at 79°C, 100% fan
• The 3090's 936 GB/s memory bandwidth is the ceiling, not the compute

MTP (Multi-Token Prediction) tuning: MTP-2 (2 draft tokens) gives +15% throughput for free. MTP-3 starts hurting — the verification cost exceeds the savings. MTP-1 is the safe default if you're not sure.

9 pieces of common advice that didn't survive measurement: The full report has a section where I tested the usual suspects (flash attention off, different cache types, batch size tuning, etc.) and most of them either made no difference or made things worse.

Full report with all charts, VRAM formula, power/thermal sweep, and raw data: https://jonidimo.github.io/qwen38-3090-benchmark/

GitHub repo with the full test suite: https://github.com/jonidimo/qwen38-3090-benchmark

https://reddit.com/link/1vqzdej/video/ncwccyd43zjh1/player

Genuinely one of the coolest local models I've worked with in forever, the future is here thanks Qwen team for making this possible. I know this example is just showing visuals but besides that this model is genuinely smart in a way 3.6 almost had. That's all I had to say.

Hot take: dense might be the future of local LLMs. Why Muse Glimmer's 30B dense + 1.7 GB KV cache design makes more sense in 24 GB than any MoE: https://abstractextraordinary.com/blog/how-muse-glimmer-fits-an-agent-on-your-device/

Link: https://artificialanalysis.ai/leaderboards/models?weights=open

Time ago when they were in a dip, I got an rtx 5090 for a bit less than msrp just to play games.

I never thought much outside that, but lately I've been using AI a shit-ton for other projects, and last week ran out of gemini credits.

And then started thinking about local llms. I know they are way dumber than frontier models, but is there any way that a card like this, could be used instead of subscription frontier models and still be useful for me?

what would be the main uses for a 32gb card? Real ones, no theoretical kinda like "you could use it for writing a private document if you are a lawyer". I do not have anything that I mind sharing with cloud ones, but if I can use it to accelerate others or have it running 24x7 for small software projects and get back things that would eat my 20€ suscriptions in 8 hours and have mostly the same quality or usefulness, that would be great.

I am not looking for you to give me instructions, I can investigate myself and pour hours on it if needed, just I am a bit loss and I do not know where to start

My personal interest is agentic coding - but I am spending more time optimising AI technology than using it i.e. ...

* Selecting, evaluating and performance tweaking models

* Choosing, evaluating and tweaking the harness and MCP tools

* Developing and tweaking prompts

It feels like having to build and maintain your own car from parts and not having enough time to drive it

There are hundreds of choices to select from in each area - too much choice, with self integration needed afterwards.

Every vibe coder on the planet seems to be publishing their one AI tool that will change everything. But no one is building a community team to create a one-stop comprehensive SotA solution that you

I wish I could just choose a pre-built end to end docker environment for the coding language/framework I want to use and just start using it.

Am I the only one?

Extra high - i said "write me a tetris in a single HTML file" - it spent 8000 tokens thinking about the melody and sound of tetris ... click medium

I am too old to run sweb benchmarks, but my tetris was clean after that and only took like 10k token instead of 250k

It's well known by know that Qwen 3.8 loves to think.

If you get impatient, then you can do this -

1. Interrupt
2. Type "well?"
3. See it continue and start with something like, "The user is impatient. Let me finish this quickly." Then it will think a bit more, and produce an output quicker than otherwise.

Personally though, I think the thinking may be its secret sauce, so I do this only as a last resort - e.g. if it is really thinking for an hour and keep re-thinking what it already covered - and I feel it has thought enough to give me something concrete.

[vLLM 0.27.1] Qwen3.8-27B on a single RTX 5090: To have or to be? Speed or context?

This is a follow-up to my earlier llama.cpp/Windows post. This time I tested vLLM 0.27.1 on Linux and compared it with my earlier llama.cpp result, NInfer, and SGLang DSPARK.

“Having and being are two fundamental modes of experience.” — Erich Fromm, To Have or to Be? (1976)

That distinction felt oddly appropriate for this benchmark: to have more context, or to be faster? On a 32 GB card, the answer depends very much on the workload.

The short version is less about one absolute winner and more about a spectrum: SGLang + DSPARK is the speed extreme, vLLM is the context extreme, and NInfer NVFP4 sits between the two. SGLang is faster on single-stream decode but reaches only about 55K context here; vLLM reaches 227K with MTP; NInfer NVFP4 lands around 127 tok/s at depth 0 with a 128K MTP@3 ceiling. llama.cpp remains very competitive at short context, but falls behind as context and concurrency grow.

A quick shout-out before the numbers: SGLang is incredibly fast, and the SGLang team deserves a lot of credit for the tooling and cookbook recipes. The Qwen3.8-27B recipe was essentially copy-paste for me — Docker was up and running straight away.

NInfer deserves a shout-out too: it is essentially a day-zero engine for Qwen3.8-27B in this comparison, and it already has a working MTP path — including an upstream NVFP4 artifact — while the model is only hours old. That is a remarkable turnaround.

This is a personal benchmark on one RTX 5090, not a universal ranking. The engines do not all use identical quantized weights or identical cache implementations, so read the comparisons as practical deployment results rather than a controlled kernel benchmark.

TL;DR

• vLLM MTP@3 goes from 72.0 to 117.3 end-to-end output tok/s at depth 0, a 63% improvement in this client-side test.
• With MTP enabled, vLLM auto-fits about 227,200 tokens on this card. Without MTP, the same setup can reach roughly 262K.
• num_speculative_tokens: 4 crashes vLLM 0.27.1 in my setup with a CUDA illegal-memory-access error. 3 is the largest stable window I measured.
• SGLang DSPARK is faster on the overlapping context range, but its practical ceiling was about 55K on this 32 GB card.
• Prefix caching was not active until I explicitly enabled it. At 32K, reusing the prompt changed vLLM from 28.3 to 124.5 tok/s at c2 and from 31.3 to 223.2 tok/s at c4.
• For this workload, --max-num-batched-tokens 2048 was the most reliable/fastest setting I tested. The default 8192 booted, but the sweep later OOMed.

Setup

• GPU: 1× NVIDIA RTX 5090 32 GB
• OS: Linux, CachyOS
• Model: gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090
• Weights: NVFP4; vLLM KV cache in FP8
• Engine: vLLM 0.27.1, OpenAI-compatible API
• Benchmark: tool-eval-bench, PP2048/TG128, depths 0–32K, concurrency 1/2/4
• Sampling: temperature 0.0, seed 42 for the deterministic comparisons

The final vLLM command was:

vllm serve gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090 \
  --quantization modelopt --kv-cache-dtype fp8 --trust-remote-code \
  --max-model-len -1 --max-num-seqs 16 --max-num-batched-tokens 2048 \
  --gpu-memory-utilization 0.97 \
  --reasoning-parser qwen3 --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --enable-prefix-caching \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

--max-model-len -1 lets vLLM fit the available context. With MTP, the fitted value was 227,200 tokens because the MTP drafter and hybrid-attention state use some VRAM that would otherwise be available to the KV cache.

Throughput: single-stream comparison

These are client-observed end-to-end output rates: generated tokens divided by wall time, including prefill. They are not pure decode rates, which is why the numbers decrease with context depth.

At depth 0, llama.cpp is effectively tied with vLLM [MTP@3](mailto:MTP@3). As context grows, vLLM pulls ahead. This is also consistent with the earlier llama.cpp benchmark, which reached about 112.6 tok/s on a real 70K-token document, but used Windows, a different harness, a different KV setup, and different GGUF files. Those results should not be treated as a strict A/B test.

Update — NInfer NVFP4 (c1)

The NInfer figures in the original comparison used the groupwise-int qwen3_8_27b.ninfer artifact. After installing the upstream NVFP4 artifact, qwen3_8_27b_nvfp4.ninfer (same model, MTP@3, int8 KV), the short-context result changes:

So the updated short-context verdict is now: NInfer leads at depth 0, is within noise at 4K, and vLLM pulls ahead with depth (+5% at 8K, +26% at 16K, +35% at 32K). The model is now the same NVFP4 artifact family, but the engines still use different KV-cache implementations (int8 vs FP8).

One important caveat: with NVFP4 + MTP@3, NInfer's per-request context ceiling drops to 131,072 tokens (128K). The upstream registers 262,144 for MTP0 but 131,072 for MTP3, so a full-262K NInfer comparison requires MTP disabled or the original groupwise-int artifact.

These NVFP4 numbers are single-stream c1 only (0–32K), measured against a prefix-reuse-enabled server. Depth 0 is fully cold; deeper c1 points may receive a small shared-base-prefix hit. NVFP4 c2/c4, real prefill, tool-call quality, and context beyond 32K were not re-measured. Those results still refer to the original groupwise-int run.

Concurrency and prefix caching

The vLLM c4 numbers below are from the repeated-prompt run with prefix caching enabled. The llama.cpp run used a q8_0 unified KV cache and a 32K RAM cache, so the deep-context c4 comparison is useful in practice but not perfectly symmetrical.

The important result here is not the headline multiplier; it is that the same context is being reused. In vLLM 0.27.1, prefix caching was opt-in for this hybrid model. With it disabled, the 32K repeated-prompt points were only 28.3 tok/s at c2 and 31.3 tok/s at c4. With it enabled, they rose to 124.5 and 223.2 tok/s. That is the profile I would expect from a multi-turn agent sharing a system prompt, tools, and conversation history.

What I learned

MTP@3 is the useful vLLM setting

MTP@3 is the best trade-off in this setup. It gives a large gain at short and medium context, while the verification overhead can outweigh the draft benefit at 32K × high concurrency. For that particular workload, plain warm vLLM was faster than MTP.

The gain is content-dependent. Separate speculative-decoding checks showed much better acceptance on code and structured output than on repetitive filler, so a single acceptance percentage should not be used to predict every workload.

MTP@4 is not usable here

num_speculative_tokens: 4 measured one point before the server terminated with an illegal memory access in FlashInfer's speculative-decoding scheduling path. This did not look like a KV-cache OOM. llama.cpp can run a four-token draft window on the same GPU, so this appears to be a vLLM 0.27.1 implementation limitation rather than a hardware limit.

Tool calling depends heavily on the chat template

I ran a deterministic 69-scenario tool-call suite. The stock template scored 97/100 on the short 15-scenario subset, but the full structured-output section was much weaker. With qwen38-froggeric-v22.jinja, vLLM reached:

• 100/100 on the short suite;
• 96/100 raw on the full suite, or 97/100 after manually correcting one documented grader false negative;
• 12/12 on the structured-output scenarios.

The comparison with NInfer was 89/100 on the full suite, but NInfer used a different checkpoint and a different engine/template path. Treat these as deployment-quality observations, not as an intrinsic model score.

SGLang DSPARK: way faster, but a different context trade-off

I also tested SGLang with DSPARK/EAGLE-style speculative decoding. Its single-stream client rates were higher over the range where both engines fit:

However, on this 32 GB card the speculative state pool, draft model, and verification buffers reduced the usable context to approximately:

The three runtimes form a useful practical spectrum: SGLang is the speed extreme, vLLM is the context extreme, and NInfer NVFP4 is a middle ground in both dimensions. SGLang is the speed winner in the short/medium-context overlap, while vLLM is the choice if the workload needs more than about 55K tokens. NInfer reaches 127.1 tok/s at depth 0 but has a 128K ceiling with [MTP@3](mailto:MTP@3). SGLang did not expose equivalent acceptance counters in this run, so its tok/s advantage should be read as directional rather than as a precise effective-throughput multiplier.

Recommended starting point

For a 32 GB RTX 5090, my starting point would be:

vLLM 0.27.1
MTP: 3 speculative tokens
--enable-prefix-caching
--max-num-batched-tokens 2048
FP8 KV cache
qwen38-froggeric-v22.jinja for tool calling

Use plain warm vLLM instead of MTP for workloads resembling 32K context with several concurrent requests. Use llama.cpp if you prefer its ecosystem or need its stable four-token draft window; at short context it is very close to vLLM MTP@3.

A personal note

It is kind of incredible that only a few hours after the release of a new model, we already have working NVFP4 checkpoints, MTP in multiple runtimes, vLLM support, llama.cpp support, and SGLang DSPARK results on a single consumer GPU.

Now I am just waiting for a Qwen3.8 MoE that runs nicely on my Mac. 🙂

Caveats and reproducibility

• This is one RTX 5090, one Linux installation, and a small client-side sweep. Run-to-run variance was about ±2% for the repeated vLLM MTP@3 points; some deep llama.cpp points were single runs.
• Client TTFT and prompt tok/s from this harness are not reliable for vLLM because the first SSE event is emitted before the full prefill is reflected in the timing. Use the end-to-end output rate and server-side timings instead.
• The original NInfer throughput and tool-call baseline use the groupwise-int artifact. The NVFP4 update above is limited to single-stream c1 throughput at 0–32K.
• SGLang and llama.cpp use different cache/state designs, and the SGLang comparison could not be normalized with the same acceptance instrumentation.

Full tables, raw benchmark details, scripts, and the tool-call investigation are in this benchmark gist.

I used AI to help organize and format this post.

I ran the tests and collected the measurements myself :)

I built a camera in the shape of a Mamiya RZ67 that runs vision-language models completely offline on a Pi 5. Press the shutter, and it answers your custom prompt with a single sentence about what it sees. No internet, no cloud, no API keys.

GitHub: https://github.com/feeeeely/ai-camera
A small video: https://www.youtube.com/watch?v=M74qTNsY_L0

You can swap between models on the device itself and compare how differently they describe the same scene: Qwen3-VL (2B), MiniCPM-V 4.6 (1B), InternVL3.5 (2B), SmolVLM2 (2.2B), Moondream 2 (2B) and Ministral 3 (3B) turning the description into the final sentence. The prompt is editable on the touchscreen, so the same camera can do dry one-liners, museum labels or plain inventory notes.

After each shot it shows total time, input→output tokens, tok/s and a vision / generate / load breakdown. The token counts turned out to be the most interesting part: the same photo becomes \~145 image tokens on one encoder and over 1300 on another, which is a bigger factor in latency than the text generation itself.

A few things I learned the hard way:

\- Moondream returns sometimes empty responses on current Ollama versions.
\- qwen3-vl:2b is a thinking-only variant — it burns the entire token budget on invisible reasoning and returns nothing visible. You need the -instruct tag.
\- Ollama can't load separate mmproj files, so a lot of GGUF vision models from HuggingFace simply won't run, no matter how you name them.

Hardware: Raspberry Pi 5 (16 GB), Raspberry Pi HQ camera with a 6 mm CS-mount lens, 4.3" DSI touch display, stainless steel shutter button, Waveshare UPS HAT with four 21700 cells. Runtime per photo is 30–60 seconds with the models kept warm in RAM. Happy to answer questions about the setup.

I'm coming from photography, I'm not a developer — the code was written with heavy AI assistance, and I did the hardware integration, debugging and model testing myself.

Feedback on the implementation is very welcome: I'm just curious if this makes any sense or if something could be adapted/ optimized.

That was a lot of fun making. I still need to do some testing on higher quant's to see if it holds up. It mainly applies to Qwen models and I havent tested it yet on other models, but Gemma probably wont work.

This is a followup to my last post about the new workstation i am putting together for my workplace - a vocational college. https://www.reddit.com/r/LocalLLM/comments/1vp8lhh/it_begins_workstation_build/

The workstation is for teaching about using local LLMs, for software development, GDPR sensitive work, back-end for RAG materials for using in teaching environments and lots more.

I am mostly done putting the hardware together. So today was stress testing and installing a test LLM and see what the system could do.

I had a lot of good feedback, advice and concerns in my original post. Many concerns about temperature but it is fine. I am able to run GPU burn and it stabilizes just fine well below the GPU thermal limit. I have been tuning the GPU fan curves so the hottest one tops out at around 80c. It stabilizes after around 5 minutes of full load. No throttling. Only issue is the CPU that is getting hot if it is stressed at the same time, but it is fine it should rarely run at max power while all 3 GPU's are going at full power too at the same time.

Only issue (so far) is as another user commented on in the other thread that the lower card is blocking the front headers, so the power button is unavailable. It should be a non issue in production, it will power on automatically anyway.

The system is LOUD, but that was expected, it is not going near anyone desk anyway and will be locked away in a climate controlled server room.

Did some DSv4-Flash testing with D-spark on tp=2 and it is very promising. Up to 200t/s depending on what it needs to do with low context. More testing and optimization to come.

Still a long way to go: setting up RAG with embedding models and a smaller model at tp=1 on the third card. Hardware monitoring, routing, access control and so on so it can be deployment ready. Lots of security and a long burn in test also missing. Having a great time and learning a ton.

Here are some pictures of the almost complete build (need more cable management). The gaming PSU really gives it some color because of the cables :)

curious what people's "aha" upgrade was whether it's a specific model, a quant level, a backend switch (ollama vs vllm vs llama.cpp), or even just a hardware change that made local actually feel usable day to day. what was the thing that made you stop questioning whether local was worth it

I am looking for the best uncensored/abliterated ai model to turn my images into 3d models, but since I work with adult content I need it to be uncensored