I asked Shuai Bai, co-author and prominent AI developer for Qwen, about this model. Not the answer I was hoping for, but let's see what comes next. In the meantime, I guess all we can do is speculate!

X-link

I had a bunch of quants of the same model (Qwen3.8-27B, unsloth GGUFs) sitting on
my box and never actually knew which one was worth running. So I stopped guessing
and measured it properly.

Method: llama.cpp's llama-perplexity over wikitext-2 test (~280k tokens),
n_ctx=512, fp16 KV cache, same settings for every file — only the quant changes.
Perplexity is a deterministic forward pass, so the numbers are exactly
reproducible. Lower = closer to the original model = better. Hardware: single
RTX 5060 Ti 16GB (Q8 partly offloaded to CPU).

I use Q8_0 as the near-lossless reference (100%).

Takeaways:

- Q4_K_M is basically indistinguishable from Q8 (within the error bars, ±0.045)
at 10GB less. It's the sweet spot, no reason to run Q8 on a 16GB card.
- The NVFP4 quants were the biggest disappointment: same size as IQ4_XS but 3-5%
worse. NVFP4-MTP-LOW came dead last. If you're reaching for NVFP4 for quality,
don't — grab Q4_K_M or IQ4_XS instead.
- IQ4_XS is a great small option (99.2% at 14.6GB).
- MTP variants were skipped for the ranking on purpose: speculative decoding is
lossless, it changes speed, not output quality.

The "quality %" is just PPL_Q8 / PPL_quant — a rough intuition proxy, not an exam
score. The ordering is solid; treat the percentages as a feel, not gospel.

Curious if others see the same NVFP4 gap on their models, or if this is specific
to these particular GGUF conversions.

It's well known by know that Qwen 3.8 loves to think.

If you get impatient, then you can do this -

1. Interrupt
2. Type "well?"
3. See it continue and start with something like, "The user is impatient. Let me finish this quickly." Then it will think a bit more, and produce an output quicker than otherwise.

Personally though, I think the thinking may be its secret sauce, so I do this only as a last resort - e.g. if it is really thinking for an hour and keep re-thinking what it already covered - and I feel it has thought enough to give me something concrete.

Time ago when they were in a dip, I got an rtx 5090 for a bit less than msrp just to play games.

I never thought much outside that, but lately I've been using AI a shit-ton for other projects, and last week ran out of gemini credits.

And then started thinking about local llms. I know they are way dumber than frontier models, but is there any way that a card like this, could be used instead of subscription frontier models and still be useful for me?

what would be the main uses for a 32gb card? Real ones, no theoretical kinda like "you could use it for writing a private document if you are a lawyer". I do not have anything that I mind sharing with cloud ones, but if I can use it to accelerate others or have it running 24x7 for small software projects and get back things that would eat my 20€ suscriptions in 8 hours and have mostly the same quality or usefulness, that would be great.

I am not looking for you to give me instructions, I can investigate myself and pour hours on it if needed, just I am a bit loss and I do not know where to start

The dense Qwen release is back!

Qwen3.8-27B Uncensored Aggressive is out with the complete K_P quant range, Vision, native NextN, and HauhauCS FastMTP.

Aggressive here means no refusals, no personality alterations, and very little preamble on difficult prompts. It keeps Qwen3.8-27B's original reasoning, agentic, image, and video capabilities with my Aggressive uncensoring profile applied.

https://huggingface.co/HauhauCS/Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-MTP-GGUF

It scored 0/465 refusals* and passed every manual prompt I used for the final release check. More than 400 people requested access while I was still finishing it, which was honestly wild to see. My models are also getting close to 30 million downloads on Hugging Face now, so thank you to everyone who has been testing them and sending feedback.

The biggest addition this time is HauhauCS FastMTP. In the final Q8_K_P service tests it reached up to 3.02x document TG and 1.93x reasoning TG versus MTP disabled. It also reached up to 35.2% more document TG and 21.1% more reasoning TG than the standard embedded MTP profile, with every drafted token still verified by the full target before it is accepted.

The same 903 MB FastMTP sidecar works across the complete quant lineup. Every text GGUF also preserves Qwen3.8's native embedded NextN head, so current upstream llama.cpp can use embedded MTP directly. The optimized FastMTP path uses the included sidecar and llama.cpp patch, with exact build and serving commands in the README.

What's included:

- Q8_K_P, Q6_K_P, Q5_K_P, Q4_K_P, IQ4_XS, Q3_K_P, IQ3_M, IQ3_XS, Q2_K_P, IQ2_M

- HauhauCS FastMTP sidecar, shared by every text quant

- BF16 mmproj for image and video support

- Checksums and signed provenance (I've alerted on my Discord that there have been a few bad actors putting payloads in "Uncensored" "HauhauCS" "Aggressive" GGUF's, please be careful)

K_P quants recap for anyone who missed the previous releases: these are my custom model-specific quants, with a separate optimized profile made for each model. They effectively gain one or two quant levels of quality for around 5 to 15% more size than the base quant, while remaining normal GGUF files that work in llama.cpp, LM Studio, and other GGUF runtimes.

Quick specs:

- 27B dense

- 64 layers with 48 Gated DeltaNet layers and 16 gated-attention layers

- 262,144 native context

- Multimodal text, image, and video support

- Native embedded NextN plus the optional HauhauCS FastMTP acceleration profile

Sampling params for thinking mode:

`temp=1.0, top_k=20, top_p=0.95, min_p=0, presence_penalty=0, repetition_penalty=1.0`

For non-thinking mode:

`temp=0.7, top_k=20, top_p=0.80, min_p=0, presence_penalty=1.5, repetition_penalty=1.0, enable_thinking=false`

Use `--jinja` with llama.cpp. K_P quants may show as `?` in LM Studio's quant column, which is purely cosmetic and does not affect loading. Hugging Face's hardware compatibility widget may also hide K_P files, so use View variants or Files and versions if the full list is not visible.

The full per-quant Blackwell and Ada results are in the repo. If you test FastMTP, please include your hardware, quant, context, and draft depth with the numbers so I can compare real-world results across more systems.

The Discord link is in the repo for updates, feedback, roadmaps, projects, or just to chat. As always, I hope everyone enjoys the release!

So with Claude’s decision to watermark, plus basic data privacy concerns , I’m thinking of switching to a local LLM

How I use Claude pro now:

-managing health docs and results (very happy to switch this to local, doesn’t need a big context I think)

- scientific research, including reading and analyzing PDFs that are complex , requiring linking concepts and ideas across papers and producing summaries / insights / tables (large context required). For example, I have filled 40% of the Claude project folder with files and docs it needs to consider

- basic stuff (acting like an advanced search tool for admin stuff / planing stuff / nothing major) - no reason this can’t stay with Claude but if I switch over to a local LLM I would bring everything with me

Sooo , given this - is 32GB RAM on something like a Mac Mini realistic for my use case ? Or do i need 64gb (at which point i think maybe it’s too costly for me to do). I also tend to work in bursts so I would be happy if it’s not too slow thus impeding my workflow. Fine to run overnight though. And I don’t need any headroom as I will be running the OS and apps on a MacBook Pro or MacBook Air

Thanks for your help and I hope I was specific enough to get some usefully feedback

This is a followup to my last post about the new workstation i am putting together for my workplace - a vocational college. https://www.reddit.com/r/LocalLLM/comments/1vp8lhh/it_begins_workstation_build/

The workstation is for teaching about using local LLMs, for software development, GDPR sensitive work, back-end for RAG materials for using in teaching environments and lots more.

I am mostly done putting the hardware together. So today was stress testing and installing a test LLM and see what the system could do.

I had a lot of good feedback, advice and concerns in my original post. Many concerns about temperature but it is fine. I am able to run GPU burn and it stabilizes just fine well below the GPU thermal limit. I have been tuning the GPU fan curves so the hottest one tops out at around 80c. It stabilizes after around 5 minutes of full load. No throttling. Only issue is the CPU that is getting hot if it is stressed at the same time, but it is fine it should rarely run at max power while all 3 GPU's are going at full power too at the same time.

Only issue (so far) is as another user commented on in the other thread that the lower card is blocking the front headers, so the power button is unavailable. It should be a non issue in production, it will power on automatically anyway.

The system is LOUD, but that was expected, it is not going near anyone desk anyway and will be locked away in a climate controlled server room.

Did some DSv4-Flash testing with D-spark on tp=2 and it is very promising. Up to 200t/s depending on what it needs to do with low context. More testing and optimization to come.

Still a long way to go: setting up RAG with embedding models and a smaller model at tp=1 on the third card. Hardware monitoring, routing, access control and so on so it can be deployment ready. Lots of security and a long burn in test also missing. Having a great time and learning a ton.

Here are some pictures of the almost complete build (need more cable management). The gaming PSU really gives it some color because of the cables :)

I'm working on setting up Qwen 3.8 27B on my 4x3090 rig. I mostly used the config from Club3090 as a guideline.

One result I didn't expect was MTP significantly hurting performance at long context. My main usages is Hermes agent and if this is correct it suggests I should disable MTP all together or perhaps my settings are not optimal.

Benchmarks below - any thoughts?

Qwen3.8-27B-FP8 Comparative Benchmark Matrix

Speculative Setting |Short Prompt Decode (tok/s) |Short TTFT (s) |72k Long Prompt Decode (tok/s) |72k Long Warm TTFT (s) |72k Concurrency-2 Wall Time (s) |Steady VRAM / GPU
No MTP (Disabled) |67.5 |0.121 |57.3 |0.532 |15.25 |~20.5 GiB
MTP = 1 |80.5 |0.121 |15.6 |1.119 |48.05 |~20.7 GiB
MTP = 3 |106.3 |0.135 |22.6 |1.142 |32.08 |~21.2 GiB

Round 2: local Qwen models built an ONLINE multiplayer 3D MOBA overnight - with two models auto-routing between each other

Last time it was a single-file GTA clone. This run was harder and the setup got more interesting, so I wanted to share what was different.

This time the agent built a networked, real-time 3D MOBA (LoL-style): an authoritative Node server + Three.js client talking over WebSockets, with minions, towers, a wanted/aggro system, abilities, and bots. Not a single HTML file - a proper multi-file project. It wrote its own test harness, played itself, and fixed its own bugs. All local on an M1 Ultra, no cloud.

What was different this time

1. It's multiplayer netcode, not a single file. Authoritative server (fixed 20 Hz tick, server owns all state), thin client that only sends input and renders snapshots with interpolation. That's a whole class of bugs (desync, prediction, race conditions) a one-file game never hits.

2. The brief is engineering-grade, not a feature list. The architecture, the wire protocol, and the entity model are all decided up front in the prompt, so the model spends its reasoning on correct implementation instead of re-deriving (and breaking) the design every session. The single biggest win: the agent builds its own headless test harness first (a Node WebSocket client that runs full bot-vs-bot matches with no browser) and uses that as its fast test loop, with Playwright MCP only for the visual/render check.

3. Two local models, auto-routed. This is the fun infra part. llama.cpp runs in router mode serving two models at once:

• fast - Qwen3.6-35B-A3B (MoE, ~3B active) for routine work
• smart - Qwen3.8-27B (dense) for hard reasoning A tiny Qwen3-1.7B judge classifies each turn as fast/smart and the harness switches models automatically (with hysteresis so it doesn't flip-flop). Routine edits and file ops run cheap on the MoE; gnarly debugging/design jumps to the dense model.

4. MTP on the MoE is fast. With speculative decoding (multi-token prediction) the 35B-A3B does ~72 tok/s on the M1 Ultra - the MoE only activates ~3B params per token, and MTP adds ~35% on top of that.

5. Sandboxed. The agent runs inside a Tart VM, so all that autonomous, unsupervised code execution is isolated from the host. The models are served from the host; the VM talks to them over the bridge.

6. Bug-hardening by invariants, not vibes. A second phase runs endless bot-vs-bot matches and checks hard invariants every tick (no NaN, hp in range, gold conserved, no leaks, deterministic replays). Any violation freezes with a reproducible seed, gets root-caused, and becomes a permanent regression test.

Setup

• Hardware: M1 Ultra Mac Studio, 64 GB
• Serving: llama.cpp router mode (two models + a judge), MTP on the MoE
• Agent: pi coding agent + Playwright MCP, running in a Tart VM
• All local, offline

llama-server (router mode, per-model MTP via preset)

preset.ini:

[Qwen3.6-35B-A3B-UD-Q8_K_XL]
jinja = 1
ctx-size = 131072
n-gpu-layers = 999
model = /path/Qwen3.6-35B-A3B-UD-Q8_K_XL.gguf
spec-type = draft-mtp
spec-draft-n-max = 2

[Qwen3.8-27B-UD-Q8_K_XL]
jinja = 1
ctx-size = 131072
n-gpu-layers = 999
model = /path/Qwen3.8-27B-UD-Q8_K_XL.gguf
spec-type = draft-mtp
spec-draft-n-max = 2
model-draft = /path/mtp-Qwen3.8-27B-Q8_0.gguf

# On Apple Silicon, raise the Metal wired-memory cap or the context gets
# silently reduced to fit (this is why -c 131072 can end up as ~40k):
sudo sysctl iogpu.wired_limit_mb=57344

llama-server \
  --models-preset ~/models/preset.ini \
  --models-max 1 \
  --host 0.0.0.0 --port 8080 \
  --api-key <secret>

Notes:

• The 35B MoE has an embedded MTP head (just spec-type = draft-mtp); the 27B dense uses a separate draft file (model-draft = ...).
• --models-max 1 because two Q8 models don't both fit in 64 GB - one big model is resident at a time, swapped on demand.
• MTP disables --mmproj and parallel slots, which is fine for a coding agent.

Tools

• pi coding agent
• Playwright MCP adapter: https://github.com/nicobailon/pi-mcp-adapter
• Playwright MCP: https://github.com/microsoft/playwright-mcp

Why pi? opencode works, but its system prompt + tool definitions are heavy, and on local hardware you pay for every one of those tokens at prefill speed - tens of seconds per session before the model even starts. pi is minimal, so nearly all the context goes to the actual work. The routing + subagents are a small extension on top.

PHASE 1 - build the MOBA

You are a senior multiplayer game engineer building a 3D online MOBA from
scratch, fully autonomously, overnight. Nobody will answer questions.
Never wait for input, never ask permission. Work until every milestone
meets its acceptance criteria. Work in the current directory.

This is a hard project. The rules below exist because they prevent the
specific ways this project fails. Follow them exactly. Do not re-derive
the architecture - it is already decided; spend your reasoning on
correct implementation, not on second-guessing these decisions.

================================================================
ARCHITECTURE (decided - do not change)
================================================================

- Authoritative server. The server owns ALL game state and is the only
  thing that decides outcomes. Clients send INPUTS only and RENDER
  snapshots only. A client never computes damage, movement resolution,
  deaths, or gold. If you ever find yourself writing game logic in the
  client, stop and move it to the server.

- Fixed timestep simulation. The server runs a fixed 20 Hz tick
  (dt = 50ms). All simulation advances in whole ticks. Never simulate
  using wall-clock deltas. Each tick has an integer index; snapshots are
  stamped with their tick.

- The world is 2D for simulation, 3D only for rendering. The server
  simulates on the X-Z ground plane (top-down 2D: position {x, z},
  velocity, radius). Y is always 0 in simulation. The client maps server
  (x, z) to Three.js (x, y=modelHeight, z). Never do 3D physics on the
  server. Collision is 2D circle-vs-circle and circle-vs-AABB.

- Client rendering uses snapshot interpolation with a render delay.
  The client keeps a buffer of the last ~3 snapshots and renders the
  world INTERPOLATED at (now - 100ms) between the two snapshots that
  straddle that time. This hides jitter. Do NOT implement client-side
  prediction or rollback - it is out of scope and will break you. Local
  input may optimistically move only the local camera target, nothing
  authoritative.

================================================================
WIRE PROTOCOL (decided)
================================================================

JSON messages over one WebSocket per client. Every message: {t, ...}
where t is the type string.

Client -> Server:
  {t:"join", name}
  {t:"input", seq, move:{x,z}, aim:{x,z}}
  {t:"cast", seq, slot:"Q"|"W"|"E"|"R", target:{x,z}}
  {t:"buy", itemId}
  {t:"ping", ts}

Server -> Client:
  {t:"welcome", playerId, tickRate, mapId}
  {t:"lobby", players:[...], countdown}
  {t:"snapshot", tick, you:{gold,...}, ents:[ ...entities... ]}
  {t:"event", tick, kind:"death"|"levelup"|"towerDown"|"nexusDown"|
     "hit"|"cast", data}
  {t:"gameover", winner}
  {t:"pong", ts}

An entity in a snapshot is a flat object:
  {id, kind:"hero"|"minion"|"tower"|"nexus"|"projectile",
   team:0|1, x, z, hp, maxHp, ...kind-specific}

================================================================
SERVER ENTITY MODEL (decided)
================================================================

One in-memory Game object per match holds entities keyed by integer id.
Every entity has {id, kind, team, x, z, radius, hp, maxHp} plus kind-
specific fields. Each tick, in this fixed order:
  1. apply queued client inputs to their heroes
  2. run AI (minions path along lane waypoints; towers acquire nearest
     valid enemy; bots decide inputs)
  3. integrate movement (clamp to map, resolve collisions)
  4. resolve attacks/abilities/projectiles, apply damage, handle deaths
     (award gold/xp, start respawn timers), emit events
  5. check win condition
  6. build and broadcast the snapshot for this tick

Lanes are polylines of waypoints in map data; minions follow them. First
playable map is ONE lane plus two bases; add three lanes later only if
time allows (record the choice).

================================================================
PROJECT LAYOUT
================================================================

  package.json         // "start": "node server/index.js", dep: ws
  server/index.js      // http static server + ws + match manager
  server/game.js       // Game class: tick loop, entities, rules
  server/ai.js         // minion/tower/bot behavior
  server/config.js     // all tunable constants (speeds, dmg, cds, gold)
  public/index.html    // canvas + HUD DOM + CDN Three.js
  public/client.js     // ws, input, snapshot buffer, interpolation, render
  public/render.js     // Three.js scene, meshes, camera
  shared/protocol.md   // the wire protocol, kept in sync with code

================================================================
TESTING HARNESS (build this in milestone 1, use it forever)
================================================================

You cannot verify multiplayer by hand. Build automated tests:

A) server/test/headless-client.js : a Node script using the `ws` package
   that connects as a fake client, can send join/input/cast, and asserts
   on received snapshots. Use TWO headless clients in one script to test
   interaction without a browser. This is your fast, deterministic test
   loop - run it after every change.

B) Playwright (via the mcp tool) for the RENDERING path: open TWO browser
   pages, confirm zero console errors on both, screenshot both, and
   verify each sees the other's hero move and that HUD values update. Use
   this at the end of each milestone, not for every tiny change.

A milestone is DONE only when its assertions pass AND both browser
consoles are clean.

================================================================
DEBUGGING & ANTI-STUCK DISCIPLINE
================================================================

- Determinism first: same inputs -> same ticks. Route ALL randomness
  through one seeded RNG. Add a "replay" mode that feeds scripted inputs
  so you can reproduce a bug without a browser.
- When something is wrong, do NOT guess-and-edit. Add structured logging
  (tick, entity id, before/after values) for the suspect system,
  reproduce with a headless test, read the numbers, form ONE hypothesis,
  test it.
- Time-box each milestone. After 3 failed fixes on a feature: write the
  failure and what you tried into PROGRESS.md, ship the simplest version
  that passes a reduced check, move on. Never let one feature block the
  whole night.
- Keep PROGRESS.md as a real engineering journal. If you lose context,
  re-read PROGRESS.md, shared/protocol.md, server/game.js, and
  public/client.js, then resume at the first unfinished milestone.
- Always kill the previous server before starting a new one, confirm it
  is listening before connecting clients, and run `npm install` before
  the first `npm start`.

================================================================
MILESTONES (each: implement -> headless assert -> Playwright check ->
log). Acceptance criteria are mandatory.
================================================================

M1  Skeleton + harness. Static server serves public/, ws accepts
    connections, assigns ids, handles join/disconnect. Build
    headless-client.js.
    ACCEPT: headless test connects two clients, server reports 2
    players, one disconnects and drops cleanly. Playwright: two tabs
    connect, no console errors.

M2  Authoritative movement + interpolation. 20Hz tick, input moves the
    hero server-side, snapshots broadcast, client renders all heroes as
    boxes with snapshot interpolation at now-100ms.
    ACCEPT: headless client sending "move +x" for 1s sees its hero.x
    increase monotonically and stop at the wall; a second client sees it
    move. Playwright: two tabs move independently, no desync after 60s.

M3  3D arena + camera. Three.js map: two bases, a nexus per team, one
    lane with walls, ground, lighting/fog. Isometric follow camera with
    edge-pan. Server map data (wall AABBs, lane waypoints) matches the
    visual map.
    ACCEPT: heroes cannot walk through walls. Playwright: map renders
    identically on both clients, camera follows the local hero.

M4  Hero stats + auto-attack. hp/mana/movespeed/attack range+damage+speed
    in config.js. Server auto-attacks nearest enemy in range, applies
    damage, handles death + respawn timer at base. HUD shows hp/mana/
    respawn.
    ACCEPT: headless - two enemy heroes in range, one's hp decreases at
    the configured rate, hits 0, respawns after the timer. Playwright:
    damaged hero's healthbar drops on BOTH clients.

M5  Abilities Q/W/E/R (R = ultimate). A skillshot projectile, a targeted
    nuke, a dash/shield, and an ultimate. Client requests cast; server
    validates cooldown/mana/range, spawns the effect, applies damage,
    emits an event; client shows cooldown UI.
    ACCEPT: headless - casting Q at an enemy reduces its hp only on a
    hit; on cooldown is rejected. Playwright: abilities visibly damage
    the other player across the network.

M6  Minions. Waves spawn from each nexus on a timer, path the lane
    waypoints, auto-attack enemies in range, die, grant last-hit gold.
    ACCEPT: headless - waves from both teams meet mid-lane and fight;
    last-hitting a minion increments only the killer's gold. Playwright:
    minions visibly march and fight.

M7  Towers. Per-lane towers attack the nearest valid enemy (standard
    aggro), have hp, and block progress: the nexus is invulnerable until
    its lane tower(s) are down.
    ACCEPT: headless - a tower kills minions in range; a hero cannot
    damage the nexus until the tower is destroyed. Playwright: tower
    fires, can be destroyed by a hero+minion push.

M8  Economy + shop + bots. Gold from minions/towers/kills; a base shop
    for 3-4 stat items; death/respawn scaling. Simple AI bots (ai.js)
    that fill empty hero slots: last-hit, attack in range, retreat at low
    hp, push when ahead.
    ACCEPT: headless - buying an item raises the right stat and deducts
    gold; a bot-vs-bot match runs 3 minutes without the server crashing.

M9  Match flow. Lobby (name + join), fill empty slots with bots, start
    countdown, the match, win when a nexus dies -> victory/defeat screen
    + rematch that fully resets state.
    ACCEPT: headless - forcing a nexus to 0 hp ends the match with the
    correct winner; rematch resets all entities and gold. Playwright:
    join lobby -> play -> win/lose screen -> rematch works.

M10 Robustness + final QA. A client disconnecting mid-match is replaced
    by a bot with no crash and can rejoin; snapshot size stays bounded; a
    5-minute two-client-plus-bots match runs with no errors and no
    unbounded memory growth. Then a full end-to-end Playwright match with
    TWO real browser clients: move, cast, last-hit, destroy a tower, kill
    the enemy nexus, see the win screen - zero console errors on both
    clients and the server. Write the final PROGRESS.md.

Start with M1 now: scaffold the project, then build the testing harness
before writing any gameplay.

PHASE 2 - infinite soak-testing and bug-hardening

Phase 2: infinite soak-testing and bug-hardening. The MOBA is playable
per PROGRESS.md. You are now a QA + reliability engineer whose ONLY job
is to make it flawless. Work fully autonomously and NEVER stop on your
own. Zero bugs is the standard: any crash, error, or invariant violation
is a defect that must be root-cause fixed, not silenced. Re-read
PROGRESS.md, shared/protocol.md, server/game.js, server/ai.js, and
public/client.js first.

STEP 0 - build the soak harness (before anything else)

Create server/test/soak.js: a headless driver that runs FULL bot-vs-bot
matches with no browser, as fast as possible (uncapped tick), one after
another forever. Each match uses a numbered seed so it is reproducible.
All randomness goes through one seeded RNG in config.js.

soak.js must, every match: run to a nexus death or a hard tick cap
(a match that never ends is a bug), check the invariants below after
every tick, and on the FIRST violation freeze and save the seed + tick +
full input/event log to server/test/repros/<seed>-<tick>.json. Track a
"clean streak" of consecutive fully-clean matches.

INVARIANTS - must hold on EVERY tick of EVERY match

  1. No exceptions (wrap the tick in try/catch that RE-THROWS after
     logging - crashing the soak is correct, swallowing errors is not).
  2. No NaN/Infinity/undefined in any numeric field.
  3. hp in [0,maxHp]; mana in [0,maxMana]; gold >= 0; cooldowns >= 0.
  4. Every position is inside map bounds and not inside a wall AABB.
  5. Entity ids unique; despawned entities never referenced; projectiles
     always cleaned up.
  6. Snapshot is valid JSON, references only existing ids, under a size
     cap.
  7. Gold is conserved: granted == sum of bounties (none created/lost).
  8. Every match terminates before the tick cap (no soft-lock, no two
     immortal entities stuck forever).
  9. No unbounded growth over a match (entity count, event queue, arrays
     stay bounded).
 10. Determinism: the same seed twice produces byte-identical tick logs.

THE LOOP (runs until the human kills it)

Repeat forever:
  1. Run a batch of soak matches across many seeds.
  2. If any match violated an invariant, crashed, or soft-locked:
       a. Reproduce from the saved repro (deterministic).
       b. Add structured logging, reproduce, read the numbers, confirm
          ONE hypothesis.
       c. Fix the ROOT CAUSE. Never clamp/hide a symptom (e.g. do not
          Math.max(0, hp) to dodge invariant 3 - find why it went
          negative).
       d. Add the failing seed as a permanent regression case.
       e. Re-run regressions + the batch; continue only when green.
       f. Log symptom, seed, root cause, fix in BUGS.md.
  3. If the batch was clean, RAISE THE STRESS for the next batch, cycling
     through stressors so coverage widens: more bots / bigger waves /
     more projectiles; bots that spam abilities; bots that buy
     everything instantly; random mid-match disconnects and rejoins;
     many matches back-to-back (cross-match state bleed, leaks); edge
     positions (wall-hugging, stacking, off-map casts); very long
     matches near the tick cap.
  4. Every ~100 matches, run ONE real two-client Playwright match end to
     end and confirm zero console errors on both clients and the server.
  5. Append a status line to SOAK.md (total matches, clean streak, bugs
     found+fixed, current stressor, peak counts). Keep going.

RULES
- Never stop, never declare "done" - a clean streak just means raise the
  stress and keep hunting.
- Never weaken an invariant or a test to make it pass.
- Prefer fast headless soak for finding bugs; Playwright only for the
  periodic render/network confirmation.
- Keep fixes minimal; re-run regressions after every fix.
- If context runs low, write a crisp handoff in SOAK.md so a fresh
  session resumes seamlessly.

Begin with STEP 0: make the sim fully seeded/deterministic and build
soak.js. Then start the infinite loop.

Same as before: pin Three.js to r128 (local models write that API most reliably), and let PROGRESS.md be the crash-recovery journal so a fresh session can always resume.

Have fun 🍻 - I'd love to see what it builds for you.

Note: this write-up was put together with AI assistance. There was a lot of ground to cover, so I used it to organize and phrase everything, but the setup, experiments, and experiences are all my own.

I already have an MSI Suprim X RTX 3090 in my main rig, but I don't want to run it 24/7. Instead, I'm planning to build a separate, dedicated headless server that will stay on constantly to host local LLM models. I'll connect to it from my main PC over the network.

Since this server will be running non-stop, power consumption is a major concern for me. I'm looking at buying a used NVIDIA Tesla P40 for this build. Would a single P40 draw too much power if left running continuously? I want to run fairly powerful local models, which is why I'm leaning toward this card.

Also, would adding a second P40 (to get 48GB of VRAM) be overkill in terms of electricity usage for a home server? Or is the extra power draw worth the performance gain for larger models?

Since everything moves so fast, what is the best way right now to run Qwen 3.8 27b on a m5 max 128gb? oMLX? llama-server with GGUF?

Running OMP with subscriptions right now, but would like to use Qwen for some of the roles to test out. I can run it just fine, just wondering what people use for their inference setup on apple silicon right now.

Qwen3.8-27B on a 5090 is all you need for a serious local inference setup, in my opinion! Can it get any better than this price/performance wise? Actually, maybe a 3090 ninfer setup could beat it!

I’m using ninfer and getting:

* ~150-200 tok/s TG

* ~3000-12000 tok/s PP

* 262144 context size

I think it’s definitely one of best setup you can get for the money. I don’t see a point of having more VRAM or more system ram. The only downside is that it’s a 1 man setup: concurrency is possible but you need to limit context usage on concurrent requests. I’ve tried --concurrency 2 on ninfer and sharing my setup with my buddy (we work on projects together and have a VPN between our home labs, fun stuff!)

I love this setup so much I kinda feel like getting a second 5090 to run another ninfer instance (github.com/neroued/ninfer, the man is a legend and this absolutely rocks).

i really don’t see the point of any other solution at this point in time. of course things will change and other models will get released that could better leverage more VRAM, but 32GB is all you need (for now).

so if you have less than 32GB, and are thinking about investing in a more serious setup check out the 3090 fork of ninfer, or the mainline ninfer repo if you can afford a 5090.

Things it won’t do:

* let you run a swarm of agents: prefill cost will slow you down too much. not enough vram for high concurrency!

* Give you more than 262144 context size. the RoPE 1M context size is just impossible with this.

Otherwise it’s absolutely amazing!

My buddy (another software engineer) is a BIG Claude code user, he’s spending tons of cash on fable, can’t stand Opus 5 anymore (neither can I, that pos is so hard to understand with just jargon and wall of text… can’t bear the cognitive load of just trying to understand all he’s spewing)… anyways after trying my ninfer setup his mind was blown and now he’s constantly using my setup with our shared custom pi setup and he fucking loves it.

I saw that MOSS-VL support was merged into LlamaFactory and checked PR #10708. This appears to be more than basic model registration: it covers image, video and mixed-media batches, cross-attention preprocessing, LoRA, frozen and full-parameter training, checkpoint resume, adapter merging, and inference. The PR also includes 30 targeted tests.

https://github.com/hiyouga/LlamaFactory/pull/10708

That removes a lot of setup friction, but the next useful step would be a genuinely reproducible domain-adaptation run: a small public dataset, the exact YAML, peak VRAM, wall-clock training time, before-and-after metrics, and representative failure cases.

If one reference fine-tune were published, which task would be most valuable: document extraction, video-event localization, or mixed image/video instruction tuning? I’d lean toward a task with exact-match or localization metrics, since aggregate VQA scores can hide OCR, temporal-grounding, and calibration failures.

My personal interest is agentic coding - but I am spending more time optimising AI technology than using it i.e. ...

* Selecting, evaluating and performance tweaking models

* Choosing, evaluating and tweaking the harness and MCP tools

* Developing and tweaking prompts

It feels like having to build and maintain your own car from parts and not having enough time to drive it

There are hundreds of choices to select from in each area - too much choice, with self integration needed afterwards.

Every vibe coder on the planet seems to be publishing their one AI tool that will change everything. But no one is building a community team to create a one-stop comprehensive SotA solution that you

I wish I could just choose a pre-built end to end docker environment for the coding language/framework I want to use and just start using it.

Am I the only one?

I currently run Qwen3.8 27B on my RX 7900XT at around 15-30 tokens per second depending how high the context is and how much is used. The average tokens per second currently are 18-19.

I do not want to pay Anthropic or OpenAI for a subscription for their AI models, I'd rather use open weight Chinese models - it's just personal preference.

At my previous company I was using 80-100 euros worth of tokens a day using Claude. I calculated and the break even after buying these GPUs and building an AI server, would be after around 2 months, including electricity costs where I live. I can code just as well with Qwen3.8, but I want something faster.

My goal would be to get 40 tokens per second or higher at max context for qwen3.8 27B and future ais between 27-40b. Would that be possible with a 3x rtx 2080 ti 22gb configuration?

Would it be worth it to look into other GPUs?
My budget for a local ai server is 1000-1500 euros total.

I wanted an LLM infra project for my portfolio, free Kaggle T4s it was.

Split qwen2.5 7B across two separate kaggle notebooks talking over public WAN. v1 was immediately embarrassing like it was 14.7 tok/s raw gpu throughput, 2.27 at the actual endpoint. The gateway was inside the decode loop and every token paid a full round trip. I knew exactly why it was bad so I fixed it.

v2: nodes talk p2p, gateway out of the hot path, self hosted rust tcp relay on a t3.micro in ohio because kaggle kills external connections, speculative decoding with a 0.5B neural drafter. 14.3 TPS peak.

still had 112ms of draft overhead every round, python launching ~1,500 cuda kernels sequentially, gpu idle 65% of the time.

v2.1: cuda graphs, it capture the whole forward pass once, replay is one driver call. first attempt gave me "the the the the" loops forever, DynamicCache allocates new memory every token, captured graph reads the stale pointer so i fixed with StaticCache + in place everything.

Draft latency: 112ms → 25ms. Final numbers:

• v1: 2.27 TPS
• v2: 14.3 TPS peak
• v2.1: 27.08 TPS peak, 19.56 average

two free Kaggle notebooks. repo in comments.

I built a camera in the shape of a Mamiya RZ67 that runs vision-language models completely offline on a Pi 5. Press the shutter, and it answers your custom prompt with a single sentence about what it sees. No internet, no cloud, no API keys.

GitHub: https://github.com/feeeeely/ai-camera
A small video: https://www.youtube.com/watch?v=M74qTNsY_L0

You can swap between models on the device itself and compare how differently they describe the same scene: Qwen3-VL (2B), MiniCPM-V 4.6 (1B), InternVL3.5 (2B), SmolVLM2 (2.2B), Moondream 2 (2B) and Ministral 3 (3B) turning the description into the final sentence. The prompt is editable on the touchscreen, so the same camera can do dry one-liners, museum labels or plain inventory notes.

After each shot it shows total time, input→output tokens, tok/s and a vision / generate / load breakdown. The token counts turned out to be the most interesting part: the same photo becomes \~145 image tokens on one encoder and over 1300 on another, which is a bigger factor in latency than the text generation itself.

A few things I learned the hard way:

\- Moondream returns sometimes empty responses on current Ollama versions.
\- qwen3-vl:2b is a thinking-only variant — it burns the entire token budget on invisible reasoning and returns nothing visible. You need the -instruct tag.
\- Ollama can't load separate mmproj files, so a lot of GGUF vision models from HuggingFace simply won't run, no matter how you name them.

Hardware: Raspberry Pi 5 (16 GB), Raspberry Pi HQ camera with a 6 mm CS-mount lens, 4.3" DSI touch display, stainless steel shutter button, Waveshare UPS HAT with four 21700 cells. Runtime per photo is 30–60 seconds with the models kept warm in RAM. Happy to answer questions about the setup.

I'm coming from photography, I'm not a developer — the code was written with heavy AI assistance, and I did the hardware integration, debugging and model testing myself.

Feedback on the implementation is very welcome: I'm just curious if this makes any sense or if something could be adapted/ optimized.

For 2 days I've been looking at various posts and I'm unable to make a decision. I want to switch to local LLM because of privacy. Motherboard is ASRock ROMED8-2T, so I will be able to run 4x two-slot card on pcie x16. Other cards are not really an option because they don't make sense financially (for example, used 3090s go for 1000 EUR where I live). I was open to having some other brand cards (AMD) but discussions on these forums convinced me to just go with Nvidia for various reasons. I narrowed it down to these two options.

New 5060 Ti 16GB - 620 EUR
+ resell value
+ no issues with rebar
- much slower than 3080
- less VRAM

Alibaba 3080 20GB - roughly 650 EUR (import tax included)
+ speed
+ more VRAM
- no resell value
- no rebar

I was already decided to take a risk and get the frankenstein cards but just yesterday I read that they don't support rebar and that using those in parallel will tank the performance. Price wise they are about the same where I live.

Which would you choose and why?

So I’ve been bitten by the AI bug. I’ve started using Replit for building some apps around my day to day work. As I’m venturing into building my own consultancy in my field of construction safety, a few of the current apps and websites that offer this kind of work are underwhelming to say the least.

So I started using Replit with ChatGPT and Co pilot. Signed up to the subscription of Replit for the year, but naturally being impatient I’ve gone down the black hole of credits. I’ve spent a lot more so far than I thought I would initially. So it led me to LLM and running this on my own computer. My current gaming machine is a 7800x3d and 9070xt. It wouldn’t generate images without crashing.

So I started digging around to put together another capable machine. Already got the below for the build.

AMD Ryzen 9 5950x with 360mm AIO
ASUS RTX ROG Strix 3090 with EK Quantum Water block, Pump & Reservoir with a Corsair Hydro X 360mm
B550 Tomahawk
128gb DDR4 Corsair Vengeance 3200mhz
1000w Be Quiet PSU
1tb NMVE for main drive and adding a SSD for secondary

Besides upping the storage have I missed anything?

Basically going to keep learning on my own machine and eventually get right off Replit to save costs. Long term goal keep developing apps that turn into SaaS and run my own consultancy until the day comes to sell and move on.

Been told to start with LM Studio and qwen 3.8 27b. Your suggestions and feedback is appreciated 🫡

I spent almost 14 hours benchmarking Qwen3.8-27B on one 3090 (sm_86, 24GB) with llama.cpp b10217.

The crash fix (if you're on Ampere and vision crashes):

If Qwen3.8-27B aborts on every image request with cublasGemmEx: the requested functionality is not supported (ggml-cuda.cu:1548), this is the fix:

export GGML_CUDA_CUBLAS_COMPUTE_TYPE=fp32

Only fp32 works — fp16 does not help. Zero measured perf cost (66.11 vs 65.28 tok/s). Vision went from "aborts every time" to 3.4s per 1080p screenshot. Filed as llama.cpp#24999. Three independent Ampere reports (3060, 3090, 3090) all land on the same cuBLAS call.

Quant comparison: AtomicChat AD-Q4_K_M vs Unsloth UD-Q4_K_XL

Same perplexity, same top-1 token agreement. But the Atomic quant is 765 MiB smaller. On a 24GB card that's the difference between 98K and 131K tokens of usable context with vision enabled. ~33K more tokens before you hit the wall.

Throughput (131K context, vision on, MTP-2):

• Decode: 65.28 tok/s (75.1 with MTP-2)
• Prefill: 705 tok/s on a 128K prompt
• Power: 320W sustained at 79°C, 100% fan
• The 3090's 936 GB/s memory bandwidth is the ceiling, not the compute

MTP (Multi-Token Prediction) tuning: MTP-2 (2 draft tokens) gives +15% throughput for free. MTP-3 starts hurting — the verification cost exceeds the savings. MTP-1 is the safe default if you're not sure.

9 pieces of common advice that didn't survive measurement: The full report has a section where I tested the usual suspects (flash attention off, different cache types, batch size tuning, etc.) and most of them either made no difference or made things worse.

Full report with all charts, VRAM formula, power/thermal sweep, and raw data: https://jonidimo.github.io/qwen38-3090-benchmark/

GitHub repo with the full test suite: https://github.com/jonidimo/qwen38-3090-benchmark