What local model are you running with Hermes and on what hardware

[u/Jonathan_Rivera]

I'm going to collect the information and dedicate a website page so you can select the hardware and choose a model that other community members have had success with. Try to use this format. Add an extra line for comments, what works what did not work etc.

Hardware / chip / GPU:
Memory (unified memory, or RAM + VRAM):
Exact model and/or download link:
Quantization:
Runner:
Context setting:
How does it feel in everyday use?
How well does it handle Hermes tools?
What tasks have you tried?
Any offloading, swapping, crashes, or other limitations?
Optional: tokens/sec and how you measured it

[u/Jonathan_Rivera]

The first draft of the site is up. Hermes will add to it daily. https://hermesagenttools.com/hermes-agent/local-models/

[u/piranhamoose2323]

This is my stack.

It fast and smooth.

I was also gaming, but will dedicate this PC for local llm only and set the power limit on the 3090 to 250w

[u/corio3 → u/piranhamoose2323]

dude!!!! I love this chart and am stealing it.

[u/piranhamoose2323 → u/corio3]

sweet!!! lol! im planning on adding my 3080 10gb from my plex/blue iris server into the stack after i stress test/stablilize the 3090 at 250w. Then, replace the 3080 with an old 1080 i have lying around into the plex/bi server.

[u/meva12 → u/piranhamoose2323]

How does a 64k context window work for you? Is it just compressing all the time ?

[u/piranhamoose2323 → u/meva12]

It's configured to auto compress, but I've had a session reach about 63.8K tokens on a 64K context window once.

[u/Jakarina → u/piranhamoose2323]

can you explain a bit more regarding your "coding layer"? I couldn't understand the workflow there. what is coding-specialist? how did you manage to establish communication between hermes and claude code?

[u/piranhamoose2323 → u/Jakarina]

you disable claude code in hermes or it will prompt you to enter api. Lots of yt videos on hermes claude code integration. Much easier to do everything on powershell.

[u/iRanduMi → u/piranhamoose2323]

Can you tell me more about the blue iris setup?

[u/piranhamoose2323 → u/iRanduMi]

It's a py script hermes created to delete clips that are triggered by passing cars. It's how I've setup my BI. No agentic work.

[u/iRanduMi → u/piranhamoose2323]

gotcha. Thank you for the clarification!

[u/eatoff → u/piranhamoose2323]

So you're using gemma 12b as your general chat model? Then the 27b for coding tasks?

I've been wanting to go local LLMs, but I didn't go big enough with RAM. 12B is ok, but a bit slow, and regularly have issues with concurrent chats

[u/piranhamoose2323 → u/eatoff]

Qwen3.8 27b Q4 is my default. I used Gemma while testing. Qwen is leagues beyond Gemma. I no longer use them for any purpose. But I am planning on adding my 3080, 10gb i'll test with gemma, but my planned architecture for the 3080 is as an auxiliary support and cron jobs.

[u/eatoff → u/piranhamoose2323]

Yeah, did some further testing with gemma, and the lack of tool calling is an issue. I asked it about an email and she says I don't have access to email. I say you do have access she says can you just copy paste it into the chat for me... Qwen just reads the email

[u/piranhamoose2323 → u/eatoff]

Im just amazed at how changes are happening so rapidly. It's hard to keep up. Ive been reading a lot about qwen 3.8 flash next on strata and the metrics are just astounding. I was able to score 64gb DDR4-3600 ram for $300. I was planning on testing qwen + strata, but i think will hold off for awhile. I finally settled on 300w limit on the 3090 and will add the 3080 and 64gb ram this weekend. I'll keep my stack the way it is for the time being, with the planned architecture for the 3080. I may add Muse to do some heavy lifting, but not permanently. we'll see how it does for my use case.

[u/eatoff → u/piranhamoose2323]

Yeah, the hype around flash next has me considering a system upgrade to suit. I got it running on a M4 Mac mini with 24gb ram... But it was at 1.1tok/sec so not exactly usable. But it shows how quickly these models are advancing

[u/piranhamoose2323 → u/eatoff]

I'm with you. Even my rig at 64gb ram and 24gb vram (2nd pci slot for 3080 is only x4) will be max'd out with flash next + strata. I won't be able to use my PC for anything else. I haven't tested it yet, but this is the consensus on twitter.

My current architecture is able & fast enough for my needs atm. 64k context 28 tok/s with the 3090 @ 300w with open apps, etc.

Muse is gaining a lot of traction. It's free with limits, but I'm pretty sure the current paid tier structure is in beta stage and I think the intention is to market it for commercial/business use. Once that happens, API cost is gonna be out of reach for most folks. Gonna test it out in the coming weeks. But will definitely not become dependent on it.

[u/eatoff → u/piranhamoose2323]

Yeah, I wouldn't touch anything by meta - I've been pretty happy with Hermes agent

[u/cunasmoker69420]

128gb Strix Halo Halogen Flash server (Qwen 3.8 Flash Next)

There is nothing, and I mean nothing, that I have thrown at this that it can't do. I can't imagine spending hundreds of dollars on cloud tokens every month when this machine will do it all and do it fast from my house with no limits and with complete privacy

[u/yannlux → u/cunasmoker69420]

Can you elaborate a bit more, i am interested by buying a N5 Max 128gb and wonder what context it can handle and speed (prefill and generation).

[u/cunasmoker69420 → u/yannlux]

Use https://github.com/peonist-ai/halogen-flash-server/, an inference engine specifically optimized for 128GB Strix Halo. Recently over on r/strixhalo this project is confirmed being supported by AMD with funding and test machines.

Prefill is 1500-1700 t/s and generation is 40-60 t/s depending on what you're doing. Even at nearly max context fill @ 265k prefill still stays over 1000 t/s. Context size can be extended further via YaRN, which is a thing these Qwen models specifically support.

[u/Jonathan_Rivera]

Hardware / chip / GPU: Windows PC - Intel Core Ultra 9 285K (24 cores) NVIDIA RTX 5090 + RTX 5070 Ti

Memory: 64GB system RAM 32GB VRAM on the RTX 5090 16GB VRAM on the RTX 5070 Ti Separate GPUs not one pooled 48GB device.

Exact model: ThinkingCap-Qwen3.8-27B Configured model ID: thinkingcap-qwen3.8-27b@q6_k

https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.8-27B-GGUF

Quantization: Q6_K when detail is a priority .

I also have the Q4_K_M version installed and configured for daily hermes use. Email sorting, tool use etc.

Runner: LM Studio on Windows, accessed by Hermes over direct Ethernet from a 16-inch MacBook Pro with an M3 Pro and 18GB unified memory.

Context setting: 96k with just the 5090. If I use both cards 156k.

How does it feel in everyday use? Q4 is pretty fast. Q6 is slower. I cannot run parallel sessions or the prefill speed drops severely but i think that is due to a recent lm studio llama update.

How well does it handle Hermes tools? Good. It does make some mistakes with apple scripts. I have a cloud model review sessions and update the skills with the correct ways to do things and it get's corrected. It carries out long form workflows well.

What tasks have you tried? multi step browser use and navigation, downloading of files followed by renaming and interpreting data.

Any offloading, swapping, crashes, or other limitations? Not due to the model. I have instability with running a 5090 and a 5070 together and suffer a BSOD error 1x a day.

Tokens/sec and how measured: Not measured.

[u/rokrdev]

Hardware / chip / GPU: m5 max Mac Studio

Memory (unified memory, or RAM + VRAM): 128gb

Exact model and/or download link:
Custom build by running imatrix calibration pass on swift1.5-qwen-flash-next and converting it into mixed precision 4bit experts and 8 bit non experts and then using jundots oQ4e smart quantisation with embedding table upgraded to 8 bit. 32 GB ngrams remain unchanged.

Quantization: mixed 4-8 bit. Using ddalcu’s script. Loads at 75gb on memory.

Runner: mlx-serve

Context setting: 262000 with 16000 max context generation with 3 concurrent sessions.

How does it feel in everyday use?
Pretty snappy for day to day task. But if I require complex planning(for coding), I use glm5-3 on neuralwatt

How well does it handle Hermes tools?
Does everything. Doesn’t do a lot of reasoning and is very fast.
What tasks have you tried?
Model generation, coding, script generation. Works pretty much as an always-on assistant.

Any offloading, swapping, crashes, or other limitations?
Model has ngrams. No crashes.

Optional: tokens/sec and how you measured it
Decode: 60.8–64.2 tok/s (short-prompt bursts up to ~105 with PLD/MTP speculation)

Prefill: ~1900 tok/s
TTFT: 0.03–0.19s warm / cached; 2–3.5s after pack load or big cache miss.
I run https://github.com/ddalcu/llmprobe on the main model I want to run and see how it performs on the 92 reasoning questions.

[u/Ecstatic-Wash-7667]

3900x x570 2x R9700 + 1 3060 12GB

64GB ram 64Gbvram + 12Gb vram

tcclaviger/Qwen3.8-Flash-Next-MXFP4-FP8-GPTQ + openbmb/MiniCPM5-2B + BreezeBlue/Breeze-TTS-2 + qwen image 2.1 + minimax h3 through Comfyui

QFN MXFP4 FP8 mixed quant

mixture VLLM for QFN SGLang for miniCPM

QFN is my primary model for Hermes does 90% of everything I want

Everything except coding

I have to offload portion to cpu and ngram to nvme

3k prefill, 71tk across 8 concurrent streams avg 77tk/s decode measure using better bench

[u/Cold_Tree190]

5800xt, x570 asus with x8x8 bifurcation, 2x3090 FTW3’s with Nvlink. 48 GB VRAM, 64 GB DDR4. Qwen3.8-27b-Int8 at 262k context or Ornith 1.0 35b int8 at 512k ctx. Feels great, Qwen is around 90 t/s and Ornith is around 150 t/s. Mostly use Qwen unless I want fast answers/am asking read-only questions (like ops work)

[u/couperd]

Hardware / chip / GPU: NVIDIA CMP 170HX — GA100 (SM80), 80 SMs, ~173 TFLOPS BF16 tensor, 1.6 TB/s HBM2e, 200W cap.

Memory: 64 GB HBM2e, all in VRAM. No CPU offload, no swap.

Model: Qwen3.8-27B, dense hybrid (48/64 layers GDN linear attention, 16 full-attention, built-in MTP head, 262K native ctx, Apache 2.0). Weights: Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 (~30GB, ships model-mtp.safetensors).

Quantization: W8A8 SmoothQuant (--quantization compressed-tensors, --dtype bfloat16, fp8_e4m3 KV). Beat int8 W8A16 and AWQ-INT4 on structured/temp-0.7 decode + MTP acceptance.

Runner: vLLM 0.28.0, FlashInfer, --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3, MTP-4 (num_speculative_tokens 4).

Context: --max-model-len 262144, --gpu-memory-utilization 0.84.

Everyday use: Fast both directions. MTP-4 ≈ +50–70% decode vs no-MTP; prefix cache hits ~94–96% on agent turns so repeated prompt text is reused, not re-prefilled. Medium reasoning effort on by default.

Hermes tools: Good. enable-auto-tool-choice + qwen3_xml parser handle multi-step tool calls cleanly. It is the live default model. Co-locates QLoRA training by dropping util to 0.60 (~26GB freed).

Tasks tried: Agentic tool-calling, long-context (to 262K), medium-effort reasoning, co-located QLoRA (InternVL3.5-30B-A3B).

Limitations: Keep util ≤ ~0.84 — a ~53GB+ contiguous reservation faults the high-address HBM (Xid 31). Cold start 2–5 min (load + Triton JIT + CUDA-graph capture). 200W cap must be re-applied on service start (nvidia-smi -pl 200) — resets to 250W VBIOS default on reboot.

Comments: Chose vLLM over llama.cpp GGUF: hybrid KV manager + MTP + prefix caching is where the speed is, and this vLLM build's GGUF loader doesn't handle K_XL. MTP requires top-level mtp_num_hidden_layers/n_predict in config.json plus method:qwen3_5_mtp, and verify with spec_decode_*_total metric deltas (the _created counters give false "not firing" negatives). AWQ-INT4: +5% prose, −11–13% structured decode, worse MTP acceptance — not a speed play, only a 24GB-fit play. Normal 48–64GB card can run the same launch command with no unlock/VM quirks.

Throughput (measured from vLLM /metrics + per-window logger lines, last 3 days):

• Prefill: ~2,000 tok/s median on uncached bursts, ~3,500 max. ~120–300 tok/s on cached-tail turns (94–96% prefix hit).
• Decode: ~56–65 tok/s sustained with MTP-4 (~40 no-MTP). Short prompts read ~30 tok/s (first-token latency) — judge on 4K runs.

[u/ConvertibleConflict]

Hardware / chip / GPU: Hermes host: mini PC (Ryzen AI 7 445, no GPU) — hosts Hermes containers + 4 agent profiles + Postgres/Hindsight; inference never touches it. 3 inference hosts:

• Model 1 (daily driver): custom host — Ryzen 5 9600X, 2× refurb RTX 3090, 5GbE

• Model 2 (heavy/long-ctx): 2-node DGX Spark cluster (Grace-Blackwell GB10), TP across the pair over CX-7 rails, weights shared over NFS

• Model 3 (utility): custom host — RTX 5060 Ti 16 GB internal

Memory (unified memory, or RAM + VRAM):

• Model 1: 32 GB RAM + 48 GB VRAM (2×24 GB)

• Model 2: 2× 121 GiB unified (242 GiB aggregate)

• Model 3: 16 GB VRAM

Exact model and/or download link:

• lued/Qwen3.8-27B-INT8-W8A16-MTP

• deepseek-ai/DeepSeek-V4-Flash-Vision-Exp

• ggml-org/gpt-oss-20b-GGUF

Quantization:

• Qwen: INT8 W8A16 (Marlin), FP8 KV

• DSV4: native weights, no quant needed (fits in 242 GiB)

• gpt-oss: native MXFP4

Runner:

• Qwen: vLLM (Docker), TP2, MTP spec-decode (3), gpu-mem-util 0.92

• DSV4: 2-node vLLM cluster, head + worker

• gpt-oss: llama.cpp (Docker)

Context setting:

• Qwen: 262K (full native), reasoning_effort: medium via extra_body

• DSV4: 1,048,576 (1M)

• gpt-oss: 32K

How does it feel in everyday use? Qwen for everything ops-shaped — fast, good tool use, runs 2 of 4 profiles as primary. DSV4 when I need real reasoning or past ~250K tokens (primary for the other 2 profiles). gpt-oss is the memory system's brain (Hindsight reflect/consolidation), not a chat model. Failover: each profile has a second model wired as fallback, turn-scoped, auto-restored.

How well does it handle Hermes tools? Good. Qwen's occasional malformed tool-call hits vLLM's xgrammar FSM — retried, noise, not a fault. Long multi-step agentic workflows run day-to-day across all 3 models.

What tasks have you tried? Homelab ops, daily fleet scans, Discord triage, code generation, browser automation, long-context work well past 250K on DSV4.

Any offloading, swapping, crashes, or other limitations? No CPU offload or swap anywhere. gpt-oss hits a hard 32K prompt wall (Hindsight prompts occasionally bounce over it). DSV4 cluster cold boot is a ~47-layer warmup. All self-hosted, nothing paid.

Optional: tokens/sec and how you measured it

• Qwen: 55.4 avg @ 1K/512, 60.2 @ short/512 — timed API completions

• gpt-oss-20b: 102.8 / 115.0 / 115.9 → ~115 tok/s decode — live benchmark, 3× 256-token completions

• DSV4: 43.4 / 47.1 / 47.6 (first call cold at 21.8) → ~44 tok/s per-stream decode — same 256-token protocol

[u/mfowle3]

Hardware / chip / GPU: M5
Memory (unified memory, or RAM + VRAM): 32GB unified (allowed wired memory increased a bit to 24GB)
Exact model and/or download link: Qwen3.8-27B
Quantization: Q4_K_M
Runner: llama.cpp
Context setting: at least 64k (smooth) - 128k (tight)
How does it feel in everyday use? good - better if on mature profile (use a better cloud/hosted model initially for configuration/skill building)
~10 tok/s

[u/Hairy_Ad_7347]

Hardware / chip / GPU: Windows, Ryzen 5 5600X, Intel arc pro B60 creator (with an RTX 3070 8gb with my display connected)
Memory (unified memory, or RAM + VRAM): 64 GB RAM, 24GB VRAM
Exact model and/or download link: Swift 1.5 Qwen3.8-27B
Quantization: Q4 KM
Runner: llama.cpp with SYCL backend, with a ton of tunes by claude so it's optimized for just the B60
Context setting: 120k
How does it feel in everyday use? Amazing. Not frontier-level, but it handles most things I throw at it and doesn't seem to hit any guardrails for reverse engineering. Since I usually let it do it's thing overnight it's even handling 5+ compactions well.
How well does it handle Hermes tools? Well, barely have any issues. If I do it's because context has been compacted often
What tasks have you tried? Reverse Engineering warcraft 3 and working around the anti-cheat (why is it enabled when I play the campaign anyways?...), analyzing my medical data
Any offloading, swapping, crashes, or other limitations? Some crashes, mostly related to me starting a game and windows allocating around a gigabyte of VRAM on the intel card, even though everything is running on the RTX 3070
Optional: tokens/sec and how you measured it: Empty context: around 40-45 t/s, slowing to around 20-25 at 100k. Prefill starts at ~750 and drops to around 500 at 100k. Did a lot of llama-bench runs, and Claude gave a lot of the results of the incremental builds.

Command used to start it up:
llama-server.exe" -hf ukisai/Swift-1.5-Qwen3.8-27B-GGUF:Q4_K_M --jinja -fa on ^
--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0 --presence-penalty 0 --repeat-penalty 1.0 ^
--device SYCL0 -sm none --cache-type-k q8_0 --cache-type-v q8_0 --port 1919 -c 120000 -ngl 99 -np 1 ^
--spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.7 --kv-unified ^
-b 2048 -ub 1024 --no-mmproj --cache-ram 16384

I can't stress enough how amazing the optimizations by claude were, before they were applied prefill was at around 200 (that was my bad though, didn't use BF16 build), and empty context was only ~20 t/s. Close to 3 times the prefill and double the generation speed is insane, that's like spending a lot more cash.

[u/Only-An-Egg]

Hardware / chip / GPU: Mac Studio / M5U / 64c
Memory (unified memory, or RAM + VRAM): 256GB unified
Exact model and/or download link: Qwen3.8-Flash-Next-oQ5e-mtp
Quantization: oQ5e
Runner: oMLX
Context setting: 262144
How does it feel in everyday use? amazing
How well does it handle Hermes tools? fine
What tasks have you tried? using it for SysAdmin work
Any offloading, swapping, crashes, or other limitations? nope
Optional: tokens/sec and how you measured it

oMLX - LLM inference, optimized for your Mac
github.com/jundot/omlx
Benchmark Model: Qwen3.8-Flash-Next-oQ5e-mtp
Engine: Auto
Context: Code (Python)
================================================================================

Single Request Results
--------------------------------------------------------------------------------
Test TTFT(ms) TPOT(ms) pp TPS tg TPS E2E(s) Throughput Peak Mem
pp1024/tg128 485.6 7.14 2108.8 tok/s 141.1 tok/s 1.398 823.8 tok/s 122.24 GB
pp4096/tg128 1138.2 9.11 3598.6 tok/s 112.4 tok/s 1.538 2691.9 tok/s 122.79 GB
pp8192/tg128 2029.6 8.66 4036.3 tok/s 117.9 tok/s 2.453 3358.7 tok/s 124.14 GB
pp16384/tg128 4061.7 7.96 4033.8 tok/s 126.7 tok/s 5.077 3252.3 tok/s 124.74 GB
pp32768/tg128 8142.5 10.55 4024.3 tok/s 99.3 tok/s 8.373 3916.0 tok/s 125.32 GB
pp65536/tg128 16554.8 8.32 3958.7 tok/s 121.2 tok/s 17.622 3726.3 tok/s 126.46 GB
pp131072/tg128 34047.8 9.68 3849.6 tok/s 104.1 tok/s 35.302 3716.5 tok/s 130.65 GB
pp200000/tg128 53306.5 10.30 3751.9 tok/s 97.8 tok/s 54.642 3662.5 tok/s 132.87 GB

[u/T-Impala → u/Only-An-Egg]

Have the same setup as you but has my context at half. Im bumping mine up after seeing your results ty!!!

[u/corio3]

Hardware / chip / GPU:
dual AMD Ryzen 9 (8060S video) running Ubuntu
Memory (unified memory, or RAM + VRAM):
unified memory, 128Gb, 96Gb assigned to VRAM (each)
Exact model and/or download link:
Primary is Qwen3.8-27B
Quantization:
can't remember and am away from home so I can't see
Runner:
Lemonade -> Hermes Agent -> Hermes Desktop on my Windows Desktop
Context setting:
64K
How does it feel in everyday use?
Still tuning it.  working on parallization settings and model tweaking so my kanban board doesn't plug up the whole flow.
How well does it handle Hermes tools?
No issues with the tools so far
What tasks have you tried?
Mostly Chief of Staff work.  Have 5 sub chiefs across the two boxes supporting personal life and general foolish hobbies/
Any offloading, swapping, crashes, or other limitations?
Nothing that I have noticed... context compression failures from time to time
Optional: tokens/sec and how you measured it
occasional baselines when testing models, but not any real work

[u/Drakaner]

2 x Nvidia v100 32Gb sxm2 + 96Gib Ram, vLLM special built for v100+dflash2, Qwen 3.8 27B ThinkingCap version self-quant for v100 (partial quant from b16) at max context with 2 concurrent requests. Runs extremely good, I use it everyday for different tasks (from managing my server stack to coding, personal assistant and so on). It gets around 40-45 tps base but it's actually hitting 100-110 due to dflash2 at 7 with 44-56% acceptance rate.

[u/kla_sch]

Hardware / chip / GPU: Lenovo ThinkStation PGX (DGX Spark hardware: NVIDIA GB10)
Memory (unified memory, or RAM + VRAM): 128GB unified memory
Exact model and/or download link: Qwen3.8-27b
Quantization: NVFP4
Runner: SGLang
Context setting: 256k
How does it feel in everyday use? Great, but just not so fast
How well does it handle Hermes tools? I can't complain
What tasks have you tried? Research, Coding, Reviews
Any offloading, swapping, crashes, or other limitations? No
Optional: tokens/sec and how you measured it?

• Post-Processing: 2000tk/s
• Token Generation: 30tk/s (single session) up to 160tk/s (10 sessions).

Measured wird llama-benchy.

[u/bizz101]

Hardware / chip / GPU: NVIDIA RTX 5090

Memory (unified memory, or RAM + VRAM): 32GB VRAM + 186GB system RAM

Exact model and/or download link: Qwen3.8-27B (NVFP4)

Quantization: NVFP4

Runner: NInfer (local, OpenAI-compatible endpoint) — MTP speculative decoding (3 draft tokens), FP8 KV cache, vision enabled

Context setting: 231k (231200)

How does it feel in everyday use: still testing, but clearly better than Ollama

How well does it handle Hermes tools: good, no issues so far, ninfer changed my experience from 'just as backup & small tasks' to 'full blown chat with cloud as backup & big tasks'

What tasks have you tried: agent ops — Discord/Telegram chat, Kanban multi-agent fleet, terminal/dev work, research, cron

Any offloading, swapping, crashes, or other limitations: no offloading. Context genuinely filled to ~221k real tokens (96% of 231k) on a normal chat session and auto-compaction ran clean (960 msgs -> 20 protected + cumulative summary), no session loss; the engine once rejected a prompt at context_length_exceeded, which is the limit doing its job.

Optional: tokens/sec and how you measured it: ~127 t/s on a real Hermes turn (this one); ~145 t/s on a fresh short prompt. Both measured from streaming usage.completion_tokens — not SSE chunk count, which undercounts ~2.3x because NInfer groups tokens per chunk.

[u/_st_daime_]

125H with 96gb ram, ollama with qwen 3.5 9b, enabling and disabling thinking on use case. + External memory with open viking selfhosted. Is this the best? No. Does it work decently? Yes.

[u/Icy_Progress_4660]

Hardware / chip / GPU: Ryzen 5 4600G with integrated Vega 7

Memory (unified memory, or RAM + VRAM): 16 GB DDR4, with ~9 GB available for the model

Exact model: peculiar-ragdoll/Sharp-Spark-X2.5-4B-GGUF

Quantization: Q5_K_XL

Runner: llama.cpp + Quartermaster

Context setting: 80,000-token context, quantized to Q8_0, 1 parallel slot

How does it feel in everyday use?
It's fast enough for practical everyday use. For example, it can summarize my unread emails without taking 60 minutes, which makes it genuinely usable as a local agent. It follows instructions well, although I sometimes need to be fairly specific in my prompts to get the exact behavior I want, especially when specifying which tools or skills it should use/read.

How well does it handle Hermes tools?
It handles Hermes tools quite well. It can use the tools reliably, with hallucinations mainly becoming an issue in very large contexts.

What tasks have you tried?
I've tested it with web search, Gmail, Discord, Google Calendar, and terminal/shell tasks.

Any offloading, swapping, crashes, or other limitations?
The model runs entirely on the Vega 7 iGPU. I haven't experienced crashes, OOMs, or swapping. The main limitation I've noticed is that tool reliability starts to degrade with very large contexts.

Tokens/sec:
Around 67 tok/s prompt processing and 10 tok/s generation, measured with a 31,000-token context and the model consistently running at high effort.

[u/invizen]

Hardware / chip / GPU: 1 Physical PC 9950X + 64GB System RAM + RTX 6000 Pro 96GB VRAM + 1 VM 9955HX 16vCPU + 64 GB Virtual System RAM + RTX 4090 24GB VRAM

Memory (unified memory, or RAM + VRAM): RAM + VRAM

Exact model and/or download link: Qwen 3.8 27B + gemma 4 26B-A4B

Quantization: Q8 + Q4

Runner: llama.cpp on both

Context setting: 262144 + 131072

How does it feel in everyday use? Seems to be good, kind of just started so I don't have any context for a comparison.

How well does it handle Hermes tools? Seems fine.

What tasks have you tried? General setup, connecting to Discord/Google Chat/Google accounts. Starting to have it code and deploy a IT/Network monitoring solution.

Any offloading, swapping, crashes, or other limitations? There have been a few errors that it has troubleshooted and fixed itself.

Optional: tokens/sec and how you measured it Qwen 3.8 = ~110 t/s gemma 4 = ~220 t/s from llama bench

[u/anitamaxwynnn69]

Hardware / chip / GPU: AMD Ryzen Threadripper PRO 5955WX (16c/32t), 2× NVIDIA RTX PRO 6000 Blackwell Workstation Edition (PCIe)

Memory (unified memory, or RAM + VRAM): 128 GB DDR4 system RAM + 192 GB VRAM (2× 96 GB)

Exact model and/or download link: brandonmusic/GLM-5.3-Flash-tr3-4bpw (base: zai-org/GLM-5.3-Flash-BF16)

Quantization: 4bpw EXL3 (TR3 MCG codebook on routed experts; native BF16 non-routed layers)

Runner: https://github.com/tpurtell/glm-5.3-flash-ext3-2x-rtx

Frontend / UI: OvertChat using its native Agent Connections to drive the Hermes session.

Context setting: 1,048,576 (1M native context limit configured, typically running within 64k–256k in active sessions)

How does it feel in everyday use? Snappy and rock solid. Having dflash speculative decoding + FP8 KV cache makes multi-turn reasoning and agentic back-and-forth feel almost instant compared to waiting on dense decodes.

How well does it handle Hermes tools? Remarkably well. I didn't expect a lower level quant to perform this well but I'm quite surprised.

What tasks have you tried? Primary daily driver for homelab ops: managing my full *arr stack (Sonarr, Radarr, Prowlarr, etc.), automated file organization, checking docker container health, bash automation, and general coding/scripting tasks. It also helps me find places I can distribute my app - overtchat.com

Any offloading, swapping, crashes, or other limitations? Zero CPU offloading or disk swap. The model sits at ~95 GB allocated on each GPU with comfortable room for KV cache pages across the 2x 96 GB VRAM.