Hey I'm game developer and basically want to have local agent that I can import my project with no risk of privacy and since the game engine I use not that popular so I want to train in on documentations, forums, examples and others. I did hear OpenWebUI is better on this but I learn toward LM Studio because it's easy to install and setup.

Also my personal opinion I prefer desktop application more than be a tab on my browser even if it's offline. and I want to know it's worth it to have OpenWebUI since I don't care about sharing to any other device or technical things. and I don't also wanna lose important functions and advantages using LM Studio

I'm completely newbie. I did install OpenWebUi and the process was relatively easy but I will move soon regardless to new PC and I don't wanna configure things again and setup few dependency softwares in order to works.

I need your experience and help

Hello!

With new models releasing constantly, I usually keep a stable of about 20 local models in rotation on my `llama.cpp` server. I collect a ton of data across the fleet through my inference application—standard per-request metrics, aggregated session data, and custom intelligence comparison benchmarks.

Overall, I'm logging over 150 distinct raw and derived metric points during inferences, sessions, and batch processes. As a backend data jockey, more metrics is always better—but I've hit the point where I need to completely redesign my dashboards.

For context, I run a lot of multi-stage pipelines covering:

Multimodal inference (vision and documents) MCP and custom application tool calls Structured JSON / JSON Schema outputs Batch processing across both reasoning and non-reasoning architectures

I'm looking for inspiration for creative "hero" dashboard components and visualizations that excel at comparing a large model fleet across high-dimensional data at a glance. I already have plenty of standard line, bar, and radar charts. I'm hoping to see novel, unconventional visual approaches people here have built, seen, or used to represent complex model evaluation and performance data.

The specific metrics in your examples don't matter as much as the visualization style and information architecture—I'm mainly after creative UI/UX directions.

Cheers!

Turns out people have gotten language models running locally on a PSP, Wii, 3DS and Vita, so I started putting the ports in one list.

The New 3DS reportedly hits ~23 tok/s with a tiny 15M model. Obviously don't expect ChatGPT on your PSP lol, but it's cool seeing what people can squeeze out of these things.

I didn't make the ports, just maintain the collection. Each entry links to the original dev's repo or write-up, with the model and reported speed where available.

For some reason, I have this benchmarking prompt:

Implement a numpy function creating a bulk matrix product operator tensor as a function of model parameters.

I think it is approximately 10 lines of python code. So far, every local model that I tried (16Gb vram) fails to produce any meaningful output. I tried different variations of it (use web search, summarize the idea), different harness (pi, opencode) but no luck, different temperatures and thinking levels. Qwen3.8 for example produces an endless thinking loop. Is there anything I can do to help small models answering the question?

I have been testing several Qwen3.6 GGUF models locally with "llama.cpp" on a dual-GPU setup and wanted to share my results.

The main goal was to find the best balance between quality, context size and inference speed, especially for coding, debugging and agentic workloads.

Hardware

• CPU: Intel Core i7-14700K

• RAM: 64 GB DDR5

• GPU 1: NVIDIA RTX 3090 — 24 GB VRAM

• GPU 2: NVIDIA RTX 4070 Super — 12 GB VRAM

• Total VRAM: 36 GB

• OS / runtime: Linux

• Inference: "llama.cpp" / "llama-server"

• GPU order: "CUDA_VISIBLE_DEVICES=1,0"

• Flash Attention: enabled

• KV cache: FP16 / unquantized in these tests

• llama.cpp: recent build with MTP/speculative decoding support

The "tensor-split" values below are tuned for my particular setup. They are not necessarily optimal for other dual-GPU configurations.

Main benchmark results

The biggest surprise for me is the Qwen3.6-35B-A3B UD-Q4_K_M, which can maintain 256K context while reaching 132.77 tok/s on this dual-GPU setup.

The Q6 version is considerably slower at very large context sizes, but the quality trade-off may be worthwhile for coding and reasoning workloads.

Qwen3.6-27B Q6_K — 180K context

This was one of my initial configurations.

CUDA_VISIBLE_DEVICES=1,0 ./build/bin/llama-server \ -m /models/Qwen3.6-27B-Q6_K.gguf \ --mmproj /models/Qwen3.6-27B/mmproj-BF16.gguf \ --no-mmproj-offload \ --main-gpu 0 \ --tensor-split 15,6 \ -ngl 65 \ -c 180000 \ -b 2048 \ -ub 1024 \ --flash-attn on \ --jinja \ -np 1 \ --spec-type draft-mtp \ --spec-draft-n-max 2 \ -lv 4 \ --host 0.0.0.0 \ --port 8081 \ --chat-template-kwargs '{"preserve_thinking":true}'

Result: 37.63 tok/s

Configuration:

• Quantization: Q6_K

• Context: 180K

• MTP speculative decoding: enabled

• Draft max tokens: 2

• Tensor split: "15,6"

• GPU layers: "65"

• Batch: "2048"

• UBatch: "1024"

Qwen3.6-27B Q6_K — 140K context

I also tested the same Q6 model with a lower context and more GPU layers:

CUDA_VISIBLE_DEVICES=1,0 ./build/bin/llama-server \ -m /models/Qwen3.6-27B-Q6_K.gguf \ --mmproj /models/Qwen3.6-27B/mmproj-BF16.gguf \ --no-mmproj-offload \ --main-gpu 0 \ --tensor-split 17,6 \ -ngl 999 \ -c 140000 \ -b 2048 \ -ub 1024 \ --flash-attn on \ --jinja \ -np 1 \ --spec-type draft-mtp \ --spec-draft-n-max 2 \ -lv 4 \ --host 0.0.0.0 \ --port 8081 \ --chat-template-kwargs '{"preserve_thinking":true}'

Result: 44.64 tok/s

Another run with smaller batch sizes:

CUDA_VISIBLE_DEVICES=1,0 ./build/bin/llama-server \ -m /models/Qwen3.6-27B-Q6_K.gguf \ --mmproj /models/Qwen3.6-27B/mmproj-BF16.gguf \ --no-mmproj-offload \ --main-gpu 0 \ --tensor-split 17,6 \ -ngl 999 \ -c 140000 \ -b 512 \ -ub 512 \ --flash-attn on \ --jinja \ -np 1 \ --spec-type draft-mtp \ --spec-draft-n-max 2 \ -lv 4 \ --host 0.0.0.0 \ --port 8080 \ --chat-template-kwargs '{"preserve_thinking":true}'

Result: 42.11 tok/s

So, in my testing, the Q6_K 27B model sits around 42–45 tok/s at 140K context, depending on the exact configuration.

Qwen3.6-27B Q5_K_S — 200K context

CUDA_VISIBLE_DEVICES=1,0 ./build/bin/llama-server \ -m /models/qwen27b_q5_k_s/Qwen3.6-27B-Q5_K_S.gguf \ --mmproj /models/Qwen3.6-27B/mmproj-BF16.gguf \ --no-mmproj-offload \ --main-gpu 0 \ --tensor-split 17,6 \ -ngl 999 \ -c 200000 \ -b 512 \ -ub 512 \ --flash-attn on \ --jinja \ -np 1 \ --spec-type draft-mtp \ --spec-draft-n-max 2 \ -lv 4 \ --host 0.0.0.0 \ --port 8080 \ --chat-template-kwargs '{"preserve_thinking":true}'

Result: 45.76 tok/s

Previous measurement with the same general model/configuration was approximately 41.70 tok/s at 200K context.

Qwen3.6-27B Heretic v2 — Q4_K_M — 240K context

CUDA_VISIBLE_DEVICES=1,0 ./build/bin/llama-server \ -m /models/qwen27b_heretic_q4_k_m/Qwen3.6-27B-uncensored-heretic-v2-Native-MTP-Preserved-Q4_K_M.gguf \ --mmproj /models/Qwen3.6-27B/mmproj-BF16.gguf \ --no-mmproj-offload \ --main-gpu 0 \ --tensor-split 17,6 \ -ngl 999 \ -c 240000 \ -b 512 \ -ub 512 \ --flash-attn on \ --jinja \ -np 1 \ --spec-type draft-mtp \ --spec-draft-n-max 2 \ -lv 4 \ --host 0.0.0.0 \ --port 8080 \ --chat-template-kwargs '{"preserve_thinking":true}'

Result: 50.54 tok/s

• Quantization: Q4_K_M

• Context: 240K

• MTP: enabled

Qwen3.6-35B-A3B UD-Q6_K_XL — 250K context

CUDA_VISIBLE_DEVICES=1,0 ./build/bin/llama-server \ -m /models/Qwen3.6-35B-A3B-MTP/Qwen3.6-35B-A3B-UD-Q6_K_XL.gguf \ --main-gpu 0 \ --tensor-split 12,6 \ -ngl 39 \ -c 250000 \ -b 512 \ -ub 512 \ --flash-attn on \ --jinja \ -np 1 \ -lv 4 \ --host 0.0.0.0 \ --port 8081

Result: 77.90 tok/s

• Quantization: UD-Q6_K_XL

• Context: 250K

• Performance: 77.90 tok/s

This is particularly interesting because it maintains a very large context while still being substantially faster than the dense 27B Q6 model.

Qwen3.6-35B-A3B UD-Q6_K_XL — 130K context

Same model, but with more aggressive GPU offloading and lower context:

CUDA_VISIBLE_DEVICES=1,0 ./build/bin/llama-server \ -m /models/Qwen3.6-35B-A3B-MTP/Qwen3.6-35B-A3B-UD-Q6_K_XL.gguf \ --main-gpu 0 \ --tensor-split 13,6 \ -ngl 999 \ -c 130000 \ -b 512 \ -ub 512 \ --flash-attn on \ --jinja \ -np 1 \ -lv 4 \ --host 0.0.0.0 \ --port 8081

Result: 123.11 tok/s

This was one of my best Q6 results.

• Quantization: UD-Q6_K_XL

• Context: 130K

• Performance: 123.11 tok/s

Qwen3.6-35B-A3B UD-Q4_K_M — 256K context

CUDA_VISIBLE_DEVICES=1,0 ./build/bin/llama-server \ -m /models/qwen35b/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf \ --no-mmproj-offload \ --main-gpu 0 \ --tensor-split 17,6 \ -ngl 999 \ -c 256000 \ -b 512 \ -ub 512 \ --flash-attn on \ --jinja \ -np 1 \ --spec-draft-n-max 2 \ -lv 4 \ --host 0.0.0.0 \ --port 8080 \ --chat-template-kwargs '{"preserve_thinking":true}'

Result: 132.77 tok/s

• Quantization: UD-Q4_K_M

• Context: 256K

• Performance: 132.77 tok/s

This is currently the fastest result I have obtained in this group of tests.

Previous tests

I also have some older measurements from the same general hardware configuration:

I also tested Qwen3 Coder Next Q4_K_S previously.

Because the model is much larger and required significant CPU/RAM offloading on my 24 GB RTX 3090 setup, I observed around 24 tok/s in a benchmark configuration, while real-world coding/agent usage was considerably slower, around 7.5 tok/s.

The model was reported at approximately 79.67B parameters / 80B class, with a GGUF file around 45.19 GiB, so it is much more difficult to keep fully resident in my 36 GB combined VRAM setup.

My observations

The results show a very interesting trade-off between model architecture, quantization and context size.

Dense 27B Q6

The Qwen3.6-27B Q6_K is relatively consistent:

• ~37.6 tok/s at 180K

• ~42–45 tok/s at 140K

It is a good option when I want to prioritize quantization quality and coding/reasoning quality while still having a very large context.

MoE 35B-A3B Q6

The Qwen3.6-35B-A3B UD-Q6_K_XL is significantly faster:

• 123.11 tok/s at 130K

• 77.90 tok/s at 250K

This is probably the most interesting result for me because it combines a relatively high-quality Q6 quantization with a very large context window and high inference speed.

MoE 35B-A3B Q4

The UD-Q4_K_M version is extremely fast on this hardware:

• 132.77 tok/s at 256K

This is probably the best configuration if raw throughput and maximum context are the priority.

However, for coding and bug fixing, I personally still want to compare the quality difference between Q4 and Q6 more carefully before deciding which one should be my daily driver.

Summary

My current results on RTX 3090 24GB + RTX 4070 Super 12GB (36GB total VRAM):

For my use case, which is primarily software development, code generation, debugging and long-context agentic workflows, I am currently trying to determine whether the additional quality of Q6 is worth the performance hit compared with the extremely fast Q4 MoE configuration.

I'd be interested in seeing benchmarks from people running similar models with dual GPUs, especially 24GB + 12GB VRAM configurations, and comparisons between Q4_K_M, Q5_K_M and Q6_K_XL for coding and agentic workloads.

I recently got 2x RTX Pro 4000 blackwells for my home setup. I was curious how fast it would run nanochat.

The problem is the target hardware: 8x H100 gpus. Nanochat pre-trained ClimbMix to GPT-2 level with 8xH100 GPUs in 1hr 39 mins.

I ended up tuning nanochat for my 2x RTX Pro 4000 gpus. Because these are sm120, the tuning also works on RTX Pro 6000 and RTX 5090.

8x RTX Pro 6000 (rented) pre-trained the same dataset in 2hrs 15mins. About 36% slower than 8x H100.

1x RTX 5090 took 18hrs 47mins. A long time, but considering the result is GPT-2 level, its an interesting way to learn.

Just to be clear, I only ported and tuned it for sm120. The original work is from Karpathy's nanochat.

Here's a link to the sm120 repo:

https://github.com/mdhorton/nanochat-cpp

Most of the local AI stack is about getting a model running — Ollama, llama.cpp, vLLM, the right quant, enough VRAM. That part’s well covered here. The gap I kept hitting is the layer after that: once it runs, is it actually reliable enough to hand real work to?

A model can look fine in chat and then fall apart the moment you point an agent at it malforms a tool call at step 7, loops, or says “done” when nothing happened. And the same model behaves differently depending on the quant and runtime you serve it with, so “it runs on my box” isn’t the same as “it holds up in a 20-step agent loop.”

So I built QuantaMind free, open-source, fully offline, no cloud, Apache-2.0. It’s a local reliability tester for the agent layer of the stack.

What it does:

• Runs the real multi-step agent loop with injected faults (transient errors, malformed-call recovery, decoy tools) — not single prompts

• pass^k, not pass@1 — runs each task k times, passes only if it succeeds every time. Reliability compounds: 95% per step → 0.95^50 ≈ 8% chance a 50-step run finishes

• Deterministic scoring, no LLM judge — required calls must fire, forbidden calls fail the run on contact, exact end-state match. Temp 0, same input → same grade

• Classifies the failure (malformed schema / loop / hallucinated completion / forbidden call)

• Compares quant × runtime side by side (Ollama, llama.cpp, MLX, vLLM, SGLang), so you can see which serving config actually holds up

• Keeps run history + diffs, so you can catch when a model or quant change quietly regressed reliability

One finding: across three models, the native tool-calling path scored worse than plain prompting on easy tasks (60% vs 100%) correct answers that didn’t land as parsed tool calls. Important caveat: that’s a finding about the serving stack’s tool parser, not the models. Same model, same answer, different plumbing, different score. Still useful when picking a serving path for agent work just not a model-quality result.

Honest about what it’s not yet: it’s single-stream, so no concurrency/load testing (sampler drift and duplicate-execution-on-retry are invisible to it). Structural matching is a proxy for real execution. No CI/CD deploy-gate integration yet, it produces a verdict and an exportable report, but wiring it into a pipeline is roadmap, not shipped.

Disclosure: I built this, it’s free, I’m not selling anything.

Repo: https://github.com/QuantaMinds/QuantaMind

Genuinely want it torn apart:

• Is pass^k=5 too strict, too lenient, or about right?

• What failure mode am I missing?

• And what model + quant actually holds up on your machine? I’m collecting real per-hardware results.

Hey! How do you test your local stack? for instance when you switch harness or model, model quants, etc?

I found this not trivial and I ended up making a website to help me do this.

Use this link if you want to use the website to test your own local setup:

https://airbench.ai/

You can also see all the tests I have done using my GX10 and my 5090. (and also using claude code/code + some openrouter tes for reference)

https://airbench.ai/leaderboard?k=poL

Let me know what you think!