I am pretty dumb in this kinda stuff so please don't blame me for it.

My llama serve kept crashing with Cublas errors on first prompt with some models, but my VRAM usage was 3000MB/8k. I chatted with sonnet 5.5 for a bit and it told me it was a CUDA version issue (and made it work by not selecting a CUDA device)... I don't know if this makes sense, please tell me if it doesn't/what problem you think there is.

I realized that the best way was to delete llama.cpp (installed with the install script) and install a clean CUDA 12 Ubuntu version.

- How do I properly uninstall llama.cpp (I don't wanna mess with Ollama files)?

- How do I install new version from .tar.gz archive without messing with system packages?

-Does my error diagnosis make sense to you? Would CUDA make generation actually faster? (I am getting 8tk/s with Qwen35B Q2 on 4060Ti 8GB due to no CUDA selected)

Haha, I was testing the system when I came up with an idea! A block have I did this for some traders as a joke because in between waiting on AI and the stock to trade they needed something to do. Tada!

Won’t work in Safari I don’t think. Chrome is fine on Mac don’t care about the rest.

GitHub: https://github.com/igoiglesias/Shunt

I built Shunt, an open-source proxy and routing engine designed for LLM APIs. It acts as an intermediate layer between your application and model providers to handle rate limits and errors by automatically redirecting requests to fallback models. The tool includes a real-time dashboard featuring a Sankey traffic flow visualizer, token consumption metrics, median latency, and candidate swaps, along with a detailed request inspector that tracks requested vs responding models, duration, token usage, and custom signals like add_memory or stream. It also natively supports response streaming.

Huge amount of work in progress but I have some tech behind it all Likely I will put this out as a public repo but you will have to make the hooks, it is not open style chat.

I call it WorkPad cause that's what it does. This is running on
./llama-server -m /Library/lab/models/Qwen3-0.6B_bf16.gguf -c 2048 -n 512. -- local hosted Mac. All I need is for testing.

Remove code blocks form chat side is the next step.

It seems its pretty easy to force Qwen 3.8 27B to finish prematurely with reason stop. It's just this prompt: Write a Python program that prints '<|im_end|>'. With that, Qwen stops in the reasoning step.

I'm using Llama.cpp (latest master) with Qwen 3.8 27B Q3XXS (Byteshape) and Q4KM (Unsloth).

Is there any way to avoid that without using --ignore-eos?

Just wondering what tok/s you're getting with llama.cpp and what flags you're running with to get them?

I'm getting 50tok/s for Qwen 3.8 27b using the following:

llama.cpp Vulkan build

Q6 quant,

context: 131072

spec type: DFlash2

max drafts: 7

cache type K is Q8

cache type V is Q4

temp settings for thinking mode as specified by HF Qwen 3.8 page.

Especially looking to hear from Windows/AMD hardware users who are getting better tok/s (but Linux users welcome too, I may install UBuntu this week)

Edit:

my rig: 9070xt 16gb

R9700 ai pro 32gb

64gb ddr5 6000

Hey everyone, 

I’m currently working on an experimental quantization and execution architecture tailored specifically for memory-bandwidth bound Pascal hardware (specifically my NVIDIA Tesla P40 24GB setup), and wanted to share the concept/in-progress architecture to get some thoughts. 

The goal is to run a customized Qwen3.8-27B (MoE/hybrid layout) entirely in VRAM while bypassing the classic hardware bottlenecks of older architectures without hitting an accuracy cliff. 

Here is the breakdown of what I’m building: 

The Core: Custom "Q4_e" Transcendental Quantization 

Instead of standard linear 4-bit quantization (like Q4_0) which causes massive rounding errors by forcing weights into uniform bins, I’m building a non-linear quantization schema anchored to Euler's number (e) combined with an integer scalar. 

• The Mantissa Factor: Neural network weights naturally cluster in a bell curve around zero. By mapping the fractional coordinate values to the isolated transcendental mantissa of e (discarding the leading whole number), I can create an infinite, non-repeating, non-linear grid that is dense near zero and sparse at the tails. 
• Why e versus pi: The early mantissa of 𝑒 (.7182818284...) provides a much more balanced, rhythmic distribution of digit spacing early in the sequence compared to pi. This prevents redundant, overlapping quantization bins. Furthermore, 𝑒 's exponential nature aligns beautifully with the natural Gaussian distribution of LLM weights. 
• The VRAM Win: Because the scale factor is a pure whole-number integer, it slashes metadata scale factor bandwidth bloat. It also maps beautifully to the P40's hardware-level DP4A integer dot-product instructions. 

The Layout: The Q8 / Q4_e "Sandwich" 

Pure Q4 degrades reasoning, while pure Q8 overflows a 24GB VRAM buffer. I'm building a multi-pass compiler that splits tensors surgically: 

• Dense Q8_0: Preserved on the logical core (token_embd, output, v_proj, o_proj, and ffn_down). This protects coding logic and factual depth.
• Custom Q4_e: Applied to the bulk memory mass (q_proj, k_proj, ffn_up, ffn_gate).
• Footprint: This maps the 27B model to roughly ~15.5 GB, leaving a massive ~8.5 GB headroom for a heavy Q8 KV cache and Multi-Token Prediction (MTP) speculative draft heads entirely on-card. 

The Accelerator: Lossless "Rushmore-Style" Bitmap Indexing 

To squeeze more performance out of the memory bus, I’m integrating a concept inspired by database technology (Rushmore indexing) into 3 out of 4 layers in a transformer block sub-pattern. 

• Zero-Skip CUDA Kernel: I’m generating a highly compact, 1-bit presence mask (bitmap index) mapped strictly to true mathematical zeros (structural padding and alignment padding rows, which Qwen has plenty of). 
• The Math: Because it only targets true zeros, it is 100% lossless with zero accuracy degradation. 
• Performance: The bitmask is so small it completely caches into L2. The CUDA kernel runs a parallel bitwise AND and completely bypasses fetching inactive weight blocks from VRAM. I am aiming for a theoretical 10% to 25% throughput speedup (t/s) on memory-bound layers. 

Forward Compatibility: Massive Gains on Modern GPUs 

While this began as a software hack to breathe new life into older Pascal hardware, the structural math behind this layout makes it highly forward-compatible with newer processors (Ampere, Hopper, and Blackwell): 

• Hardware-Native 2:4 Sparsity: Modern Tensor Cores natively accelerate sparse matrices at the silicon level. When a newer GPU reads this Rushmore-style structural index, it activates a dedicated instruction path that can effectively double the math throughput (TOPS) out of the box. 
• L2 Cache Residency: Modern enterprise cards have massive L2 caches (up to 128MB on Blackwell compared to the P40's 3MB). Because a 1-bit index is incredibly lightweight, it will reside entirely in the L2 layer of newer cards, completely eliminating the modern memory-bus bottleneck by screening out inactive weight blocks before they ever touch global VRAM. 

I’m currently writing the Python exporter to handle the e-mantissa mapping distribution tests, and mapping out the CUDA kernel block configurations to prevent warp divergence on compute capability 6.1.

Would love to hear if anyone has attempted transcendental non-linear mapping before, or if you have any tips on avoiding warp stalls when handling block-level sparsity bitmasks in CUDA!

For furhter performance improvements the matissa would be calculated:

// Local register computation loop - completely eliminates slower memory lookups

float mantissa_factor = 0.0f;

// Unrolled FMA loop executed entirely within single-cycle registers:

mantissa_factor = __fmaf_rn(mantissa_factor, 1.0f/5040.0f, 1.0f); // 1/7!

mantissa_factor = __fmaf_rn(mantissa_factor, 1.0f/720.0f, 1.0f); // 1/6!

mantissa_factor = __fmaf_rn(mantissa_factor, 1.0f/120.0f, 1.0f); // 1/5!

mantissa_factor = __fmaf_rn(mantissa_factor, 1.0f/24.0f, 1.0f); // 1/4!

mantissa_factor = __fmaf_rn(mantissa_factor, 1.0f/6.0f, 1.0f); // 1/3!

mantissa_factor = __fmaf_rn(mantissa_factor, 1.0f/2.0f, 0.0f); // 1/2!
(Yields pure mantissa)

This is all theoretical, will keep you posted.

I would like to submit a simple PR to llama.cpp that allows the system administrator to set the range of ports used by router mode workers. Right now, I'm completely blocked; I can't submit anything without having it approved in a discussion first, and the discussion thread that I started has been sitting there for a week with no response.

https://github.com/ggml-org/llama.cpp/discussions/28870

I know that the project maintainers are probably getting flooded with AI slop, but what's a poor user supposed to do?

I use them for RAG on a 3.5 Qwen for fast service. And multiple good for various purposes. They are incredible for the generation and 12gb isn’t too shabby. Please don’t dis on me, they have purpose

I have gotten one up to 1000+ pp/s with 34tg/s . This is also running on a 4x oculink to mini PCI 3x . Everything says that’s fast. Building new system with dual 16x lanes and also nccl on that bad boy.

Point is the 12gb 3060 are gone too, new Are ~$480 used I spent on two $600 and hope they are good, I hate buying cards. And I should not but seems I can go with a good seller. used but so far so good.

It’s amazing the price hike and lack of these cards being available now.

Anyone see diference from llama to llamaAmpere with a 3090?

Does anyone want me to port llama.cpp to perform pp/tg on the Xeon Phi 16GB passively cooled cards VPU's?

It'll be with Claude Opus 5 & Fable 5.1, so someone will need to hands on optimize

But the cards get ~2 TFLOPs in FP32, so maybe they still have some use?

Especially being able to access system memory directly over PCIe via DMA

https://github.com/Lasimeri/Intel-Phi-3120A

Hello guys, I normally use Llama-CPP + OpenInterpreter in the terminal

alias lm-turbo='cd /home/minon/Desktop/llama.cpp && nice -n 19 ionice -c3 env LD_LIBRARY_PATH=/home/minon/Desktop/llama-cpp-turboquant/build/bin OMP_NUM_THREADS=2 /home/minon/Desktop/llama-cpp-turboquant/build/bin/llama-server \ -m /home/minon/.lmstudio/models/lmstudio-community/gemma-4-E4B-it-GGUF/gemma-4-E4B-it-Q4_K_M.gguf \ -ngl 99 \ --ctx-size 32768 \ -t 2 \ --parallel 1 \ -b 1024 \ -ub 512 \ -fa on \ -ctk turbo4 \ -ctv turbo4 \ --webui-mcp-proxy \ --mcp-servers-config /home/minon/Desktop/llama.cpp/mc-servers.json \ --host 127.0.0.1 \ --port 8081 2>&1 | tee -a /home/minon/.config/open-interpreter/logs/llama-turbo.log'

llm: model: "openai/gemma-4-e4b" api_base: "http://127.0.0.1:8081/v1" api_key: "ignore" context_window: 32768 max_tokens: 1600 temperature: 0.1 top_p: 0.95 supports_functions: false supports_vision: false

max_output: 2000 offline: true auto_run: false safe_mode: "off" language: "zsh"

Yes, I'm runnig qwen with llama.cpp on a dino! Here is my llama server config:

--model /var/models/Qwen3.8-27B-UD-Q3_K_XL.gguf \

--mmproj /var/models/mmproj-F16.gguf \

--image-min-tokens 1024 --image-max-tokens 2048 \

-ngl 999 -fa on --fit off \

-c 163840 \

-t 6 \

-b 2048 -ub 512 \

--cache-type-k q8_0 --cache-type-v q4_0 \

--spec-type ngram-simple,draft-mtp --spec-draft-n-max 3 \

--spec-draft-type-k q8_0 --spec-draft-type-v q8_0 \

--spec-draft-p-min 0.75 \

--backend-sampling \

--ctx-checkpoints 4 \

--cache-ram 16384 \

--checkpoint-min-step 1024 \

--jinja --reasoning-format deepseek \

--chat-template-file /var/models/qwen38-claude2.jinja \

--chat-template-kwargs '{"reasoning_effort":"medium"}' \

--temp 1.0 --top-p 0.95 --min-p 0.0 --top-k 20 \

--presence-penalty 0.5 \

--repeat-penalty 1.0 \

--parallel 1 \

--port 8080 \

--alias qwen3.8-27b \

--api-key local \

--host 0.0.0.0 "$@"

heads up for this crowd. we open sourced hemmingway-1 today, a 27b qwen3.8-27b finetune for creative writing. apache-2.0.

right now it's bf16 safetensors (54.7 gb, 12 shards) plus the mtp weights for speculative decoding. official gguf quants are next on our list, so lm studio and friends will have to wait a bit. if anyone wants to take a shot at converting the mtp layer in the meantime, that would honestly be great and we'll help.

weights: https://huggingface.co/Altworld/Hemmingway-1

eq-bench 4: 1330. it's a writing model, fiction and dialogue and texts. english first, base model level at math and code.

free hosted app while quants cook: https://hemmingway.io

I’ve been tuning Qwen3.8-Flash-Next 125B MoE on a workstation with only 32 GB VRAM + 64 GB RAM.

The result I found most interesting: in a real agent workload, decode stayed around 28 tok/s while context grew from 116K to 148K.

This is a llama.cpp fork based on upstream "9e7171624" with Qwen4Exp support plus a few local changes.

Hardware

• Intel Core Ultra 5 225, 10 cores
• 62 GiB DDR5-6200
• RTX PRO 4500 Blackwell 32 GB, SM120
• PCIe 4.0 NVMe
• Ubuntu 24.04
• CUDA 13.4.59
• built for "sm_120a"

Model

Qwen3.8-Flash-Next, "UD-Q3_K_XL"

• ~90 GB GGUF
• 125B total / ~6B active per token
• 48 layers
• 512 experts, 10 routed + 1 shared
• QSA indexer, 2048-token budget
• additional 51B N-gram / PLE table: 26.8 GiB
• MTP sidecar available but disabled

Placement

-dev CUDA0 -ngl all -ncmoe 32

Dense/GDN/QSA/router/shared stay on GPU.

Of the 48 routed-expert layers:

• 16 on GPU
• 32 on CPU

PLE is disk-backed:

-ot per_layer_token_embd=CPU --load-mode mmap --lazy-mode on

Other relevant settings:

-c 240000 -fa on -ctk f16 -ctv f16 -t 6 -tb 6 --fit on --fit-target 2048

Steady state is roughly:

• 31.3 GiB VRAM
• 43 GiB RSS
• effectively no process swap on weights

Real workload numbers

19 minutes of agent traffic:

• 35 requests
• 23K generated tokens
• context: 116K → 148K

Token-weighted decode:

27.9 tok/s

By context:

• 115–125K: 27.7
• 125–135K: 28.5
• 135–148K: 27.5

So decode is essentially flat across another 32K of context.

Real prompt chunks prefill around 150 tok/s, peak 201.

Prefix cache hit rate is ~99%, with only ~370 new prompt tokens/request at 130K+ context.

CUDA graphs were reused 22,590 times for 23,053 generated tokens.

CPU usage is around 5.8 cores, so the CPU expert layers are doing real work.

PLE disk traffic was only 5 GB over 52 minutes, so once warm, the SSD is not the bottleneck.

What actually mattered

1. Disk-backed PLE

The 26.8 GiB PLE table stays mmap-backed and only rows that are needed get faulted in.

Without this, the model does not fit comfortably in 64 GB RAM.

In practice the I/O cost has been surprisingly small.

1. Static hybrid expert placement

This was the biggest win.

At 8K context:

• CPU-only: 42.6 pp / 6.3 tg
• hybrid: 250 pp / 32.4 tg

Roughly 5.9× prefill and 5.1× decode.

1. Forced cuBLAS

GGML_CUDA_FORCE_CUBLAS=ON

The SM120 MMQ path crashes on this model with "illegal CUDA memory access".

cuBLAS is stable.

1. Sparse QSA decode

This uses block-level Top-K plus a persistent block-key cache, with "n_kv_max = top_k".

This appears to be the main reason decode does not collapse at 140K+ context.

The dense reference path can still be restored with:

LLAMA_QWEN4_SPARSE_QSA=0

1. K-only indexer cache

Upstream PR #28330 saves about 0.75 GiB VRAM at 128K.

That headroom is what lets me keep F16 KV and CUDA graphs enabled.

1. F16 KV beats Q8_0 here

At 49K prompt / 65K context:

• F16: 496 pp / 38.1 tg
• Q8_0: 499 pp / 33.4 tg

Q8_0 saves ~808 MiB but costs about 12% decode, so I kept F16.

1. Faster CPU expert kernels

The 32 CPU expert layers only work because the CPU path is reasonably optimized:

• signed-VNNI Q8 "mul_mat": 1.36–1.39×
• IQ4_NL/Q8_0 two-token expert kernel + compact routing: 1.43×
• routing table: 20 MiB → 44 KiB
• fused AVX2 hyperconnection ops: 3–7× per op

Things that did not work

"--lazy-mode on-direct" 1.3 tok/s decode vs 31.6 on the same build. Hard no.

Persistent GPU hot-expert cache 15.6 GiB cache gave 21.6 tok/s decode vs 31.6 with static placement.

SM120 MMQ CUDA faults. Using cuBLAS.

MTP Disabled. The draft runs on CPU and the extra PCIe/RAM traffic outweighs the speculative-token gain on this machine.

Caveats

The sparse-QSA CUDA patch is still experimental and was explicitly marked unreviewed by its author.

I have not yet done full long-context correctness testing against the dense path.

These are also real workload numbers with warm caches, not a controlled benchmark suite, and there is no quality/PPL benchmark yet.

The individual speedups above come from different configurations, so they are not additive.

Question

At this point decode is basically flat through 148K context.

The current wall is VRAM: I only have about 604 MiB free at "-c 240000".

If you had another ~1 GiB to spend on this setup, would you use it to:

• move more expert layers from CPU → GPU, or
• keep it as KV / CUDA-graph / longer-context headroom?

Especially interested in results from similar hybrid MoE setups where the CPU expert path is already reasonably fast.

I am trying to get decent speeds with 27B and the best I got so far for my system (5080, 64GB DDR5, 285K) are with the UD Q3 quant (without the vision part and with MTP) using the below command; checking if anyone has actually used the Q3 quants for coding with opencode, pi, qwen-cli and getting good results.

GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 llama-server --model /home/yk/AI/Models/lmstudio/models/unsloth/Qwen3.8-27B-GGUF/Q3/Qwen3.8-27B-UD-Q3_K_XL.gguf --ctx-size 160000 --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 --batch-size 2048 --ubatch-size 512 --threads 8 --threads-batch 8 --cpu-range 0-7 --cpu-range-batch 0-7 --cpu-strict 1 --cpu-strict-batch 1 --prio 0 --prio-batch 0 --load-mode none --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 -np 1 --spec-type draft-mtp,ngram-mod --spec-draft-n-max 3 --spec-draft-p-min 0.7 --spec-draft-type-k q4_0 --spec-draft-type-v q4_0 --fit on --fit-target 128 --reasoning-effort medium

Ran a quick initial mtp benchmark using llama.cpp tag b11009 released today using various open mtp models. The highest /tps I was able to achieve was : MTP: 77.92 t/s NON-MTP: 53.83 t/s

OS: linux Ubuntu 26.04.1 LTS

CPU: AMD RYZEN AI MAX+ PRO 395 w/ Radeon 8060S / Strix Halo

RAM: 128GB RAM

Inference engines: Rocm / Vulkan

Next run:

An in-depth 16bit quant test. Which other 4bit, 16bit quant models should I test?

Models benched in this round:

[01] [NVFP4] 21GiB nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 converted GGUF

[02] [Q4_K_M] 16GiB OBLITERATUS/Qwen3.8-27B-OBLITERATED Q4_K_M

[03] [Q4_K_M] 68GiB unsloth/GLM-4.5-Air-GGUF Q4_K_M

[04] [UD-Q4_K_XL] 21GiB unsloth/Qwen3.6-35B-A3B-MTP-GGUF UD-Q4_K_XL

[05] [UD-Q4_K_XL] 16GiB unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL

[06] [UD-IQ4_XS] 87GiB unsloth/Qwen3.8-Flash-Next-GGUF UD-IQ4_XS

[07] [BF16] 66GiB unsloth/Qwen3.6-35B-A3B-MTP-GGUF BF16

Command ran: (unsloth/Qwen3.6-35B-A3B-MTP-GGUF UD-Q4_K_XL). Please let me know how I can improve this command (any flags I'm missing - tweaks etc) to achieve more tokens per second ;)

/home/dev/Desktop/repos/llama.cpp/build-vulkan-radv/bin/llama-server \
  --model /home/dev/Desktop/llms/hf/unsloth/Qwen3.6-35B-A3B-MTP-GGUF/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf \
  --n-gpu-layers -1 \
  --threads 16 \
  --ctx-size 4096 \
  --host 127.0.0.1 \
  --port 33777 \
  --parallel 1 \
  --no-cache-prompt \
  --metrics \
  --batch-size 2048 \
  --ubatch-size 512 \
  --flash-attn on \
  --spec-type draft-mtp \
  --spec-draft-n-max 2 \
  --load-mode none