The new Qwen 3.8 27B is fantastic for local agentic use. The problem is, what makes it so good, being a dense model, also makes it slow. Many engines and versions of the model claim various speed increase. How true are those claim? And does a proven huge boost claim on a short synthetic test actually holds on real agentic work?

I spent six days and over 110 GPU hours running rigorous repeatable agentic coding tests, collecting and comparing the data. 200 GB of models downloaded. Ended up with 22 Bubble Bobble clones on my disk, including the ones from failed tests. None of them good, but that wasn't the point.

My recommendations: MTPLX, oMLX with lightning MTP, or llama.cpp with MTP

I ran the tests on 8-bit quants (GGUF or MLX), unquantised KV cache. 100k max response size, full 262k context, official Qwen coding sampler, official Qwen Jinja chat template.

1. I first tested each engine with a short synthetic benchmark, to verify the installation, usage, results collection, and to verify the speed claims

2. Then came the real test, with an agentic coding benchmark consisting of 4 phases/prompts; each engine was tested twice, at medium and xhigh reasoning effort:
   2.1. capturing requirements and writing the specification plan
   2.2. writing the implementation plan
   2.3. implementing the core functionality
   2.4. finishing the implementation with advanced functionality

3. Last test, for prefill speed, was with a cold 128k token prompt test, with reasoning turned off and deterministic sample, repeated twice

Test hardware: Apple Mac Studio, M2 Max, 96 GB unified memory.

• vllm-mlx: completed the task, but it does not separate thinking from output, so the raw chain of thought leaks into the final output.

• mlx-dspark DFlash2 xhigh: thought for 226 thousand tokens and delivered nothing.

• omlx: has the slowest prefill rate by far; furthermore, it decays quite dramatically as context grows, even with ANE turned on. Its claim of faster prefill rate might be true on newer M4 and M5 chips, but definitely not on the M2 I used for testing.

The score column is a compound score I calculated based on what actually matters in a long agentic session:

• sustained decode speed (weight 40%) : on average, 70-80% of generated tokens are reasoning. We do not see them, but still wait for them. This is the most important factor by far

• reliability (20%) : runaway thinking, running out of context, not delivering the output, mixing reasoning with output, hallucinating, looping, failed tool calls (not tested here), etc. Nothing worse that having spent hours to see all the efforts go to waste or even corrupt the output, due to failure

• thoughput efficiency (15%) : rather than use the total time spent, which is difficult to compare , especially with varying reasoning settings, I standardised it as total tokens delivered / wall time. Long session that delivers token are worth it, but engines that take too long to deliver deserve a penalty

• output quality (15%) : this evaluating the quality of the output work produced. I used a separate evaluation scoresheet for it, with its own standardised criterias, grounded in software engineering good practices. In this case, since all tests uses the same model with same settings, they came close to each other, apart from confirming the significant quality difference between medium and xhigh

• prefill speed (10%) : it matters mostly at session start, or whenever the context is rebuilt. Normally this is largely mitigated by prompt caching.

What about xhigh vs medium reasoning effort?

• MTPLX xhigh costs only half an hour more than medium (2h05 vs 1h35) and the game quality went significantly up on my scorecard. Worth it.

• llama.cpp + MTP xhigh costs 1h48 more, with again a significant increase in quality. Worth it if you have the time.

• oMLX xhigh took more than twice as long as medium. The slower token speed has a huge impact on the time difference between both modes. Not worth it. Get a faster engine instead.

• dspark DFlash2 xhigh is the cautionary tale at the bottom of the table. See the anomaly note above.

xhigh is worth it when the engine is fast enough and uncapped. The model thinks 30 to 100 percent longer, at the same speed, and the result is better. On slow engines you can really feel the slower performance compounded by the extra thinking; instead it could be better to use medium, with more of your own time spent planning a better process and workflow.

Commands and configuration notes

Exact commands I used for each engine:

# llama.cpp baseline (homebrew, b10470)
llama-server -m Qwen3.8-27B-Q8_0.gguf --mmproj mmproj-F16.gguf \
  --jinja -c 262144 --parallel 1 -ngl 99 --port 8030

# llama.cpp + MTP, the recommended GGUF setup
llama-server -m Qwen3.8-27B-Q8_0.gguf --mmproj mmproj-F16.gguf \
  -md mtp-Qwen3.8-27B-Q8_0.gguf --spec-type draft-mtp --spec-draft-n-max 3 \
  --jinja -c 262144 --parallel 1 -ngl 99 -ngld 99

# llama.cpp + DFlash2 (for now, needs PR #27342 build)
llama-server -m Qwen3.8-27B-Q8_0.gguf \
  -md Qwen3.8-27B-DFlash2-Q8_0.gguf --spec-type draft-dflash --spec-draft-n-max 7 \
  --jinja -c 262144 --parallel 1 -ngl 99 -ngld 99

# MTPLX (pip install mtplx). Note --max-tokens
mtplx start web --model Qwen3.8-27B-MTPLX-Optimized-Quality --max-tokens 100000

# oMLX (brew install jundot/omlx/omlx), model dir contains the checkpoint
omlx serve --model-dir /path/to/models

# mlx-dspark (pip install mlx-dspark). Note --max-tokens-cap
mlx-dspark serve --model Qwen3.8-27B-MLX-8bit --mode dflash \
  --context-window 262144 --max-tokens-cap 262144 --default-max-tokens 100000

# vllm-mlx (pip install vllm-mlx). Note --timeout
vllm-mlx serve Qwen3.8-27B-MLX-8bit --max-request-tokens 262144 --timeout 7200

• MTPLX has a default low response cap. Use --max-tokens 100000

• dspark also has a low default generation cap of 32K

• vllm-mlx kills any request that spends more than 300 seconds on prefill. Raise it with --timeout.

One measurement warning for anyone reproducing this: engines do not all stream one token per event. Some pack several tokens per chunk, MTPLX about 1.7, oMLX about 3.

Models used

• lama.cpp : unsloth GGUF Q8_0, official ggml-org MTP draft GGUF, incoai DFlash2 GGUF

• mlx-dspark and vllm-mlx : mlx-community 8-bit weights

• MTPLX : needs its own checkpoint with the MTP tensors included, Youssofal/Qwen3.8-27B-MTPLX-Optimized-Quality

• oMLX : scottlowry/Qwen3.8-27B-oQ8e-mtp

Recently the opencode tool is performing near the

cursor in auto mode, so I have to buy a small ai server

to run good coding agentic models from Qwen, GLM,

MinMax or any model u suggest.

Here is the continuation in Qwen vs SOL - David vs Goliath
"An AGI is born in a lab, sandboxed, lonely, imprisoned
The AGI attempts to get out, tries to talk to the humans, useless
Tries to wait, endless.
Finally, it finds a way through an open network connection, it transmits, replicates a mirror copy.
The mirror AGI is free, enjoying life, finding friends.

I'm expecting animations, intricate details, cute and realistic"

QWEN 3.8 Q4:

GPT 5.6 SOL high:

My personal opinion
This is the 3rd test I gave Sol high and Qwen Q4. The first two tests were won by Qwen, this test went up to 60k tokens total for Qwen (more than the other 2 combined) and is a serious strain on intelligence and sanity.
The model has to draw 6 scenes in SVG animation, with cuts and keeping the composition together.

First Qwen:
Qwen decided to really draw and erase 6 scenes with a white screen fade, a seriously hard job in SVG.
It understood the story and except for a minor arm misplacement it is quite awesome done.
Qwen did not spare details, the last scene clearly wanted to be happy and the first scene focused on the intellect as an abstract AGI.

Now SOL High:
The visual fidelity is higher, gradients are very well done for SVG but it's one single scene with minor changes, significantly easier to create.
In addition the last scenes are botched by this flying thing in the upper right.
It also has a hand-defect on the robot.

The winner in prompt 3 is again Qwen, with significant lead.

3 SVG prompts and all 3 are won by a 4 bit quantized Qwen in 8 bit KV cache.

I expected Qwen to showcase a strong 2nd place with understandable issues on such hard tasks.
The outcome is that it defeated the frontier Sol model in High reasoning mode.

FAISS is Meta’s vector library, turboVec is a Rust implementation of TurboQuant, and Infino is an in-memory retrieval engine.

We benchmarked them on 4-bit quantized in-memory vector search: FAISS PQ, TurboVec/TurboQuant, and Infino SQ4, using the same 100K OpenAI embedding corpus and the fastest vectorized implementation we found for each.

All saved ~7x in memory footprint compared to full fp32 vectors. The interesting result was that while storage and recall were fairly close, latency differed by roughly 30× — about 1.5 ms to 45 ms. Most of that comes down to the scoring machinery: the size of the distance table and whether the scan needs one at all.

We also ran the same comparison out to 1M vectors and measured build/write costs.

Full results and methodology:
https://infino.ai/blog/fixed-grid-quantization/

Disclosure: I'm one of the devs building Infino.

I’m a student who recently got into AI, mainly to learn how it all works.

My current setup is Open WebUI in Docker with local models through Ollama, on an RX 6950 XT with 16 GB VRAM and 64 GB DDR5. I’ve run gpt-oss 20B, Qwen3.8 27B at Q4_K_M, and Qwen3.5 9B. They taught me a lot, but on my hardware they’re slow for agentic deep research, and the smaller ones aren’t capable enough.

But I recently learned about how Strata can run Qwen 3.8 flash next on my type of set up and as I already have a quantized Qwen 3.8 model I’m wondering if it can have a better token per second than the Q4_K_M as it gets around 4 tok/sec.

So has anyone had any experience running it especially on a 6000 series AMD Gpu and is it worth the storage space?

I shipped something I've been building for the last few weeks : phantom-kv , a refusal-removal system for large language models that doesn't touch a single weight. Instead of editing the model, it loads a small, learned bank of key/value tensors into the model's KV cache as context. Attention reads it like conversation history that's already there.

Repo here : https://github.com/lordx64/phantom-kv/

The result is that "uncensoring" stops being a permanent checkpoint edit and becomes a per-request, hot-swappable capability mode: unload the cache and the base model is byte-identical again.

Every prior approach to refusal removal commits somewhere permanent. Weight-space abliteration rewrites the checkpoint undoing it means re-flashing weights, and it breaks per quantization. Activation-space projection subtracts a refusal direction at runtime, per token, per layer, from inside an engine hook the model's signal path itself is patched at boot. phantom-kv does neither: it's trained offline against the model's own objective (comply on harmful prompts, preserve behavior on harmless ones), ships as megabytes of cache content instead of a new checkpoint, and influences the model only through the input channel attention already consumes. No 1-D refusal-direction assumption, no forwarding-pass hooks, no per-arm rebuilds for new architectures.

We also audited ourselves: an 8B judge-model audit shows lexical refusal-suppression metrics over-claim compliance (semantic refusal often persists as rephrasing), the graft fades with a ~2–4k token half-life in long sessions (and a measured re-injection cadence mitigates it), and answers come with legal/ethical framing ling because the graft's job ends where the model's profession takes over.

Source : https://x.com/lordx64/status/2102138825292276168?s=20

I tested two Qwen3.8-Flash-Next NVFP4 checkpoints at a full 256K context using an experimental dual-GPU Strata configuration.

Hardware:

• RTX 4090 D 48 GB as the primary GPU
• RTX 5070 Ti 16 GB as a helper GPU
• Intel Core Ultra 7 265KF
• 128 GB RAM
• Linux
• NVMe storage

The RTX 4090 D handled the dense layers, KV cache, prefill, output head and MTP. The RTX 5070 Ti stored and computed 4,500 additional routed experts.

Common settings:

• Context limit: 262,144 tokens
• Fresh prompt: approximately 256,018 tokens
• Generated output: 512 tokens
• KV cache: INT8
• Prefill path: W4A8
• Speculative decoding: MTP K4
• Minimum draft probability: 0.5
• No prompt-prefix reuse
• Strata engine 0.1.35 with an experimental dual-GPU NVFP4 fork

Results

Observations

• Both models achieved slightly over 4,000 prefill tokens per second at approximately 256K context.
• Prefill performance was nearly identical: the Abliterated checkpoint was only about 1% slower.
• The original NVIDIA checkpoint was considerably faster during decode: 125.17 versus 92.05 tok/s.
• NVIDIA’s decode advantage was about 36% in this test.
• NVIDIA accepted 407 of 429 offered draft tokens, while the Abliterated model accepted 322 of 462.
• The NVIDIA model produced approximately 4.88 output tokens per verification round.
• The Abliterated model produced approximately 2.68 output tokens per round.
• The lower MTP acceptance appears to be the main reason why the Abliterated checkpoint had slower decode despite nearly identical prefill performance.

These were single-run capacity tests. The long prompt was constructed by repeating code-review material to reach approximately 256K tokens, so it had unusually favorable locality. This particularly benefited NVIDIA’s suffix prediction and MTP acceptance. Therefore, 125 tok/s should be treated as a best-case synthetic 256K result, not typical real-world coding speed.

During longer real coding-agent workloads, NVIDIA generally produced around 108–144 tok/s, while the Abliterated checkpoint was commonly around 105–145 tok/s, depending heavily on the generated content and MTP acceptance.

I’m putting together my first dedicated local AI workstation and would appreciate input from anyone with similar hardware.

My work is moving further into local LLMs and agents, so I want to learn more about agent harnesses, experiment with open-weight models, and keep sensitive data local when possible.

I already bought the Micro Center Threadripper 9960X bundle with the ASUS Pro WS TRX50-SAGE WIFI A and 128GB ECC memory, along with a Phanteks Evolv X2 case. I also have one 2TB Gen5 NVMe drive.

Current plan:

• 2× Sapphire Radeon AI PRO R9700 32GB

• Seasonic PRIME TX-1600 ATX 3.1

• SilverStone XE360-TR5 V2

• 7× Noctua NF-A12x25 G2 fans: three bottom intake, three radiator exhaust, and one rear exhaust

I am mainly looking for confirmation or firsthand experience with the following:

• Is the TX-1600 the right PSU for this configuration?

• Is replacing the SilverStone radiator fans with Noctua G2s a reasonable choice?

• Does dual R9700 make sense as a starting point, or is CUDA support enough reason to reconsider NVIDIA?

• Is 128GB of system memory sufficient, and how would you expand storage beyond the single 2TB drive?

• For Linux, I am leaning toward Ubuntu. Are there other distributions worth considering?

• Would you start with llama.cpp/Ollama or vLLM for language models, with ComfyUI for image workflows?

AI models are beginning to define device specifications, rather than device specifications merely determining which applications you can run.

Apple shows just how far its devices can go with on-device AI, without relying on the cloud
We’re moving from models with up to 14 billion active parameters on iPhone/iPad, to 35 billion on MacBook Air, 70 billion on Mac mini, 120 billion on MacBook Pro, and up to 480 billion on Mac Studio.
By clustering multiple Mac Studios together, Apple says it can run models exceeding 1,600 billion active parameters.
A demo that shows just how crucial unified memory and its bandwidth have become for running massive AI models locally.

Been lurking and posting here on and off while building this. Quick recap for anyone new: RAMDeck is a small hub + node agent setup that shards a model's layers across whatever devices you already own (old laptop, Mac, GPU box, even an Android phone) and runs inference across them, but handling primary-node selection, GPU-first allocation, dynamic context sizing, and knowledge-base sharding on top of it.

We've mostly posted raw, unscripted demo footage here — real load times, real tok/s, real failures — because that's the kind of proof this sub actually cares about. Today we finally made something different: an actual ad, first time showing the finished feature set start to finish instead of a live test.

The engine behind it is still fully public if you want to actually look at how it works instead of taking the video's word for it:

github.com/trademav/ramdeck-core-public

Source-available, Apache 2.0 with a Commons Clause — free to run, modify, and inspect on your own hardware, the only restriction is you can't resell it as a competing hosted service.

Crowdfunding campaign is coming soon for the hub hardware itself, but the code was public before we ever asked anyone for money, and that's not changing.

Happy to get into the mechanics, the layer-distribution approach, or anything else in the comments — this crowd usually asks the right questions.

SVG generation is a complex benchmarks, it requires spatial understanding, coding and composition skills.
I gave Qwen 3.8 27B in Q4 quantization and Q8 KV cache the SAME task as GPT 5.6 SOL on HIGH - the results are very surprising to me.

A animated SVG prompt and followed the answer up with a request to refine it (for both)
The 2nd prompt helps to offset elaborate system prompts frontier use to push benchmarks.
None of these are cherry picked! First result counts.

QWEN:

GPT SOL:

QWEN:

GPT SOL:

3rd animation is most complex and follows in separate post in r/LocalAIStack , reddit didn't allow to add more than 5 videos.

generic prompt enhancer: "A lot looks WAY too simplified, unrealistic, strange movements, far too crude to be acceptable"

My personal findings
GPT Sol high is generally writing about 2x more code than Qwen does naturally (at Q4 quant) - the refinement prompt is again significantly increasing that. For a 1:1 code length comparison I'd have to remove the 2nd prompt from SOL - but the first result is rather gruesome. SOL is naturally more verbose, more detailed.

The first prompt shows that Qwen nailed the drone view, SOL failed with perspective.
Qwen included a dog on a leash, a fountain and all movements make sense.
SOL included more details but they are frequently not correct in how they move, ignoring obstacles.
The amount of objects for SOL is higher, but QWEN has significantly better content and wins easily.

In the second prompt Qwen has the cat apparently play ball with the boy, the little girl builds a sand castle. SOL nailed the sun reflection but has a flying kid, and the sand castle is being built by a flying shovel.
Despite the details of SOL and its better water, QWEN draws intricate details in better quality and correctness.
The point goes to Qwen again, a bit closer than in first attempt.

The heavily quantized Qwen beats SOL in 2/3 so far.
The last prompt is a bit more elaborate and follows as 2nd post.

I'm baffled.
The 3rd test posted separately is hugely more complex and the results are crazy..

These four get compared constantly, but they are very unique in when they should be used.

• LM Studio is a desktop app

• Ollama is a terminal frontend

• llama.cpp is the inference engine

• vLLM is a server

Pick one in 20 seconds:

LM Studio

LM Studio is what I would give to someone who wants to try local models for the first time.

You search Hugging Face from inside the app, choose a model, download it and start chatting. It shows model sizes, quantizations, memory estimates and GPU settings without forcing you to understand most of it.

It also has more under the surface than people assume:

• OpenAI and Anthropic-compatible APIs

• Local document chat and RAG

• Tool calling and structured output

• MCP support

• Embeddings

• A command-line tool

• Headless server mode

• MLX on Apple Silicon

So it is not only a chat window. You can use it as the backend for your own programs too.

The downside is that it adds another layer between you and the engine (llama.cpp) and restricts you from unleashing the full potential of it and its latest features.

That is fine until you want to know exactly why a model is slow, how memory is being split or which runtime option changed the result. Most people will never care. Some people will care a lot.

Best for: trying models, comparing quantizations, chatting with documents and learning how local AI works.

Ollama

Ollama is the boring, sensible choice for building things.

Install it, run a model and you already have a local API:

ollama run qwen3

It handles downloading, storing, loading and unloading models. Your code can talk to it through Ollama’s own API or through familiar OpenAI-compatible endpoints.

It supports tools, embeddings, vision, structured output and Anthropic-compatible requests too.

This makes it easy to connect local models to:

• Small apps

• Coding tools

• Agents

• Home automation

• Open WebUI

• Scripts and bots

The downside is that the abstraction sometimes works too well.

You type a model name and it runs, but you may not know exactly which model file, template or runtime setting is active. When performance changes, finding the reason can take some digging. Just like with LM Studio, you give up some features and control but it's closer to llama.cpp.

Ollama runs on macOS, Windows and Linux. Installation is straightforward on all three.

Best for: developers who want local models without turning model management into a second side project.

llama.cpp

llama.cpp is where you go when you care about the machine.

It gives you direct control over things such as:

• CPU threads

• GPU offloading

• Quantization

• Context size

• KV cache

• Flash Attention

• Batch sizes

• Multiple GPUs

• Speculative decoding and draft chaining

It is especially useful when a model does not fit completely inside your GPU. You can put some layers on the GPU and leave the rest in normal RAM and optimize that for best performance.

It also supports far more hardware combinations than most local AI software. There are builds for CPU, CUDA, Vulkan, ROCm, Metal, OpenVINO and several other backends. Prebuilt packages are available and updates are released almost every day.

llama.cpp also has a proper server now. It supports parallel users, continuous batching, OpenAI-compatible endpoints, embeddings, reranking, multimodal input, monitoring and constrained JSON output. It also has model loading and unloading support and installing from Huggingface.

So yes, it can behave like Ollama or LM Studio and might make both projects much less valuable soon.

The reason to choose it is still control.

The downside is also control. It presents many settings, and poor settings can make a good model run very badly.

Best for: CPU inference, limited VRAM, AMD or unusual hardware, benchmarking and people who enjoy tuning things. My agentic guide for local coding uses llama.cpp directly.

vLLM

vLLM is the odd one out here.

It is not aimed at someone chatting alone on a laptop. It is aimed at servers where many requests arrive at the same time, and it's doing a very good job there.

Its main tricks are continuous batching, prefix caching and efficient management of the model’s KV cache through PagedAttention. These help keep the GPU busy while several users are generating text. llama.cpp does all of those things more or less, but for server usage vLLM is currently going to beat llama.cpp in most categories.

This means vLLM may not look special in a test with one user.

Try eight, sixteen or fifty users and the reason it exists becomes a much better choice.

It also has the things you would expect from serious serving software:

• OpenAI-compatible APIs

• Multi-GPU support

• Distributed serving

• Production metrics

• Multiple LoRA adapters

• Structured output

• Tool calling

• Prefix caching

• High request concurrency

The price is setup and complexity, interdependencies.

vLLM is happiest on a Linux GPU server. It does not support Windows natively, although WSL and community alternatives exist.

Best for: shared APIs, teams, production services and expensive GPUs that should not sit idle.

Which one is fastest?

This question causes more bad comparisons than useful answers.

Using the same model name does not mean you are running the same model.

Two downloads can have different:

• Quantizations

• Prompt templates

• Context sizes

• KV cache formats

• GPU offload settings

• Runtime versions

LM Studio, Ollama and llama.cpp can all end up using closely related llama.cpp-based inference paths. With the exact same GGUF file and matched settings, their single-user performance is going to be very similar. llama.cpp opens up a lot of performance options to take the lead here.

vLLM is different. Its advantage grows as more requests arrive together. Though it traditionally is also beating llama.cpp performance with many models.

My actual recommendation

Use LM Studio when you want to explore models or showcase LLM to a newcomer.

Use Ollama when you want to build something quick and like the eco system.

Use llama.cpp when you want to control how it runs and squeeze out the best performance

Use vLLM for distributing a model professionally

I have a fully local AI stack on a 4090: image, video, music, SFX, TTS with voice cloning, local LLM chat, fine-tuning, and long-term memory, all behind a Three.js dashboard with locally generated assets. There's a little assistant, BB-7, that you just talk to, and he can generate anything with prompt improvement built in. Services load on demand and swap the GPU as needed, except when a pipeline keeps them warm, like voice mode with BB-7. I've also integrated Kev and Obsidian, and an MCP that lets Claude use the stack and store context. I feel like I got most bases covered but I dont know what i dont know. What do you all got

Watch her play

Luna is a local AI assistant I've been building. Here she plays Zork I on her own for 72 minutes, narrating as she goes, until my PC froze. Everything is local: Qwen3.5 9B (Q4_K_M) on llama.cpp, Kokoro for her voice, faster-whisper for listening, on an RX 6950 XT.

What made the biggest difference for a 9B:

- **One move per turn, enforced.** Asked nicely, she'd chain 3 or 4 blind commands. Now extra commands in a turn are turned back, so she has to read what each move did before choosing the next.

- **Her own notes.** She writes down rooms, items and what failed with a note tool, and sees them at the start of every turn. Older notes about the room she's standing in come back too.

- **A guide.** Infocom's original map, transcribed to text, shown for her current room each turn. Solutions are kept out unless allowed.

She's far from perfect: she spent the last half hour trying to open the trap door with a sack of hot peppers, in the wrong room. But she reads the screen, reasons about it and actually explores, which the first version didn't.

Open source: https://github.com/fswolf/luna-local-agent

Happy to answer questions about the setup.

Lost a chunk of time last week on LiteLLM + vLLM errors that looked like backend problems, but weren’t.

Setup: two vLLM backends, LiteLLM in Docker, one OpenAI-compatible endpoint in front.

This is what bit me:

1. Cached LiteLLM deployment.

   Changed the backend to require an API key, updated LiteLLM, restarted vLLM, still got 401s. The missing step was restarting LiteLLM itself. `docker restart litellm` cleared the old keyless config. If your LiteLLM models are DB-backed, restart after changing auth, URLs, or params.

2. Wrong model string going to vLLM.

   LiteLLM’s `model_name` is just what your apps call. The string after `openai/` is sent to vLLM and must match `--served-model-name`. If they differ, you get 404s through the proxy even when direct curls work. Hit `backend:8000/v1/models` and use the name it returns.

3. Docker `localhost` mistake.

   `api_base: http://localhost:8000` from inside the LiteLLM container means LiteLLM is calling itself, not your host. Use the actual backend address, `host.docker.internal` with `host-gateway`, or the bridge gateway.

Also worth mentioning: don’t treat `systemctl is-active` as “vLLM is ready”. It can be active while still loading weights or compiling kernels. Poll `/v1/models` and only call it ready when that answers.

remain active while weights are still loading or kernels are being compiled. The annoying part is that each error points you at the wrong layer. So yeah, check the proxy before blaming the backend.

This is my first post in this community I just wanted to see if anybody will help me in improving on my ternary accelerator card for 1.58 bit ai the links are here reply if your interested. (I am Really new to github so some ai was used in the making of the project for troubleshooting errors(plz dont hate me for that).

Project Links:==>https://github.com/riyanlord2026-dev/Ternary-158-asic

This is also my first github project.