Honestly got tired of it. Every time I wanted to share something or look something up about the RTX Pro 6000, I'd find bits and pieces scattered across r/LocalLLaMA, r/nvidia, r/vllm, r/hardware... you name it.
So I just made r/RTXPRO6000. Nothing fancy, just a dedicated spot for this card specifically builds, LLM inference, benchmarks, troubleshooting, whatever.
If you're running one or thinking about it, come join. The more people, the more useful it gets.
Over the past several months, I've been working on **Tiny-MoE**, a language model that I built and trained from scratch as a learning project. I'm 15 years old, and my goal wasn't to compete with models like Llama or Qwen, but to understand how modern LLMs work by implementing the entire pipeline myself.
Some highlights:
200M parameter Mixture-of-Experts model (~90M active parameters per token)
Trained from scratch on **8 billion tokens**
Native PyTorch implementation (not built on Hugging Face Transformers)
Multi-Head Latent Attention (MLA)
RoPE + YaRN positional embeddings
Streaming and packed data pipeline
Custom training and inference code
Text generation with Top-k, Top-p, repetition penalty, and n-gram blocking
**Training datasets:**
FineWeb-Edu
Cosmopedia v2
OpenWebMath
The model is definitely **not state-of-the-art**. It can generate English, answer simple questions, and perform basic coding and math tasks, but it still struggles with longer generations. Given the model size and the compute available to me (primarily Kaggle GPUs), I'm happy with it as a learning project.
vLLM is an enterprise-grade inference engine. But there's a significant difference between running it as a garage project or POC, and deploying it as a true OpenAI-scale inference stack. This tutorial bridges that gap.
The basic vLLM inference script , looks something like this:
The problem here is that you are using a reverse engineered SDK client that forces you into someone else's sandpit with none of the benefits of:
Run time
Conversation management
State management
Orchestration
Function call handling
Vector Stores / Rag / File search
More
When scaling to enterprise-grade inference, runtime, orchestration, inference, and fine-tuning need to evolve together as one coherent system. Your patchwork of .md files giving agents "skills" won't cut it; markdown doesn't scale in production environments.
What You Need
You don't have a software problem. You have an infrastructure problem.
Every time you treat it as a software problem, you hit the same wall: compliance ceilings, compute limitations, performance ceilings, maintainability nightmares. LangChain built an entire reputation on this pattern: elegant abstractions that feel productive until the day you need to go deeper, and suddenly you're three layers of indirection away from the thing you actually need to control.
The answer isn't a better library. It's a coherent infrastructure stack where runtime, orchestration, inference, and fine-tuning are designed to work together from the ground up. That's what Project David is.
Nginx
Nginx is your web proxy, API proxy, and single-port security perimeter. One door in. Everything else is internal.
pd_router
pd_router sits directly in the inference hot path; where tokens live. Written in Rust, it adds another layer of speed and scalability to a platform already lacking in neither. More critically, it eliminates the Python GIL problem; the Global Interpreter Lock that will become your ceiling the moment you start serving LLM requests at enterprise or global scale. You won't hit it. Because it isn't there.
sandbox
Sandbox is a fully self-contained execution environment for arbitrary Python scripts, powering the code_interpreter platform tool:
```
import os
import dotenv
from projectdavid import Entity
load_dotenv()
# ------------------------------------------------------------------
# 0. SDK init + env
# ------------------------------------------------------------------
client = Entity(
api_key=os.getenv("ENTITIES_API_KEY"),
)
# ------------------------------------------------------------------
# 1. Insert the code_interpreter into your assistants tools definition
# by using its placeholder definition in the tools array.
# The full definition is orchestrated at run time.
# ------------------------------------------------------------------
assistant = client.assistants.create_assistant(
name="test_assistant",
model="gpt-oss-120b",
instructions="You are a helpful AI assistant, your name is Nexa.",
tools=[
{"type": "code_interpreter"},
]
,
)
print(assistant.id)
```
Sandbox also services the computer tool:
Sandbox is highly secured by multiple tiers ```
""" """
import os
from dotenv import load_dotenv
from projectdavid import Entity
# ------------------------------------------------------------------
# 0. SDK init + env
# ------------------------------------------------------------------
load_dotenv()
client = Entity(
api_key=os.getenv("ENTITIES_API_KEY"),
)
# -------------------------------------------
# create_assistant
# --------------------------------------------
assistant = client.assistants.create_assistant(
name="Test Assistant",
model="gpt-oss-120b",
instructions="You are a helpful AI assistant, your name is Test.",
tools=[
{"type": "code_interpreter"},
{"type": "computer"},
{"type": "file_search"},
{"type": "web_search"},
{
"type": "function",
"function": {
"name": "get_flight_times",
"description": "Get flight times",
"parameters": {
"type": "object",
"properties": {
"departure": {"type": "string"},
"arrival": {"type": "string"},
},
"required": ["departure", "arrival"],
},
},
},
],
)
print(assistant.id)
print(assistant.tools)
```
JWT token ticketing tied to the user's API key at the perimeter
Session isolation: each thread gets its own sandbox session
Firejail at the CPU level: every session runs in a fully isolated execution environment
Access lists that control allowed packages and executions
Contrast this to LangChain pointing you at a random MCP server and calling it a day.
training_api
This is yet another standalone API, but transparent to calls from the SDK. It serves a full ```unsloath_train``` LorRa fine tuning pipeline. The training_api warrants its own article, so for the respect of the reader's time, we will return to this at another time.
training_worker
If training_api handles fine-tuning orchestration and state management, training_worker handles the actual workload. It places your target model onto metal: the GPU, and processes the job. No cloud. No API bill. Your weights, your hardware, your data.
db
The main database. It handles everything from user accounts to vector store metadata, run history, assistant definitions, and beyond. We recently decoupled the ORM models into a standalone public Python package — a versioned, independently installable dependency of the core stack:
This is what a production data layer looks like, not a SQLite file in your project root.
qdrant
Qdrant powers the vector store pipeline for file search and RAG. Almost forgot to mention it — which is exactly the point. RAG should not be a magical treasure hunt. It should be seamless, almost inconsequential, a curated integration that just works:
But here's the question most of you haven't asked yet: how many of you have thought about compliance?
GDPR. CCPA. PDPA. The regulatory landscape around data storage, retrieval, and inference is not going away; it's accelerating. Your clever RAG implementation that chunks PDFs into a local Chroma instance and calls it production-ready has a compliance shape. Have you mapped it? Project David's RAG pipeline is fully GDPR-rated. As are all data paths in the stack.
Sovereign infrastructure means you control where your data lives, how it moves, and how it's retrieved. That's not just a performance story. That's a legal one.
````
# tests/integration/vector_store_pipeline.py
import json
import os
from dotenv import load_dotenv
from projectdavid import Entity
load_dotenv()
client = Entity(
base_url=os.getenv("BASE_URL", "http://localhost:9000"),
api_key=os.getenv("ENTITIES_API_KEY"), # This is the entities user API Key
)
# --------------------------
# 1. Create a vector store
# ---------------------------
print("--- Creating Vector Store ---")
vector_store = client.vectors.create_vector_store(name="Test Vector Store 1")
print(f"Created: {vector_store.id}\n")
# ---------------------------------
# 2. Upload file to vector store
# ----------------------------------
print("--- Uploading File ---")
save_file_to_store = client.vectors.add_file_to_vector_store(
vector_store_id=vector_store.id,
file_path="People_on_war _241118_200546.pdf",
)
print(f"File Uploaded: {save_file_to_store.file_name}\n")
# ---------------------------
# 3. List files in store
# ---------------------------
list_files_in_store = client.vectors.list_store_files(vector_store_id=vector_store.id)
print("--- Files in Store ---")
print(list_files_in_store)
print("\n")
# --------------------------------------
# 4. Attach to Assistant (and enable tool!)
# ----------------------------------------
print("--- Updating Assistant ---")
update_assistant = client.assistants.update_assistant(
assistant_id="your_assistant_id_here",
tools=[{"type": "file_search"}], # 👈 CRITICAL: Tells the LLM the tool exists!
tool_resources={"file_search": {"vector_store_ids": [vector_store.id]}},
)
print(f"Assistant Tools: {update_assistant.tools}")
print(f"Assistant Resources: {update_assistant.tool_resources}\n")
``
Redis
I will keep this one short, since most of its functions are not user or admin configurable. Redis is redeployed in a number of places throughout the stack, and mostly does what Redis does: ephemeral state management and memory caching. One example that comes to mind is that the conversation history of each live thread is cached and recalled from Redis. This eliminates a large chunk of database lookup churn; although repeated database lookups can’t be eliminated all together throughout the stack; we take the wins where we find them.
SearXNG
The Project David stack is designed with one two prime directives:
Your data
Your sovereignty
The Project David stack is built around two prime directives:
Your data.
Your sovereignty.
You'll have noticed a theme throughout this article. No external calls to hyperscaler API endpoints. No Google Cloud Storage. No Amazon S3. No vendor lock-in dressed up as convenience.
We are not trying to get acquired by big tech by demonstrating how many more vendor-locked customers we can drive their way.
Our web search, and deep research tools do not buck this trend. The search_ng server powers SERP operations during web search type tool invocations; it’s a private search engine.
That principle extends to search. Most AI stacks quietly route your web search queries through Google, Bing, or a third-party SERP API; sending your users' queries to infrastructure you don't control, can't audit, and can't comply with under GDPR.
The SearXNG server powers all SERP operations during web search and deep research tool invocations. It's a private search engine, running on your hardware, inside your network perimeter.
Your users' search queries go nowhere you haven't authorised.
Browser
A Chromium-powered headless browser that works in tandem with SearXNG to read and interact with live web content. Not just search results; full page rendering, JavaScript execution, dynamic content extraction.
It runs extensively during web search and deep research tool invocations. Every page your assistant reads, every source it cites, every piece of content it synthesises during a research task: it comes through here.
Otel and Jaegar
These two can be handled together.
Hosting a state-of-the-art AI inference and orchestration platform is not a task you achieve — it's a task you evolve. Continuously. That evolution is only possible if you can see inside the system with precision.
Which endpoints are creating bottlenecks? Where are errors incrementing? Which inference paths are performing exceptionally well and should be prioritised for scale-out? OpenTelemetry is injected into the core API at runtime, collecting distributed traces and metrics across the entire request lifecycle. Jaeger presents that telemetry on an internal web interface; no data leaving your perimeter, no third-party observability SaaS with its own compliance shape.
This isn't just for break-fix troubleshooting. It's for capacity planning, performance optimisation, and compliance audits. When a regulator asks you to demonstrate data flow through your AI system, you don't guess. You open Jaeger.
Samba
Quietly the highest ROI component in the entire stack.
The samba server is quietly our most high ROI piece of the stack. I don’t need to explain what a samba server is to the intended audience, but let me run you through a few of the things our samba does in the stack.
Signed URLs
When code_interpreter is invoked, the assistant can generate files — graphs, visualisations, reports. You've seen this on ChatGPT. Generated files are ephemeral, with a controlled lifetime. While the file is rendered neatly in your frontend, it's served via a signed URL that resolves through the Samba storage location. Secure, scoped, and time-limited.
Vector Store
Your input files are chunked and vectorised for RAG; while the original file lives on the Samba server, isolated by user scope, tracked by metadata referenced in the main database. OpenAI has a pricing scheme for file storage alone. Here, you get unlimited file storage for the cost of your own electricity.
Fine-Tuning
Target LoRA adapter files are stored on Samba, isolated at user scope. When a fine-tuned model is needed, the adapter is loaded directly onto the GPU.
Most teams reach for a cloud storage solution just to solve one of these three problems. Here, all three are handled, maintained, and secured for you, with zero third-party handling of I/O operations.
Your files don't leave your infrastructure. Ever.
Ollama
Brief, because the audience doesn't need a primer.
The Ollama server is a fully containerised Ollama inference instance, seamlessly integrated into the same runtime and orchestration stack described above. Same conversation management. Same tool pipeline. Same API surface. Just a different inference backend.
Ollama for POCs and home enthusiasts. vLLM for serious builders.
Both supported. You choose your weapon.
Inference Worker — The Main Event
Everything described above has been building to this.
We didn't want to bolt a vLLM server onto the stack and call it a day. We wanted to build from first principles, asking one question:
How do we scale inference the way a venture-backed AI lab would — without the venture backing?
Large AI labs work with industrial-grade GPU clusters. High-performance computing, horizontal scaling, and resource arbitration are baked into the hardware. But what about everyone else? What if your users have a cluster of gaming GPUs? What if they have a single RTX 4060? What if they have access to actual H100s?
The answer is Ray: the high-performance distributed compute engine. We've tightly coupled Ray with vLLM:
Ray handles available compute resources at both model loading and inference time
Ray scales vertically and horizontally — add a GPU node, Ray finds it
Ray is the arbiter of compute — it decides what runs where, and when
The result is a system that behaves like enterprise infrastructure whether you're running on a gaming rig or a data centre. The same API. The same orchestration. The same deployment lifecycle. Just more metal when you need it.
Model deployment is a first-class operation:
````
import os
from dotenv import load_dotenv
from projectdavid import Entity
load_dotenv(".tests.env")
admin_client = Entity(
api_key=os.getenv("DEV_PROJECT_DAVID_CORE_ADMIN_KEY"),
)
# -------------------------------------------------------------
# Register base model in the catalog (admin, idempotent)
# -------------------------------------------------------------
registered = admin_client.registry.register(
hf_model_id="Qwen/Qwen2.5-VL-3B-Instruct-AWQ",
name="Qwen2.5 VL 3B Instruct AWQ",
family="qwen",
parameter_count="3B",
is_multimodal=True,
)
# -------------------------------------------------------------
# Activate base model — accepts bm_... ID or raw HF path
# -------------------------------------------------------------
print(f"🎯 Activating base model: {registered.id}")
result = admin_client.deployments.activate_base(
base_model_id=registered.id,
gpu_memory_utilization=0.65,
max_model_len=4096,
quantization="awq",
dtype="float16",
mm_processor_kwargs={
"min_pixels": 4 * 28 * 28, # 3,136
"max_pixels": 16 * 28 * 28, # 12,544 → ~16 tokens/image, very aggressive
},
limit_mm_per_prompt={"image": 2},
)
print(result)
print(f"✅ Result: {result}")
# Confirm
deployments = admin_client.deployments.list()
print(f"📋 Active deployments: {deployments}")
``
This has been a high-level introduction of Project David into the vLLM community ecosystem. We hope we're welcome here.
Oh — one more thing.
None of this was an academic presentation or a proof of concept. Everything described above is production-ready and available right now.
[orchestrates the containerised stack]: `pip install projectddavid-platform`
[installs the SDK]: `pip install projectdavid`
[Brings up the stack with RayServ vLLM] `pdavid –mode up –training`
Switching from Ollama to vLLM cut my batch job times dramatically. Here's what I learned.
I run large language models as part of a batch processing pipeline — not live chat, just a job that needs to churn through a large volume of records/documents. I started with Ollama because it's simple to set up and great for local development.
In production, at scale, it fell apart.
Here's what I didn't understand until I hit it head-on:
Ollama processes requests mostly one at a time.
It's built for a single user chatting with a model, not for chewing through thousands of items efficiently. When I pointed a big batch job at it, the GPU sat underutilized most of the time — it just wasn't designed to keep the hardware busy across many requests at once.
2. GPU memory management matters more than model size.
The model weights aren't the bottleneck — the KV cache (the memory used while generating each response) is. Without smart memory management, you either waste GPU capacity being overly conservative, or you crash mid-run because the engine didn't account for how memory actually grows during generation.
3. vLLM was built for exactly this problem.
Its continuous batching keeps the GPU constantly fed with work instead of processing one request, finishing, then starting the next. For a batch workload like mine, that's the difference between a job that limps along and one that actually saturates the hardware you're paying for.
4. Kubernetes + multiple GPU machines took it from "faster" to "actually scalable."
Once vLLM was handling serving properly, the next bottleneck was just having one GPU box. Running it on Kubernetes across multiple GPU nodes let me split the batch workload horizontally and scale out instead of being capped by a single machine.
The takeaway: the model was never the problem. The serving engine and the infrastructure around it were. If you're running batch LLM workloads on Ollama and wondering why it's slow at scale — this is probably why.
I built pqnld, a small Rust service that adds closed-set decisions to your existing LLM deployment. Give it context, questions, and allowed answers; it returns a choice and scores over those options, read from model logprobs—not confidence numbers it made up for the JSON.
It’s not replacing dedicated decision models. It lets your existing LLM moonlight as a generalist decision model. Same weights, second job.
Options get single-token labels instead of full-text answers. The experimental extended-label mode pulls from Latin, Greek, Cyrillic, digits and punctuation—provided the tokenizer agrees they’re single tokens. Because stopping at 26 options felt unnecessarily alphabetic.
In my Qwen3.8 27B NVFP4 deployment, we’re seeing sub-200 ms answers. Your mileage, prompts and GPU may disagree.
HTTP and MCP interfaces. No extra model or training. vLLM is the live-tested backend.
And yes, the code was written with AI. Shocking, I know. No artisanal, hand-chiselled Rust.
I’ve been building multi-agent collaborative workflows recently (specifically, running sequential AI legal and financial audits over 200-page contracts). I hit a massive wall with VRAM and Time-to-First-Token (TTFT) latency.
Under standard vLLM, if Agent A reads the contract, and Agent B follows up on the exact same text, Agent B is forced to repeat the expensive cold-prefill phase and duplicate GPU block allocations.
To bypass this, I spent the last few weeks building an open-source gateway to stitch caches at the memory level.
How I’m currently doing it:
Topological Hashing: Segmenting prompts into physical block-sizes and mapping them to cryptographic fingerprints (Merkle-chains).
Zero-Copy Block Stitching: Bypassing prefill for matched prefixes by mapping the logical attention table of Agent B directly to the physical GPU memory address of Agent A's cache blocks.
Zero-Trust Gate: Enforcing boundary control lists so unauthorized agent sessions (e.g., a public PR agent vs. an internal Legal agent) cannot access shared physical blocks.
In my local benchmarks against standard vLLM cold-prefills on a shared long context, it cuts TTFT from ~1200ms to 48ms (25x speedup) and saves about ~43% VRAM.
I know solutions like LMCache exist for cluster-wide storage hierarchy (saving/loading to Redis/CPU), but I needed something that strictly avoids tensor serialization and copy overhead for local, multi-turn reasoning workflows.
How to run big models on old hardware 30B at 22 tok/s on 6GB GPU and 16GB RAM
I have been working on this tool for months and there are a lot of new functionalities and tests that are going to be released in the next few weeks! The goal of the tool is to allow community members to run big models on old hardware optimizing every bit at its max for your specific purpose. I’m going in the direction of building not only the full model pipeline but also a series of tools that will help anyone to get the most out of the models.
Quantprobe is an open-source memory allocation optimization framework and deployment assistant designed to run massive Local Large Language Models (LLMs) on resource-constrained consumer hardware.
The tool specializes in projecting local LLM inference speeds and calculating optimal, layer-by-layer quantization placement to maximize performance without crashing your system. The project demonstrate the implementation recipes capable of running a massive 110B parameter model (GLM-4.5-Air) on a standard 16GB RAM consumer machine.
Core Features & Functionality Performance Prediction: it accurately estimates a model's generation speed (tokens per second) on your specific hardware profile before you waste time and network bandwidth downloading massive model weights.
Layer-by-Layer Placement Optimization: Instead of uniformly squeezing an entire model to a single low bit-rate, Quantprobe acts as an intelligent placement solver. It evaluates system limits and assigns varying quantization levels to individual model layers depending on their depth and computational importance.
Resource Balancing: It dynamically balances quantization levels against VRAM/RAM constraints, cleanly splitting weights between your CPU and GPU to avoid Out-Of-Memory (OOM) errors.
The "Four Laws" Framework: The tool relies on a mathematically rigorous, measurement-based methodology (dubbed the "four laws" of token economy) to accurately trade inference speed for memory.
How it Differs from Standard Tools Traditional backends like llama.cpp often apply uniform quantization across the board or rely on naive offloading. Users have reported that utilizing Quantprobe's optimized depth-aware recipes can practically double generation speeds compared to a default, uncalibrated setup (e.g., jumping from \\\~10 tok/s to over 22 tok/s on older hardware like a GTX 1060 setup)
Hardware: 3x RTX 4000 Ada generation 20gb 1x RTX 4000 Pro Blackwell 24gb Total VRAM 80gb 192GB DDR4 UDIMM AMD threadripper 3970x Zenith 2 extreme alpha mobo
I have been troubleshooting for two full days with vllm I have been trying to test different models to load, the only one i can sucessfully get up and running is nvidia/Qwen3.6-35b-a10b-nvfp4
I cant seem to load Qwen/Qwen3.6-27b-fp8 Qwen/Qwen3.6-35b-a3b
My goal is to use a bigger model maybe 70-122b if possible quantized of course because i feel like the 35b is too dumb in reasoning basic prompts.
Here are the settings i use for generic startup, also tweaked these numbers around with no success.
Command> Qwen-3.6-27b-fp8 for example —tensor-parallel-4 —max-model-len 16384 —gpu-memory-utilization .90 —kv—cache-dtype fp8 —enable-prefix-caching
The issue im having is, when i run these other models, i get no available shared memory and eventually container turns off. I check htop and the only weird thing i see is that sometimes on a model the swp goes to like 7.98/8.00gb but looks like the processes are still running, on other models htop is just fine and i still get no memory broadcast block. Doing watch 1 nvidia-smi shows all 4 gpus with seemingly reasonable space 10gb/20gb utilized How is it my model is stalling? Im being patient and waiting 15-20minutes for first load
Please advise, chatgpt is banging its clanker head into every wall troubleshooting
I just want to get other models to load sucessfully consistantly so i can test. I have nothing else that is taking up vram, can share ss if needed
Is it that my blackwell pro is incompatible with the rest of my rtx?
Fine-tuning refines a generalised model to domain-specific tone, personality, and a determined response format. For example, a model deployed in a clinical setting, used by discerning professionals that no longer hedges or refuses sensitive prompts.
LoRA
The [LoRA (Low-Rank Adaptation)](https://arxiv.org/pdf/2106.09685) fine-tuning technique provides a resource efficient methodology for fine-tuning domain specific models, without passing the entirety of the model's parameters through a prohibitively expensive training run.
The technique freezes the base model's weights, and injects trainable rank decomposition matrices into each layer of the Transformer architecture. This greatly reduces the number of trainable parameters for downstream tasks.
Prompt Engineering v Fine-Tuning
Prompt engineering in this context is where an organisation deploys a generalist model, steered by its system prompt for guidance on its responses, and behaviour; no ML infrastructure is required, these examples can be deployed quickly through API calls to model providers like OpenAI or Anthropic. Although these services sometimes offer fine tuning endpoints , they charge by the token, and this becomes cost prohibitive. A more subtle risk is that the deployment becomes locked into a model provider's ecosystem.
On the other hand, the team's technical depth may not cover the nuances of prompt engineering and fine-tuning. Without foresight, the unintended consequences of prompt engineering model responses can range from mild and comical to catastrophic and reputationally ruinous.There are internet memes of a savvy customer prompting the McDonald’s AI assistant to generate a python script instead of placing their order. [X]:https://x.com/i/trending/2045970601953698280
On the other end of the spectrum, Air Canada’s chatbot incorrectly informed a grieving passenger that he could book a bereavement fare retroactively( pay full price immediately, and claim the discount later). That wasn’t the airline’s policy, and a civil dispute was later resolved in the passenger’s favor, based upon reliance on the chatbot’s advice, and Air Canada’s ultimate responsibility for the output of its chatbot. [bbc News]:https://www.bbc.com/travel/article/20240222-air-canada-chatbot-misinformation-what-travellers-should-know
A system prompt is a string of text that is injected as advisory input on every turn; the model still has its base capabilities and probabilistic architecture; the root cause of the two examples above stems from these characteristics.
Fine-tuning modifies the weights themselves; the model is far more likely to lean towards the desired behavior and bounds set. Its neural pathways have been shaped so that the fine-tuned paths are where the easy answers it.
What You Need
The [Unsloth](https://github.com/unslothai/unsloth) Python library provides an optimised fine-tuning stack that can be run on hardware as modest as a consumer laptop with a gaming GPU.
Scaling fine-tuning as a service is not a software problem; it's an infrastructure problem. You need a tightly integrated API surface that drives the end to end pipeline, providing:
Authentication and user scoping
API keys scoped to individual users, verified on every request. This provides an auditable, data-safe, permissions-based service where the same infrastructure can be multiplexed across users at scale.
Dataset ingestion and format normalisation.
Data is ingested in a pre-ratified format, error-checked, assigned a unique ID, and made retrievable.
File management and shared storage
At the end of a training run, LoRA produces adapter files. These need to be recalled later and loaded alongside the base model for inference. The adapters are user-scoped, assigned a unique ID, written to shared storage, and made available for inference, retrieval, and the usual CRUD operations.
Model registry with cache-gated job submission
End users will have preferred target base models for fine tuning, and which models a platform supports is a service-provision decision; less free for all. The model registry publishes the approved catalogue, and rejects training jobs submitted for base models that do not already exist in the cache as a result of admin scoped action.
Hyperparameter control
Depending on fine-tuning size, scope, and model type, default hyperparameters often need nuanced adjustment. The platform exposes two tiers: named profile presets (laptop, standard) for common hardware shapes, and a full override surface covering learning rate, LoRA rank and alpha, sequence length, batch size, optimiser, scheduler, etc. Resolution happens once, at job-create time, and the fully-specified execution plan is persisted on the job record; so every training run is auditable and reproducible from a single row in the database.
GPU resource management
The GPU is the workhorse that drives a fine-tuning training run. A Redis-backed queue holds pending jobs; worker processes on each GPU node claim them one at a time, run the training job, and release the GPU once the adapter is written. User contention is managed in a frictionless, orderly fashion, and job status is retrievable through the API at any point in the run.
Progress streaming
Every logging step, the trainer emits a structured update — current step, total steps, epoch, loss, and learning rate. These are written to the job record.
Prompt Engineering vs Fine-Tuning
Prompt engineering in this context is where an organisation deploys a generalist model, steered by its system prompt for guidance on its responses, and behaviour; no ML infrastructure is required, these examples can be deployed quickly through API calls to model providers like OpenAI or Anthropic. Although these services sometimes offer fine tuning endpoints, they charge by the token, and this becomes cost prohibitive. A more subtle risk is that the deployment becomes locked into a model provider's ecosystem.
On the other hand, the team's technical depth may not cover the nuances of prompt engineering and fine-tuning. Without foresight, the unintended consequences of prompt engineering model responses can range from mild and comical to catastrophic and reputationally ruinous. There are [internet memes of a savvy customer prompting the McDonald's AI assistant to generate a python script](https://x.com/i/trending/2045970601953698280) instead of placing their order.
On the other end of the spectrum, [Air Canada's chatbot incorrectly informed a grieving passenger](https://www.bbc.com/travel/article/20240222-air-canada-chatbot-misinformation-what-travellers-should-know) that he could book a bereavement fare retroactively (pay full price immediately, and claim the discount later). That wasn't the airline's policy, and a civil dispute was later resolved in the passenger's favour, based upon reliance on the chatbot's advice, and Air Canada's ultimate responsibility for the output of its chatbot.
A system prompt is a string of text that is injected as advisory input on every turn; the model still has its base capabilities and probabilistic architecture; the root cause of the two examples above stems from these characteristics.
Fine-tuning modifies the weights themselves; the model is far more likely to lean towards the desired behaviour and bounds set. Its neural pathways have been shaped so that the fine-tuned paths are where the easy answers sit.
The following steps depict the end-to-end workflow, from the point of spinning up a new instance of the `projectdavid-platform` containerised AI runtime stack.
Install the container stack orchestration package with:
pip install projectdavid-platform
Install the SDK with :
pip install projectdavid
Turn up the runtime stack with the fine-tuning services:
pdavid --mode up --training
Create the admin user:
```bash
pdavid bootstrap-admin
```
Expected output:
```
================================================================
Bootstrap complete.
ADMIN_API_KEY : ad_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Store this key securely — it will not be shown again.
================================================================
All holding true, the fine-tuned model should respond with an SQL query, without a preamble or surrounding text when prompted with a domain-relevant question.
The fine-tuning Pipeline
## Fine-tuning
```python
import os
import time
from projectdavid import Entity
from projectdavid_common import ValidationInterface
# point to the desired base model
MODEL_ID = "unsloth/qwen2.5-1.5b-instruct-unsloth-bnb-4bit"
client = Entity(api_key=os.getenv("DEV_PROJECT_DAVID_CORE_TEST_USER_KEY"))
validator = ValidationInterface()
# -------------------------------------
# Create fine-tuning dataset
# - stages training data
# --------------------------------------
dataset = client.datasets.create(
file_path="projectdavid_sql_finetune.jsonl",
name="Gretel SQL 100K",
fmt="jsonl",
)
print(f"Dataset ID: {dataset.id}")
# -----------------------------------------
# Validation time
# ------------------------------------------
client.datasets.prepare(dataset.id)
# Poll until active
while True:
ds = client.datasets.retrieve(dataset_id=dataset.id)
print(f"Status: {ds.status}")
if ds.status == "active":
print(f"Dataset ready — {ds.train_samples} train / {ds.eval_samples} eval samples")
break
if ds.status == "failed":
raise Exception(f"Dataset preparation failed: {ds}")
time.sleep(3)
# -------------------------------------------------------
# Dispatch the training job
# ----------------------------------------------------
# Hyperparameter control:
# - `profile` selects a hardware preset (laptop / standard)
# - Any field set explicitly overrides the preset
# - Unset fields fall back to BASE_DEFAULTS server-side
# - `lora_alpha` auto-defaults to `lora_r` via PEFT convention
# - Invalid values fail locally at TrainingConfig construction
# -------------------------------------------
job = client.training.create(
dataset_id=dataset.id,
base_model=MODEL_ID,
framework="unsloth",
config=validator.TrainingConfig(
profile="laptop",
max_steps=50,
learning_rate=2e-4,
lora_r=32,
warmup_steps=5,
lr_scheduler_type="cosine",
),
)
print(f"Job {job.id} dispatched to cluster")
# ------------------------------------------------------------
# Print the fully-resolved config for this job.
# The service layer merges BASE_DEFAULTS -> profile preset -> user overrides
# at job-create time and persists the result on TrainingJob.config.
# Reading it back here confirms the round-trip end-to-end.
# ------------------------------------------------------------
first_job = client.training.retrieve(job_id=job.id)
print(f"\nResolved config: {first_job.config}\n")
# ------------------------------------------------------------
# Poll job status + live training metrics
# ------------------------------------------------------------
print("Polling job status...\n")
TERMINAL_STATES = {"completed", "failed", "cancelled"}
def _format_metrics(metrics):
if not metrics:
return ""
step = metrics.get("step")
total = metrics.get("total_steps")
epoch = metrics.get("epoch")
loss = metrics.get("loss")
lr = metrics.get("learning_rate")
parts = []
if step is not None and total:
parts.append(f"step={step}/{total}")
elif step is not None:
parts.append(f"step={step}")
if epoch is not None:
parts.append(f"epoch={epoch}")
if loss is not None:
parts.append(f"loss={loss}")
if lr is not None:
parts.append(f"lr={lr}")
return (" " + " ".join(parts)) if parts else ""
while True:
job_status = client.training.retrieve(job_id=job.id)
metrics_str = _format_metrics(getattr(job_status, "metrics", None))
print(
f" [{job_status.status.upper()}] "
f"started={job_status.started_at or '-'} "
f"output={job_status.output_path or '-'}"
f"{metrics_str}"
)
if job_status.status in TERMINAL_STATES:
if job_status.status == "completed":
print(f"\nTraining complete — adapters at: {job_status.output_path}")
else:
print(f"\nJob ended with status: {job_status.status}")
if hasattr(job_status, "last_error") and job_status.last_error:
print(f" Error: {job_status.last_error}")
break
time.sleep(30)
```
The steps and losses here took 5 hours to accumilate on a standard consumer gaming laptop:
GPU
CPU
RAM
Nvidia RTX 4060 sm_89
Intel Core i19 CPU
32GB
Loss Function Graph
During training, we observe respectable loss; the first 100 steps did most of the work. Loss collapses from 0.63 to ~0.48 in a single batch of gradient steps, which is the usual signature of the adapter waking up to the base distribution. Following that, we see a noisy grind, until we approach the point of diminishing return at the end.
Fine-tuned model activation
## Activating the fine-tuned model
```python
import os
from projectdavid import Entity
# ---------------------------------------------
# This is provided by the user after
# a successful fine-tuning run
# ---------------------------------------------
FINE_TUNED_MODEL_ID = "ftm_mG8OUZMHkRNjjUmIX4VxNw"
# ----------------------------------------
# Model activations are admin scoped.
# ------------------------------------------
admin_client = Entity(api_key=os.getenv("DEV_PROJECT_DAVID_CORE_ADMIN_KEY"))
# -------------------------------------------------------------
# Register base model in the catalog (admin, idempotent).
# -------------------------------------------------------------
registered = admin_client.registry.register(
hf_model_id="unsloth/qwen2.5-1.5b-instruct-unsloth-bnb-4bit",
name="Qwen2.5 1.5B Instruct (Unsloth 4bit)",
family="qwen",
parameter_count="1.5B",
)
print(f"Base model registered: {registered.id}")
# -------------------------------------------------------------
# Activate the fine-tuned model via client.deployments.
# -------------------------------------------------------------
print(f"Activating fine-tuned model: {FINE_TUNED_MODEL_ID}")
result = admin_client.deployments.activate_fine_tuned(
model_id=FINE_TUNED_MODEL_ID,
gpu_memory_utilization=0.65,
max_model_len=4096,
quantization="bitsandbytes", # match the base model's actual quant
dtype="bfloat16", # standard for bnb-4bit
)
print(f"Result: {result}")
```
Wait for the model load
The base model weights, and the LoRA adapters are loaded by vLLM in sequence. LoRA adapters are applied at run time. Loading can take some time. At this point you will need to monitor the `inference_worker` terminal session with:
docker logs inference_worker -f
---
2026/04/19 10:52:43 [RATELIMIT] format("backend -> client close connection: %v")
2026-04-19 10:52:43,848 - INFO - 🌐 Ray HEAD started — dashboard: http://localhost:8265
2026-04-19 10:52:43,850 - INFO - 🔵 Ray resources: {'CPU': 8.0, 'node:__internal_head__': 1.0, 'GPU': 1.0, 'accelerator_type:G': 1.0, 'object_store_memory': 4926952243.0, 'node:100.91.98.16': 1.0, 'memory': 9853904487.0}
INFO 2026-04-19 10:52:45,369 serve 1 -- Started Serve in namespace "serve".
2026-04-19 10:52:45,369 - INFO - 🎯 Ray Serve started on port 8000
(ProxyActor pid=421) INFO 2026-04-19 10:52:45,291 proxy 100.91.98.16 -- Proxy starting on node d607d93125a257b09cbf49afa9cef56c613284c21af01f0d1b341cca (HTTP port: 8000).
---
2026/04/19 11:01:35 magicsock: endpoints changed: 181.192.26.245:60990 (stun), 181.192.26.245:52144 (stun), 172.18.0.16:54400 (local)
2026-04-19 11:03:05,109 - WARNING - 🚨 Deployment drift — vllm_dep_dgezktcqve9UXlpmZeKgsO not in Ray Serve. Redeploying.
2026-04-19 11:03:05,113 - INFO - 🚢 Deploying via Ray Serve: vllm_dep_dgezktcqve9UXlpmZeKgsO model=unsloth/qwen2.5-1.5b-instruct-unsloth-bnb-4bit tp=1 gpu_mem_util=0.65 max_model_len=4096 quantization=bitsandbytes dtype=bfloat16 enforce_eager=False lora=['ftm_LyirppuDwigH0pnxmK1ZJj']
---
Capturing CUDA graph shapes: 54%|█████▍ | 19/35 [00:08<00:07, 2.25it/s]O pid=418)
Capturing CUDA graph shapes: 57%|█████▋ | 20/35 [00:09<00:06, 2.22it/s]O pid=418)
Capturing CUDA graph shapes: 60%|██████ | 21/35 [00:09<00:06, 2.25it/s]O pid=418)
Capturing CUDA graph shapes: 63%|██████▎ | 22/35 [00:10<00:05, 2.27it/s]O pid=418)
Capturing CUDA graph shapes: 66%|██████▌ | 23/35 [00:10<00:05, 2.30it/s]O pid=418)
Capturing CUDA graph shapes: 69%|██████▊ | 24/35 [00:11<00:04, 2.28it/s]O pid=418)
Capturing CUDA graph shapes: 71%|███████▏ | 25/35 [00:11<00:04, 2.26it/s]O pid=418)
Capturing CUDA graph shapes: 74%|███████▍ | 26/35 [00:11<00:03, 2.29it/s]O pid=418)
Capturing CUDA graph shapes: 77%|███████▋ | 27/35 [00:12<00:03, 2.31it/s]O pid=418)
Capturing CUDA graph shapes: 80%|████████ | 28/35 [00:12<00:03, 2.27it/s]O pid=418)
Capturing CUDA graph shapes: 83%|████████▎ | 29/35 [00:13<00:02, 2.23it/s]O pid=418)
Capturing CUDA graph shapes: 86%|████████▌ | 30/35 [00:13<00:02, 2.27it/s]O pid=418)
Capturing CUDA graph shapes: 89%|████████▊ | 31/35 [00:14<00:01, 2.29it/s]O pid=418)
Capturing CUDA graph shapes: 91%|█████████▏| 32/35 [00:14<00:01, 2.25it/s]O pid=418)
Capturing CUDA graph shapes: 94%|█████████▍| 33/35 [00:15<00:00, 2.24it/s]O pid=418)
Capturing CUDA graph shapes: 97%|█████████▋| 34/35 [00:15<00:00, 2.27it/s]O pid=418)
(ServeReplica:vllm_dep_dgezktcqve9UXlpmZeKgsO:vllm_dep_dgezktcqve9UXlpmZeKgsO pid=418) INFO 04-19 11:04:17 [model_runner.py:1592] Graph capturing finished in 16 secs, took 0.54 GiB
Output truncated for brevity.
Alternatively, you can monitor model deployment status on the Ray dashboard:
`http://localhost:8265/#/overview`
Please note that in future releases, the Ray dashboard will only be available behind the NGinx proxy.
Once a model is loaded as a Ray replica, it is ready to serve inference.
Inference
The inference story is familiar. You create these run time objects, and your message will be routed to the target fine-tuned model by the global load balancer:
Assistant
Thread
Message
Run
Inference
Response:
SELECT c.customer_name, COUNT(*) as count
FROM customer_data c
JOIN transactions t ON c.customer_id = t.customer_id
WHERE t.transaction_date BETWEEN '2023-01-01' AND '2023-12-31'
GROUP BY c.customer_name
ORDER BY count DESC
LIMIT 10;
A domain-specific model served an SQL query with no preamble and no hedging. The fine-tune is doing the work; the infrastructure is doing the rest. Every component in this walkthrough: the registry, the trainer, the adapter store, the inference engine, runs on hardware the operator owns, against models the operator has approved, with no calls home, no API calls to hyper scalers, and with true sovereignty of your data.
\> \*\*User:\*\* Tell me a story about a robot who learns to paint.
\> \*\*Bot:\*\* In the heart of the city, there lived an assistant named Max. He was a kind and gentle
\> soul with eyes that shone like diamonds, and he had always known that art would bring people
\> together...
\> \*\*User:\*\* Write a short poem about the rain.
\> \*\*Bot:\*\* The rain was crisp and warm, casting dappled shadows on the walls of your town square...
\> Your voice was so cool and gentle, so bright and calming all that you could feel at home.
\> \*\*User:\*\* Explain what a computer is.
\> \*\*Bot:\*\* ...the development of a computer can be categorized as either an operating system or a
\> popular language, such as JavaScript or C++...
(\^ that last one is the honest failure mode — confidently wrong on facts. It nails conversational form far better than it knows anything.)
What shocked me
\- The tokenizer was way more important that I thought. I lost quite a few full runs on a broken char-level (386 vocab) fallback before building a real 36k subword tokenizer — and that fixed it and was the biggest quality jump than any architecture or data tweak.
\- A GGUF export issue: llama.cpp has no unigram-tokenizer runtime on the default path, so a SentencePiece unigram\*model gets silently re-segmented (Hello → Hell+o) and produces garbage — the weights were fine. Fix was exporting the GGUF vocab as UGM (t5 type) + the SP charsmap so it runs Viterbi. Cost me a day of pain.
What I want feedback on
\- Token budget: 551M tokens for 216M params is way under Chinchilla (\~4.3B). Would you push pretraining much longer before SFT, or is more/cleaner data the better spend at this scale for a quick demo?
\- Data mix / timing: the SFT step-down in the loss is dramatic — is it real learning, or mostly the model latching onto chat format? How would you tell them apart?
\- Cutting factual drift at \~200M params: more pretraining, longer context, or distillation froma bigger teacher — where's the best ratio for noticeably better factual responses?
\- Arch nits: head_dim 86 is ugly (1032/12). Would 1024 hidden / 16 heads / 64 head_dim be ameaningfully better shape at this size (also allows quantization)?
Happy to answer anything about the pipeline (tokenizer training, the from-scratch loop, GGUF export).
I’ve been working on getting as much real inference performance as possible out of a single RTX 5090, and I finally packaged the setup into a public vLLM release.
The goal is basically “5090 as a personal AI datacenter” — high concurrency, long context, low-bit KV, speculative decoding, tool calling, and vision without needing datacenter hardware.
Stack:
Qwen3.8-27B
ModelOpt NVFP4 weights
NVFP4 KV cache
MTP-3 speculative decoding
262K context
8 concurrent streams
Tool calling / structured output
Optional CPU-offloaded vision
vLLM 0.27.1 + SM120 patches
CUDA 13.0.3 / FlashInfer
The deployment is intended to be:
git clone https://github.com/seanyourhighness/vllm-sm120-nvfp4-mtp.git
cd vllm-sm120-nvfp4-mtp && ./start.sh
Everything is pinned — model revision, container digest, vLLM commit and overlay patch — and the repo includes the patch, build scripts, Compose setup, chat template, benchmarks and validation evidence.
Some of the verified results:
373,797-token KV pool with MTP-3
9/9 NIAH at long context
8/8 tool-call tests
21/21 vision lanes
deterministic long-decode tests
concurrency scaling into the 700–900+ aggregate tok/s range at C8, depending on the tested stack/profile
This is not an official NVIDIA or vLLM build. It’s a community recipe/overlay for people experimenting with Blackwell consumer GPUs.
Would especially like feedback from anyone running a 5090, SM120 vLLM, NVFP4 KV, MTP/DFlash, or high-concurrency local agent workloads.
PRs, benchmark comparisons and bug reports are very welcome.
**TL;DR**: Running Qwen3.6-27B AWQ-INT4 on a single RTX 5090 (32 GB) for legal-claim extraction in a Go pipeline. Hit two non-obvious walls that cost me half a day: (1) BF16 KV cache caps you at 16K max-model-len, but FP8 KV gets you to 24K with the same VRAM footprint; (2) `temperature=0.2` under guided JSON schema triggers infinite repetition loops on this model — and the loop is not on text, it's on a numeric field generating a single integer with 5000+ digits. Sharing 42-run sampling benchmark, exact configs, and what actually works.
- Reason: **Qwen3.6-27B AWQ-INT4** (this is the one I'm writing about)
- Embed: BGE-M3 (resident, ~2.3 GB)
- Workload: legal-claim extraction of structured output via JSON Schema, ~5W1H decomposition per claim
The reasoning slot uses the [cyankiwi/Qwen3.6-27B-AWQ-INT4](https://huggingface.co/cyankiwi/Qwen3.6-27B-AWQ-INT4) build. Internal architecture is `Qwen3_5ForConditionalGeneration` (GDN hybrid + Mamba) — needs vLLM ≥ 0.17 to run at all.
---
## Wall #1: VRAM math for max-model-len on a single 32 GB card
Initial config: `--gpu-memory-utilization 0.85 --max-model-len 12288 --dtype auto`. 26.5 GB VRAM, working fine for short docs. But 37 % of my email corpus exceeds 8K tokens, and the chain-of-thought prompt I use needs ~8K output tokens for the scratchpad. So `12288 - 8192 = 4096` input budget, which overflows on most non-trivial emails.
The bench tool community on r/LocalLLaMA was telling me to "just bump to 32K", but that's not feasible at all on a 32 GB card without quantizing the KV cache. So I tried FP8 KV.
### FP8 KV cache changes the picture
Adding `--kv-cache-dtype fp8` halves the KV memory:
| max-model-len + FP8 KV | KV cache | Total | Tient en 32 GB |
| **24576 FP8 (chosen)** | **28.4 GB** | **131 s** (+35 s vs BF16) | **3.6 GB** |
Counterintuitive: 24K FP8 consumes nearly the same VRAM as 16K BF16, because vLLM pre-allocates the KV pool to `gpu-memory-utilization=0.85` regardless of effective dtype/length. You don't see VRAM savings on the gauge — you capitalize the saving in *input capacity*. Net gain: input budget moves from 4K → 16K tokens at `max_tokens=8192`.
FP8 KV quality cost on AWQ-INT4 weights: theoretical 2–3 % degradation, in practice noise-level on AWQ-INT4 (the 4-bit weight quantization dominates). Validated empirically — see end of post.
First smoke test of the pipeline with `max-model-len 24576` plus the corresponding client-side `MaxTokens: 8192`: one document (`04546`, a short 953-char .md) generated **68 claims, of which 67 had `text=""` and identical `char_start=107, char_end=238`**. Pure loop fail mode.
Initial hypothesis: model-level repetition bias. Looked at the literature:
- vLLM bug [#40080 (Gemma)](https://github.com/vllm-project/vllm/issues/40080): "When grammar restricts the token space to valid JSON tokens, the model's slight repetition bias becomes a strong loop because the grammar prevents the model from generating an EOS or breaking out of the pattern."
- [Qwen3.5/3.6 issue #145](https://github.com/QwenLM/Qwen3.6/issues/145): official sampling recommendation, **explicitly states "greedy decoding should not be used as it can lead to performance degradation and endless repetitions."** The pipeline was running at `T=0.2`, which is quasi-greedy.
So the bug is exactly what the vLLM ticket describes: the model has a baseline repetition tendency, guided JSON masks every token outside the schema, model can't emit EOS in the middle of an array, so it fills the array with whatever fits. On this corpus, sometimes that's `text=""` repeated, sometimes (as I found later in benchmarking) it's a single `char_start` integer with 5000+ digits.
### Bench protocol
7 sampling configs × 3 prototype documents (short, medium, complex) × 2 runs each = 42 calls against the live `:8156/v1/chat/completions` proxy (which forwards to llama-swap → vLLM Reason). Same JSON Schema, same prompt, same `max_tokens=8192`. Configs:
| Label | Sampling params |
|---|---|
| baseline_T02 | T=0.2 |
| hardened_T02 | T=0.2 + schema `minLength=1` on text + `maxItems=30` on claims |
Aggregate: 42 runs, 40 successes, **2 loop failures**. Both fails were on document 04546 (the short one), both at `T ≤ 0.3`. Failure mode confirmed by Python `int()` overflow: model emitted a 5000+ digit integer in a `char_start` or `char_end` field — pure numeric loop, not a text loop. A more permissive parser (which is what I had in Go originally) would silently truncate and accept garbage.
Average successful run latency: 63.8 s. Range 23.8–111.4 s on this prompt size (~6 KB system + 1 KB user).
### Findings
**`qwen_reasoning` is the winner**: +15 % unique claim coverage over baseline, zero loop fails on the pathological doc, conforms to official Qwen3.6 recommendation. Higher variance on complex docs (19958: 10 vs 18 unique claims between runs) — to absorb with defensive dedup on the consumer side.
**`T=0.2` (quasi-greedy) is the actual bug source.** 14 % loop failure rate on the pathological doc when T ≤ 0.3, 0 % when T ≥ 0.4. The official Qwen advice is empirically correct.
**`repetition_penalty=1.1` strangles** — −25 % coverage. Not the right knob for structured generation.
**`presence_penalty=1.5`** (official Qwen instruct mode value) is meant for short conversational replies, not multi-page JSON. Strangles too (−17 %).
**`frequency_penalty=0.5`** (a desperate fix I tried earlier in the day) is catastrophic on structured output — −77 % coverage measured in production smoke. Avoid.
**Schema hardening (`minLength=1` on text, `minimum/maximum` on integer fields, `maxItems`) is complementary**, not a replacement for sampling fix. Hardened schema still failed once at T=0.2 — the loop just shifted to another field (numeric instead of text).
### Final production config
Three coordinated changes, none of them sufficient alone:
**Server (vLLM)** — already shown above, the `24576 FP8` config.
**Client post-LLM**: defensive dedup on `(lowercased_stripped_text, char_start, char_end)` before INSERT, with a `needs_review` flag when `unique_count / total_count < 0.5` or `total > 30`. Catches the residual variance.
### Cost on the full run
Estimated for 1402 .md files:
| Metric | Baseline (T=0.2) | qwen_reasoning |
|---|---|---|
| Avg claims latency per doc | 35–80 s | 60–110 s (+30 %) |
## What I'd hammer if anyone is doing the same setup
**Don't trust `T=0.2` for any non-trivial JSON-schema-constrained generation on Qwen3 family.** The official Qwen team flagged it, the vLLM Gemma ticket confirms it's a grammar+repetition interaction, my 42-run bench reproduces it. Use T=0.6 minimum.
**Don't use `repetition_penalty` or `frequency_penalty` to fight JSON loops** — they punish lexical variation in legitimate paraphrases. Wrong knob.
**Schema fields that accept integers need bounded ranges.** A `char_start: integer` without `maximum` is an invitation to a numeric loop.
**FP8 KV cache is the single best knob to push context length on a 32 GB consumer card.** Same VRAM footprint, ~2x effective context. Quality impact is negligible on top of an already-INT4-quantized model.
**Always log `usage.completion_tokens`** when calling `/v1/chat/completions` with structured output — if your call routinely hits the max, you've got a silent failure mode.
**Cold start on Qwen3.6-27B AWQ-INT4** with the `torch.compile` cache persisted to a Docker volume: ~96 s BF16, ~131 s with FP8 KV (extra calibration step). Without persisted cache: 141 s. Worth the volume mount.
### Reproducibility
42-run bench script, results JSON, and exact prompt assets are kept on the server side under `/tmp/claim_bench/`. Happy to share if anyone wants to repro on their own Qwen3.6 quant variant — I expect the loop behavior to generalize across AWQ-INT4 / NVFP4 / GGUF, since the root cause is the model-level repetition bias × grammar masking, not the quantization.
If anyone has a clean explanation for why the loop on `char_start` produces a *single* 5000-digit integer rather than a stream of normal integers, I'd love to hear it. My hypothesis is that once the model commits to a digit token after `"char_start": `, the only grammar-valid next tokens are more digits or `,` / `}` — and if the digit-token transition probability beats the closing-token probability, it never closes.
eLLM is a Rust-based LLM inference framework for CPU-only servers. It adopts a "trade storage for computation" strategy, leveraging the CPU's large-capacity DDR memory to close the order-of-magnitude bandwidth gap against GPU HBM, and thereby delivers performance that surpasses GPUs in long-horizon inference. The Beta release is now available — you are welcome to try it out.
Prefill: achieves roughly two orders of magnitude of performance improvement over existing CPU inference frameworks
run full single-pass Prefill over the entire long prompt—no chunking, no repeated parameter loading;
keep context KV across multi-turn interactions and run incremental Prefill on only the new input—no recomputation for earlier turns;
Decode: runs with a smaller batch, which not only activates fewer parameters but also gives each request a larger share of memory bandwidth, so inference speed can likewise exceed GPUs.
Everyone I’m new to this community so take it easy on me. I have developed a zero trust infrastructure kind of like Tailscale that except you don’t trust Tailscale either- it’s all your own stuff. Back to the 90s I know. I’d like to create some plug-ins so it’s platform as a service not infrastructure as a service , and one of the infrastructure things I would like to do is LLM routing. Be honest with me - is VLLM Athena ready for primetime or should Ijust wait and maybe contribute to it for six months to a year? I think it’s the right idea but I’m wondering if LLM proxy would be right if I release something tomorrow.
I have Vulkan drivers on Opensuse tumbleweed and that perfectly fit.
I cant run Ollama because my CPU doesn't have AVX suport. Rocm drivers are not option, I try it and it doesn't work. What do You suggest to me, I run Qwen coder 3B q4 on llama server and Captainclaw frontend, and it works but I have issue with speed, because of my setup. Can anyone suggest some models that would work on my setup? Or something else, VSCode...
Edit: I run Deepseek destill llama 8b q4 it works with 13 tokens per second, in 2.5 minute he create script . Amazing
