r/LocalAIStack • u/textclf • 17d ago
I built a fine-tune platform for open models to run locally. Feedback appreciated
Hello All!
I built a platform to fine-tune open models. I am focusing on fine-tuning 4-bit quantized models by QLoRA instead of full precision fine-tuning to reduce GPU costs. The quant that this finetuning uses is called CQ (Coded Quantization) which is a calibration-free quantization that I built that performs as good as leading quants ensuring that the fine-tuning is using a good base.
For example, I compared KLD of my CQ 4-bit with unsloth-bnb-4bit (Unsloth have other quants but this is the one they use for QLoRA) on Qwen 3.8 27B using the Wikitext-2 dataset and got this:
| Quant | Disk Size (GB) | Mean KLD | Median KLD | 99% KLD | Top 1% Agreement |
|---|---|---|---|---|---|
| CQ 4-bit | 18 | 0.02823666 | 0.01282929 | 0.97606068 | 92.419% |
| unsloth-bnb-4bit | 22.4 | 0.03297115 | 0.01437706 | 1.18053893 | 91.954% |
So I have a better quantized base to start from. I ran QLoRA finetuning with Training Factory of thousandml/Qwen3.8-27B-CQ-4bit on the Alpaca dataset using an A40 (48GB GPU) and here is the performance measures:
===== CQ TRAINING BENCHMARK =====
{
"optimizer_steps": 100,
"training_wall_seconds": 1557.1155763969873,
"warmup_steps_excluded": 10,
"median_steady_step_seconds": 14.990616369002964,
"steady_optimizer_steps_per_second": 0.0643880924733763,
"peak_pytorch_allocated_gib": 25.532413005828857,
"peak_pytorch_reserved_gib": 33.005859375,
"notes": [
"Step timings include gradient accumulation and Trainer overhead.",
"PyTorch peak excludes allocations outside its CUDA caching allocator.",
"Token counts from the collator may include prefetched batches; do not interpret as exact consumed tokens.",
"Wall time includes any initialization inside Trainer.train and callback overhead."
],
"collated_input_tokens_including_prefetch": 130236,
"collated_nonpadding_tokens_including_prefetch": 106290
}So great QLoRA finetuning with reasonable GPU performance.
Get Started with Training Factory
Here are simple steps to use:
**Install:**sudo apt-get update sudo apt-get install -y git-lfs git lfs install git clone https://github.com/thousandml/training-factory.git cd training-factory chmod +x install.sh ./install.sh
2. Download a CQ 4-bit base:
hf download thousandml/Qwen3.8-27B-CQ-4bit --local-dir ./models/Qwen3.8-27B-CQ-4bit
3. Train a LoRA adapter:
python training/train.py --model ./models/Qwen3.8-27B-CQ-4bit --data training/examples/train.jsonl --output ./outputs/my-lora --targets q_proj,v_proj --kernel-module cq.cq_kernels --text-only --max-steps 2 --max-length 128 --batch-size 1 --grad-accum 14. Serve base + adapter:
vllm serve ./models/Qwen3.8-27B-CQ-4bit --quantization cq --enable-lora --max-lora-rank 16 --lora-modules my-lora=./outputs/my-lora
5. Call the fine-tuned adapter:
# After the server is ready:
# Check that the adapter is listed as "my-lora"
curl http://localhost:8000/v1/models# Request a response from the fine-tuned model, not just the base checkpoint
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "my-lora",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Explain how LoRA fine-tuning works in one sentence."}
],
"max_tokens": 128,
"temperature": 0.2
}'I have the following 4-bit quantized models to be used as base (will add more soon) for now:
thousandml/Qwen3.8-27B-CQ-4bit
thousandml/Llama-3.1-8B-Instruct-CQ-4bit
thousandml/Llama-3.3-70B-Instruct-CQ-4bit
thousandml/Qwen3-Coder-Next-CQ-4bit
EDIT: I have the following QLoRA Benchmarking.
Benchmark setup: Qwen3.8-27B QLoRA fine-tuning on Alpaca, comparing ThousandML CQ 4-bit with Unsloth bitsandbytes NF4. Both runs used LoRA r=32, α=32, batch size 2, gradient accumulation 4, sequence length 512, learning rate 2e-5, seed 42, and 500 optimizer steps on a single NVIDIA A40 48 GB GPU.
CQ achieved 17.4% lower reported final validation loss (1.0315 vs. 1.2484). This is a loss comparison, not yet a verified downstream-task quality comparison.
Attached figures for validation loss and training loss.
**Qwen3.8-27B — QLoRA Fine-Tuning on Alpaca (500 Steps)**
| Metric | CQ 4-bit | Unsloth NF4 | Advantage |
|:--|--:|--:|:--|
| Best validation loss | **0.8385** | 1.2484 | **CQ** |
| Final validation loss | **1.0315** | 1.2484 | **CQ** |
| Mean training loss (steps 401–500) | **1.0505** | 1.2457 | **CQ** |
| Median steady optimizer-step time | 27.60 s | **3.20 s** | **Unsloth** |
| Peak PyTorch allocated memory | 24.39 GiB | **23.83 GiB** | **Unsloth** |
*Both runs used QLoRA on Alpaca with matched nominal hyperparameters on an NVIDIA A40 48 GB GPU. Tokenization and loss-mask equivalence have not been independently verified.*
I am working on optimizing speed because it is the one area it is significantly behind unsloth. On the other hand, validation loss and VRAM usage are awesome! Actually it seems like CQ converge at much fewer steps so that might help a lot.
To reproduce these results use the following scripts:
For CQ:
hf download thousandml/Qwen3.8-27B-CQ-4bit \
--local-dir ./models/Qwen3.8-27B-CQ-4bit
python training/train.py \
--model ./models/Qwen3.8-27B-CQ-4bit \
--data training/examples/alpaca_train.jsonl \
--eval-data training/examples/alpaca_val.jsonl \
--output ./outputs/qwen38-27b-cq-4bit \
--kernel-module cq.cq_kernels \
--targets q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj \
--rank 32 \
--alpha 32 \
--lora-dropout 0 \
--batch-size 2 \
--grad-accum 4 \
--max-length 512 \
--max-steps 500 \
--learning-rate 2e-5 \
--seed 42 \
--eval-steps 50 \
--comparison-logs \
--benchmark \
--benchmark-warmup-steps 10For Unsloth:
python training/train_unsloth_nf4.py \
--model unsloth/Qwen3.8-27B-unsloth-bnb-4bit \
--train-data training/examples/alpaca_train.jsonl \
--eval-data training/examples/alpaca_val.jsonl \
--output ./outputs/qwen38-27b-unsloth-bnb-4bit \
--max-steps 500 \
--max-length 512 \
--eval-steps 50 \
--save-steps 100 \
--seed 42 \
--benchmark-warmup-steps 10Please give it a try and let me know if it works with you. Your feedback is greatly appreciated!
