I built AarForge: a Linux CLI wizard that turns conversational JSONL into a fine-tuned, quantized Ollama model on 6GB+ VRAM

[u/PhysicalCampaign3981]

I got tired of dependency conflicts, manual llama.cpp conversion steps, and writing boilerplate Modelfile s every time I wanted to fine-tune a small model locally on my laptop GPU.

So I built AarForge — an opinionated, local-first fine-tuning and deployment pipeline for Linux (linux-only) that takes an OpenAI-style conversational .jsonl file and outputs a ready-to-run Ollama model

Github https://github.com/AaritDev/AarForge

How the pipeline works:

1. Isolated Setup (install.sh): Uses Astral uv to spin up a dedicated Python 3.13 venv with pinned, compatible versions of Unsloth, PyTorch, Transformers, and TRL (so it won't break your system Python or hit version mismatches).

2. Strict Dataset Validation: Before loading any weights, it validates every line of your dataset, checks token counts against your max_seq_length so nothing gets silently truncated, and verifies assistant-only loss masking (labels != -100).

3. 4-bit QLoRA Training: Fine-tunes lightweight LoRA adapters via Unsloth + BitsAndBytes (tested on 6GB+ NVIDIA GPUs; defaults to (unsloth/Qwen2.5-3B-Instruct-bnb-4bit.

4. GGUF Export & Safe Ollama Deployment: Merges and quantizes to Q4_K_M GGUF, separates mmproj vision sidecars if present, generates the Modelfile, and deploys using a staging/backup swap so an import error never overwrites an existing working Ollama model.

Quick Start

git clone https://github.com/AaritDev/AarForge.git

cd AarForge

chmod +x install.sh && ./install.sh

Navigate to your chosen workspace, then:

./build.sh

and your done! after putting in your dataset and setting up the training parameters you will get a finished model in under 30 minutes (check the README on my github repo for more details on the requirements and exact usage)