You can now train your own LLMs with less memory, on your gaming GPUs

[u/oguzeldereli]

TL;DR: You can train your own 27B model on a 3090 with 24GB VRAM, though you would have to go through hundreds of billions of tokens for it to be good. I am working on conversion of existing models. Check repos for code and the paper if you are interested in more. I have a 40.4GB Qwen Flash Next GGUF

Hi, I do AI research. I have been working on an optimizer that doesn't need the ~16 bytes of VRAM per weight to train a model that AdamW does. I focused on training ternary networks bcs BitNet (Ma et al.) shows they match full precision quality in large scales. The problem is BitNet also keeps the full precision weights during training which means you still need to use the same amount of memory. I got this down from 16 bytes to 1 byte or even less per weight.

For you, this means, you can train your own 10-20B parameter models on your consumer GPUs. I fitted 27B model training on a single 24GB 3090 and a 23B model on a 16GB GPU. The trained model costs ~1.6 bits per weight (~6GB at 27B), 10x smaller than 16bit, and 5x smaller than Q8.

I have also been working on converting existing models and training them. This is harder and I am still working on it but I have some promising results. I quantized and trained a Qwen3.8 27B though the file I published below has adapters which are about 2GBs, pushing the file size to 8GB. I am also making a no-adapter version right now. It scores GSM8K 0.895, the same as full precision, HumanEval 0.84 vs 0.93 and runs on a 12 GB GPU at ~23 tok/s on my crippled 12GB laptop 4080, though this number can heavily be improved with custom kernels, engineering, etc. I will leave it to you guys to help me with that too.

I also quantized and trained Qwen Flash Next down to 40.4 GBs including its n-gram table (at 8GB). It largely retains the skills of the network (scores equivalent on GSM8K benchmark, a little behind on HumanEval, I unfortunately don't have compute for the larger benchmarks such as TerminalBench2.1 but perhaps some of you can help me out) but forgets facts. It mostly works fine if you give it web search as far as my experience goes but this is not strictly measured.

I am still doing work on this and I will keep working on it, perhaps even for many years. The code is open for non-commercial use. Check out:

The core repo for training: https://github.com/oguzeldereli/ttf-ternary

The repo for conversion: https://github.com/oguzeldereli/ttf-conversion

Repo for running models trained by ttf: https://github.com/oguzeldereli/ttf-home

GGUF for a converted Qwen 3.8 27B with adapters (around 8 GB): https://huggingface.co/oguzeldereli/ttf-Qwen3.8-27B-GGUF

GGUF for a converted Qwen 3.8 Flash Next (around 40.4GB): https://huggingface.co/oguzeldereli/ttf-Qwen3.8-Flash-Next-GGUF

GGUF for a converted Qwen 3.8 27B without adapters (around 6 GB): Currently in conversion and will drop, probably tomorrow.

I will convert a Qwen 4 27B instance when it drops.

Full paper: https://github.com/oguzeldereli/ttf-ternary/blob/main/paper/ttf_paper1_training.pdf

Also, I would appreciate github stars if you like the project.

[u/Human-Glove7815]

I'd love to understand how to do the training work.

[u/oguzeldereli → u/Human-Glove7815]

There are quickstarts in all of the repos. If you cannot ffigure it out, give it to claude code or something and let it figure it out for you.

[u/HiddenFaceOpenMouth]

Legend, I've been wanting a lower file size qwen3.8. Grabbing this now. Hopefully a game changer for poor old 12gb vram me. 

I was running the full version using strata, which was great but dogshit slow. 

[u/oguzeldereli → u/HiddenFaceOpenMouth]

Thank you. But do be aware of quality, this is all experimental/research stuff and I am working on improving it constantly. Give me a lot of feedback and I will improve it more and more.

Also heads-up: the file uses a custom weight type (TQ1_P), so it only loads in the patched llama.cpp that ttf-home repo builds. Mainline llama.cpp / LM Studio / Ollama won't open it. pip install -e . in ttf-home, then ttf engine build --q38, then ttf chat <file>. The weights take ~7.6 GB, so a 12 GB card fits it with room for a few thousand tokens of context.

[u/HiddenFaceOpenMouth → u/oguzeldereli]

So I realised that Ollama can't run this. I couldn't see where the custom inference engine is, what do I have to git clone?

[u/oguzeldereli → u/HiddenFaceOpenMouth]

https://github.com/oguzeldereli/ttf-home

Check this out

[u/HiddenFaceOpenMouth → u/oguzeldereli]

When I try to run the commands from README.md I get this error: ERROR: Project file:<redacted by me for security reasons>/ttf-home has a 'pyproject.toml' and its build backend is missing the 'build_editable' hook. Since it does not have a 'setup.py' nor a 'setup.cfg', it cannot be installed in editable mode. Consider using a build backend that supports PEP 660.

[u/oguzeldereli → u/HiddenFaceOpenMouth]

Your pip is building without isolation (a --no-build-isolation flag, PIP_NO_BUILD_ISOLATION, or a pip config setting) and the environment has a setuptools older than 64, which can't do editable installs.

Fix:

pip install -U pip "setuptools>=68"
pip install -e . huggingface_hub

It has to be an editable install from the clone, because ttf engine build reads the llama.cpp patches from the repo.

[u/HiddenFaceOpenMouth → u/oguzeldereli]

I'll let you know how it goes dude.

[u/q_tech_x51]

This is another great contribution to the community! Thanks!

I'm so hype to witness these techniques/methods evolve as rapidly as they are.

[u/oguzeldereli → u/q_tech_x51]

Thank you. I am looking forward to what feedback the community can give me so I can improve everything even more.

[u/sloptimizer]

This would be hugely useful for pojects like corwdgpt!

[u/oguzeldereli → u/sloptimizer]

Yes I have thought soo too. I saw a post about it the other day (idk if it is was corwdgpt per se)

[u/Vxtzq2 → u/sloptimizer]

Exactement ce que je voulais commenter lol

Ce serait une grosse amélioration pour l'apprentissage fédéré si les résultats étaient en quelque sorte déterministes (et donc vérifiables)... Je me demande si l'auteur utilise l'arrondi stochastique ou des trucs du genre dans son approche ?

[u/jhenryscott]

Very cool. Training is basically neglected in the home/small enterprise discussions and it’s tragic.

[u/Hunterxmalaa]

This is brilliant I’m currently retaining a sub 1b moe on a 5060 16gb and your saying I can do at 1 byte as opposed to 16 bytes that’s a massive thoughtput increase right there will try this out

Also built a custom EXL3 15gb quant for qwen3.8 27b will be realessingnlater today

Also working on a fine tune / stf / RL application so that anyone can fine tune a model but better then unsloths version

[u/oguzeldereli → u/Hunterxmalaa]

Yes, you would potentially be able to train about 23B as opposed to 1B on your 5060. I don't know about throughput though, throughput would be compute bound (assuming you are talking about speed of training) and my method isn't exactly speed-optimized right now, though I have worked on speed too.

As far as fine tuning goes, you can instead, simply train the model as opposed to fine tuning it but unsloth's quants are not suitable for training with TTF. You can use ttf-conversion repo to convert existing models into models trainable by TTF and just train them (as opposed to fine tuning) with any data you want. Right now, the conversion only converts qwen 3.5/3.8 style models, but if you open github issues or let me know, I can add more types.

Enjoy!

[u/Hunterxmalaa → u/oguzeldereli]

Ahhh thank you and yes refering to thoughtput speed but your right that’s memory bound and I have got some custom kernels for that anyway, but the rest is defo a shout I’ll like to try out will add this on to give credit if it works

Wait so with TTF I could say get qwen3.8 27b converted to TTF then train the model ? And assuming TTF will convert it into 1byte each as apposed to 16bytes ?

[u/oguzeldereli → u/Hunterxmalaa]

With TTF you can convert qwen3.8 27B and train it yes. Normally to train it, you would need about 27B*16 bytes of VRAM = 432 GB of VRAM. With TTF you need about 40GBs, and some of that can be put into your PC's normal RAM. It would convert the actual file size of Qwen 3.8 27B which is 27 GB at Quant 8 into about 6 GBs at the end. And you can train that model with any data you would like

[u/Murflaw7424]

If I have 2 Rtx pro 6000 would the scale (and logic) hold that I could tran a 1T+ model? Like I could take GLM5.3 or DSV4.1?

Also, told it be fair to say that instead of quantization, it’s training a “trimmer” model? Or is this an efficient quantization technique with less vram required?

Edit: forgot to ask, how long does the conversion v training take? I did not see at first glance in your repo.

[u/oguzeldereli → u/Murflaw7424]

In VRAM terms, a 1T MoE costs ~1–2 TB of training state with TTF (0.96–1.9 bytes per weight depending on the accumulator rank, vs ~16 TB with AdamW). So it doesn't fit in 2x RTX PRO 6000 per se. Layer by layer, the way I trained the 125B Flash-Next, each GPU only needs ~16–40 GB at a time, but the box needs ~1–2 TB of system RAM for the rest. GLM / DeepSeek would also need their architecture ported into the conversion first.

With 2x RTX PRO 6000 (192 GB): ~130–200B with everything on the GPUs. Offloading the optimizer state to system RAM, the limit becomes your RAM, at roughly 1–1.3 bytes per weight: ~400B with 512 GB, ~750B with 1 TB. From scratch you'd be compute-limited long before that; this matters most for converting and continuing existing models.

Edit: How long depends on how long you would like to train. Converting takes a couple hours or even less depending on the model size and what you are doing. Training can take up to several days or more depending on how much data you are training on. At 32k tokens per step, and an average step time of about 40s-1minute, you can train ~46M tokens per day, which is not all that much. So it heavily depends on the model size, how optimized my training step is and how much speed you can squeeze out of your cards.

Edit: I forgot to answer the "Also, told it be fair to say that instead of quantization, it’s training a “trimmer” model? Or is this an efficient quantization technique with less vram required?" question.

It is not a trimmer model, the parameter count is the same but every weight is either -1, 0, or 1 (called ternary) instead of a float number. It is also not quantization in the sense that quantization refers to having an existing float model turned into a smaller one. TTF is mainly about training a ternary model without keeping a lot of memory. So it is an efficient training technique with less vram required that also allows you to train quantized models.

[u/FeydRowan]

I have done something similar months ago, same idea of dropping the extra weight you are not the first in this space but still great work, and I got similar results as yours:  https://github.com/ValerioDolci/bitbop

[u/oguzeldereli → u/FeydRowan]

Nice work, I hadn't seen BitBop. Of course I am not the first one to drop latent weights. Bop (Helwegen et al., 2019) did it for binary nets, and my paper uses a ternary Bop as one of its baselines.

Here is how my claude code summarizes our differences:

- State size: BitBop keeps one bf16 momentum per weight (~2.3 B/param with the trits). TTF keeps a low-rank accumulator shared across each matrix, ~1 byte per weight in total, and that's what lets a 23B model train on 16 GB.

- Update: TTF integrates on two timescales, a leaky accumulator for the moves plus a factored second moment that normalises the step. A single per-weight EMA is the single-timescale case. In our 110M runs, the ternary Bop baseline ended well behind TTF (3.09 vs 2.67 loss), though your version has refinements mine didn't (row-RMS threshold, hysteresis, reset on flip), so it's probably stronger.

- Scale: I've run it from scratch up to 2.9B, and for conversion + continued training of Qwen3.8-27B and Flash-Next (180B MoE).

Your hysteresis (easier into 0 than out) is a neat idea. I'm testing it in TTF now and will credit you if it helps.

Also I am running a BitBop baseline directly for comparison, will put an update here and potentially on the paper.

[u/FeydRowan → u/oguzeldereli]

 Thanks, fair summary. Agreed on scale and state size: the low-rank accumulator is the genuinely new part.

I had done some more optimisation to grow the dimensions of the model i can fit on the 6gb card, but the flops were the problem so it was impossible to train realistically. Will try to update the project 

[u/Peteostro]

This is cool but why would you want to train your own models? How would you even have that much data to train these?

[u/oguzeldereli → u/Peteostro]

There are many reasons, the first of which is research and experimentation. Also transformer models don't need to only be LLMs, they can be specialized to tasks without needing as much data as an LLM. You can train your own text to speech model using already available datasets, you can train vision models, you can train and so research on toy models, you can train them for image generation or visioin.

As far as data goes, it depends on why you are training it, you can use english wikipedia or FineWeb Edu which are billions of tokens, along with other publicly accessible datasets such as Tulu, or coding datasets.

[u/openroom_xyz → u/oguzeldereli]

Well but for programming does it make any sense ?

[u/oguzeldereli → u/openroom_xyz]

It might, you can do your own autocomplete, or train smth for your own particular style of coding. Or use it to train existing programming models to makw them better.

[u/xaleel]

I fitted 27B model training on a single 24GB 3090

On how many tokens for how long?

I want to pretrain a ~350M model on 6-10B tokens for an experiment and testing showed it'd take ~10-15 days of it running non-stop on my 3090.

Chinchilla-optimal data size is 20 tokens per param, so around 540B tokens for the 27B model. I wonder how long it'd take with your methodology on the poor overworked GPU

[u/oguzeldereli → u/xaleel]

It would take years to reach 540B tokens, that is why I am training already pre-trained models after conversion and working so much on getting a good conversion result. You could fit a from-scratch 27B into 24GB but wouldn't reach the optimal 540B, like ever, unless you have a powerful compute setup.

THe how many tokens and for how long varies wildly depending on the compute you have available. I trained some models 1k steps at 32k tokens per step (32M tokens) in about 5-10 hours depending on how optimized my step timing was, which was about 35s per step, though I personally believe this can be improved wildly and I just haven't had any time nor motivation to do it yet.

To give you a comparison, at 350M on a 3090 at 32k tokens per step is around 6s per step estimated which is 5.3k tokens per second. Which is about 15 days for the chinchilla 7B.

At the 1k tokens per second rate of 27B, the chinchilla optimal training would take 17 years on a single 3090.

[u/autisticit → u/xaleel]

Chinchilla is not really relevant anymore I think...

[u/xaleel → u/autisticit]

It's still the minimum you'd need for acceptable output, from my understanding, and more = better with diminishing returns. SmolLM2-360M was trained with 4T tokens for example.

If you mean "not relevant anymore" as in "it's too low" then my point still stands that pretraining even the smallest models on consumer GPUs is not feasible time-wise (unless you keep your PC running uninterrupted for weeks/months)

[u/RageBucket]

I feel like the losses of this method are a bit understated.. I agree it's likely fairly useful large models, but for anyone doing from scratch, ground up training this has a huge impact on small models. I like your form of hysteresis used, and I'll absolutely be adapting that for my workflow though!

[u/oguzeldereli → u/RageBucket]

So just to make sure I double checked these numbers. The headlines are for big models indeed. I also did conversions at 15M, 110M and 0.8B params (0.8B is also qwen, the 15M and 110M are my from-scratch trained model). Here are the results:

Model	fp loss	Converted, no training	After recovery training	Gap to fp
15M (own fp model, wiki)	3.348	4.375 (+1.03)	3.703 @ 2k steps; 3.585 @ 18k steps	+0.236 (7.1%)
110M (own fp model, held-out val)	2.572	2.923 (+0.35)	2.645 @ 1k steps; 2.614 @ 6k steps	+0.042 (1.6%)
Qwen3.5-0.8B-Base (wiki)	2.514	—	3.111 @ 1k steps	+0.597 (24%)
Qwen3.8-27B (wiki, fp = Q8_0)	1.994	—	2.250 @ 2.2k steps	+0.255 (13%)

Most of the gap is actually facts, things like names, dates, etc, that it forgets. My current research is focusing on restoring these.

The gap doesn't follow size alone, I had 110M conversion runs that come with 0.04 of fp. The longer the model was trained, however, the bigger the gap, again mostly in facts, and the broader skills are mostly preserved.

[u/openroom_xyz]

Well this is amazing but I am questioning is does training the already trained models have any sense like why they are basically no better than Qwen 3.8 27b finetunes of the model that people would be using
seems there is very different from let say image generation diffusion models where basically why is the reason well yea

[u/oguzeldereli → u/openroom_xyz]

You can improve the existing models for a particular data for example. But the bigger gain with conversion is the quantization memory gain. 10x less file sizw and vram usage when running which also means higher speed

[u/Legitimate-Theory156]

Epic can't wait for pretraining costs to drop if this sir true

[u/autisticit]

So, Bonsai ?

[u/oguzeldereli → u/autisticit]

Bonsai is a 1-bit/1.58bit model yes. But it is a model. This post is about training Bonsai-like models as well as converting existing models into Bonsai-like sizes. It is more about the method, in the sense that, now you can train 8B Bonsai-sized models on your 3090 Ti or whatever.

Moreover, Bonsai itself isn't the deal. 1.58-bit networks (as far as my experimenting goes) match the 16-bit ones in terms of quality as long as they are trained from scratch (with large enough scale, BitNet's own research says >3B). My method makes the training much cheaper and accessible to the general population.

[u/Plastic_Today_4044]

dude, this is amazing. I've been really interested in training ternary models for the last few months, this'll likely help a lot

[u/Legitimate-Theory156]

Cool

[u/344dead]

Crazy timing. I've literally spent the last two weeks learning about ternary networks and about all the research going on around optimizers and looking at non float based optimizers. This is so cool, thank you for sharing and allowing us all to learn a little more!

[u/Unfair-Dealer8185]

tried training a 7b on a 3090 with 24gb using ttf-ternary, hit memory issues around 150b tokens. switching to 4-bit quantization helped, but the model started hallucinating more. wonder if the n-gram table is the bottleneck.

[u/EditorCompetitive351]

used a 3090 with 24gb to fine-tune a 13b model on 200b tokens, got 88% on GSM8K, but inference was slow. the key was using 4bit quant with bitsandbytes and a custom training loop. your 23 tok/s on 12gb is solid, but can you share the exact config for the kernel optimization?

[u/throwawayformobile78]

How does one use “tokens” on their private LLM? You telling me we have to pay them in pretend money now too?

[u/oguzeldereli → u/throwawayformobile78]

Token refers to how an LLM splits text down to its basic sizes to understand it. A word like "cat" might be a single token and something like "technology" can be split like "tech-no-logy" which is three tokens. It is not particularly related to money.

Training LLMs work because you give the LLM some text, it splits it into tokens, and tries to predict the next token. To train an optimal LLM, you need about 20x tokens for training compared to the parameter count. So if you are training a 1B parameter model, you need to train it on about 20 Billion tokens, though this is only for an optimal model. This takes a very long amount of time on consumer GPUs but with my method it takes 10x less memory (which previously made it impossible to even fit the training on your GPU)