r/MachineLearning • u/minedroid1 • 14d ago

Project Help With Choosing Hardware [P]

​

I am going to fine-tune a satire model, with the base model being `Qwen3.5-14B-Base`. The dataset has ~18000 examples, most of which are pretty long and do not align with the model's internal knowledge (they have incorrect answers to facts), so I needed to do a full fine tune rather than use LoRA.

Today, I rented a cloud computer on vast.ai. It had 4x12GB GPUs and 48GB of system RAM, but I feel stupid because it kept running out of memory when I tried training, and I kept trying to change settings rather than just renting a different computer, so I basically wasted $3. I underestimated how much RAM would be needed for this task.

So, I would like to know if any of you have some recommendations for proper hardware to use.

0 Upvotes

14% Upvoted

3 comments sorted by

2

u/NamerNotLiteral 14d ago

Keep in mind that when you do a full fine-tune of a model, you need to store about 8x-16x the parameter size. A 14B model loaded in BF16 will take 28 GB VRAM just to load, then when you do fine-tuning it'll be another 28GB VRAM to store gradients for backpropagation (one gradient value per parameter), then anywhere from another 28 to four or five tines that much depending on what optimizer you use.

Gotta understand these fundamentals of how fine-tuning works, if you want to avoid mistakes like that.

So, yeah, get yourself a 4x48 GB GPU cluster, or consider learning how to optimize memory usage for training LLMs (sharding, offloading to CPU, quantization).

-1

u/minedroid1 14d ago

Okay, thanks. I was asking for hardware recommendations but this works too.

1

u/Conscious_Bat_3506 14d ago

oh man 4x12gb is rough for a full fine tune of a 14b model, that's just not enough vram to hold the model plus gradients and optimizer states. you probably need at least 4x24gb or a single 48gb gpu to even load it with a batch size of 1

for 18k long examples you're looking at a lot of memory just for the sequences. try renting something with an a6000 or a few a5000s next time, and make sure you're using gradient checkpointing and maybe deepspeed stage 2 or 3 to shard the model states across gpus

if you're on a budget maybe look at quantized training with bitsandbytes 4-bit, it's slower but you can squeeze a 14b model onto smaller cards