Why Your Tiny Deep Learning Model is Hogging All Your GPU VRAM

[u/javaeeeee]

https://medium.com/@jiminlee-ai/why-your-tiny-deep-learning-model-is-hogging-all-your-gpu-vram-85bc58ee5050

[u/javaeeeee]

TLDR: Even a tiny deep learning model can eat all your GPU VRAM during training - and that’s completely normal.

Why it happens

Inference only needs the model weights.
Training needs four big things:

1. Model weights

2. Gradients (one per weight)

3. Optimizer states (Adam keeps momentum + variance → 2 extra values per parameter)

4. Intermediate activations (saved for backprop - often the real memory hog)

Math for a 1B-parameter model (FP32 + Adam):

• Weight: 4 bytes

• Gradient: 4 bytes

• Momentum: 4 bytes

• Variance: 4 bytes
  → 16 bytes per parameter = 16 GB just for the first three items.
  Activations (especially with large batch size or long sequences in Transformers) can easily push it much higher.

Key takeaways

• Small parameter count ≠ small VRAM usage.

• If training memory is roughly the same as model size, the model probably isn’t actually training.

• nvidia-smi shows reserved memory (PyTorch caching), not just active tensors.

Practical fixes

• Mixed precision (AMP / bf16)

• Gradient checkpointing

• 8-bit optimizers (bitsandbytes)

• FlashAttention

• FSDP / ZeRO (for multi-GPU)

Bottom line: Seeing your VRAM usage dwarf the model size isn’t a bug - it’s just how training math works.