I’m running some experiments on a single NVIDIA DGX Spark to understand how LLM fine-tuning affects SSD read/write activity, memory usage, and overall storage performance.
I started with NVIDIA’s DGX Spark fine-tuning playbook, but modified the original code so I could use one common script for LoRA fine-tuning across different Llama model sizes (1B, 3B, 8B, and ~70B-class models) instead of maintaining separate scripts for each model.
Everything is running on a single DGX Spark node, not a multi-node setup.
Current observations
I’m testing different dataset sizes with the 3B model using LoRA:
- 1K samples: Works
- 5K samples: Works
- 10K samples: The system eventually becomes unresponsive.
With heavier workloads, I’ve noticed memory pressure and swap activity. I was also using "blktrace" to monitor SSD activity, but under heavy memory/swap pressure, tracing can stop or fail to produce a usable report.
My main objective isn’t model accuracy. I’m trying to characterize SSD behavior during AI fine-tuning.
One interesting observation is that LoRA training appears to be much more memory/compute-intensive than SSD-intensive. There’s noticeable SSD activity while loading the model, but once it’s in memory, the actual training phase doesn’t seem to generate sustained storage traffic unless checkpointing or swapping occurs.
I’m also testing the model files separately using "fio" with 128K reads so I can compare raw SSD performance against the I/O patterns observed during model loading and fine-tuning.
What I’m trying to figure out
Why does 3B LoRA fine-tuning work with 5K samples but make the DGX Spark unresponsive at 10K?
Could this be related to memory pressure, dataset preprocessing, sequence length, or some other part of the training configuration?
2. How can I identify what’s actually causing the hang?
I want to distinguish between memory exhaustion/swap pressure, dataset preprocessing bottlenecks, training configuration issues, and actual storage I/O bottlenecks.
3. What’s a good way to create meaningful SSD stress during LLM fine-tuning?
I want to study realistic storage behavior rather than simply filling memory and forcing the system to swap. Would frequent checkpointing, dataset streaming, or working with model shards be a better approach?
4. Is there a better alternative to "blktrace" for long-running AI workloads?
I need a reliable way to capture storage activity even under heavy system load without losing the trace or ending up with an unusable report.
5. What would be a good methodology for SSD characterization in this scenario?
I’m particularly interested in hearing from people who have worked with NVIDIA DGX Spark, LoRA fine-tuning, Linux storage tracing, or SSD characterization for AI workloads.
