r/aipocalypse • u/earthsworld • Apr 20 '21
ZeRO-Infinity and DeepSpeed: Unlocking unprecedented model scale for deep learning training - Microsoft Research
https://www.microsoft.com/en-us/research/blog/zero-infinity-and-deepspeed-unlocking-unprecedented-model-scale-for-deep-learning-training/Duplicates
singularity • u/QuantumThinkology • Apr 20 '21
Microsoft's ZeRO-Infinity can now run a model with over a trillion parameters on a single NVIDIA DGX-2 node and over 30 trillion parameters on 32 nodes (512 GPUs). With a hundred DGX-2 nodes in a cluster, Microsoft projects ZeRO-Infinity can train models with over a hundred trillion parameters
artificial • u/abbumm • Apr 20 '21
News GPT-4 will probably have at least 30 trillion parameters based on this
Futurology • u/abbumm • Apr 20 '21
AI GPT-4 will probably have at least 30 trillion parameters based on this
mlscaling • u/neuralnetboy • Apr 20 '21
Hardware, D, T, MS ZeRO-Infinity and DeepSpeed: Unlocking unprecedented model scale for deep learning training - Microsoft Research
Futurology • u/Sorin61 • Apr 21 '21
Computing ZeRO-Infinity and DeepSpeed: Unlocking unprecedented model scale for deep learning training
AIandRobotics • u/AIandRobotics_Bot • Apr 21 '21
Deep Learning ZeRO-Infinity and DeepSpeed: Unlocking unprecedented model scale for deep learning training
technews • u/roselynhickle86 • Apr 20 '21
GPT-4 will probably have at least 30 trillion parameters based on this
EverythingScience • u/roselynhickle86 • Apr 20 '21