Want to create an interactive transcript for this episode?
Podcast: The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)
Episode: Parallelism and Acceleration for Large Language Models with Bryan Catanzaro
Description: Today we’re joined by Bryan Catanzaro, vice president of applied deep learning research at NVIDIA.Most folks know Bryan as one of the founders/creators of cuDNN, the accelerated library for deep neural networks. In our conversation, we explore his interest in high-performance computing and its recent overlap with AI, his current work on Megatron, a framework for training giant language models, and the basic approach for distributing a large language model on DGX infrastructure. We also discuss the three different kinds of parallelism, tensor parallelism, pipeline parallelism, and data parallelism, tha...