Biography
Nouamane Tazi is a Machine Learning Engineer at Hugging Face, focused on large-scale distributed LLM training. He is the lead author of the Ultra-Scale Playbook and a core developer of Nanotron, Hugging Face's open-source distributed training library. His work spans projects like StarCoder2, SmolLM3, and Mixture-of-Experts scaling. He is passionate about making large-scale training practical and accessible.
Session
- AI For TechSpeaker
Workshop 1 - The Ultra-Scale Talk: Scaling Training to Thousands of GPUs
Tuesday 22 September, 10:00 – 11:20Workshop Room ATraining large language models demands more than raw compute. it requires a deep understanding of parallelism and architecture choices. This talk dives into the practicalities of ultra-scale training: how 5D parallelism makes it possible to stretch a single run across massive GPU clusters, how Mixture-of-Experts architectures introduce new scaling dimensions and stability challenges, and the performance tuning and communication patterns that drive throughput. Drawing from the Ultra-Scale Playbook and real-world scaling efforts, we'll cover benchmarks, hard-earned lessons, and hands-on recommendations for engineers and researchers ready to train state-of-the-art models efficiently!