Fine-tuning LLMs on Multi GPUs
The second training session in the series titled Foundations of LLM Mastery, organized by EuroCC Austria, is scheduled for 26 February 2025. This event, called Fine-tuning LLMs on Multi GPUs, is being conducted in collaboration with the VSC Research Center and TU Wien.
Scaling fine-tuning large language models (LLMs) to multiple GPUs can unlock new levels of performance and efficiency, making it accessible for industries of all sizes. In this 3.5-hour course, participants from start-ups, SMEs, and large enterprises will gain hands-on experience with powerful multi-GPU fine-tuning techniques, optimizing their LLM workflows for both speed and scalability.
Key topics include:
- Distributed Data Parallel (DDP): This technique efficiently distributes data and model updates across multiple GPUs.
- ZeRO with DeepSpeed: This approach allows for scaling large models while effectively managing memory.
- Fully Sharded Data Parallel (FSDP): Another method to reduce memory usage by sharding model parameters.
- Hugging Face Accelerate: A tool that simplifies multi-GPU training with user-friendly tools and configurations.
Through guided exercises and real-world examples, this course provides participants with the skills to fine-tune LLMs efficiently across multiple GPUs, preparing them to tackle complex, large-scale projects. By the end of this course, participants will have developed a foundational understanding of techniques to distribute model parameters and states across GPUs on several nodes.