Scalable Distributed Deep Learning Optimization

• Applicant: Dr. Sebastian Stich
• Subject Area: Methods in Artificial Intelligence and Machine Learning
• Term: since 2025
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 553954458

The exponential growth of datasets and model complexity in machine learning necessitates scalable optimization methods to efficiently leverage distributed systems. This proposal addresses the challenges in distributed deep learning optimization, particularly in non-convex settings. Traditional optimization methods and insights from convex optimization fail to generalize well to these scenarios, leading to inefficiencies and increased resource usage. Our research aims to develop and validate new optimization techniques robust to the unique challenges of distributed deep learning. We will focus on scalable federated learning and decentralized optimization methods, addressing issues such as drift correction, parameter scaling rules, and the integration of advanced neural network architectures like transformers. By benchmarking these techniques in various simulated environments and real-world applications, we aim to derive comprehensive guidelines that enhance the efficiency and scalability of distributed deep learning. Our interdisciplinary team combines expertise in stochastic optimization, federated learning, and neural network training, ensuring a thorough approach to solving these complex problems. The outcomes of this project will contribute significantly to the field of distributed machine learning, offering practical solutions for efficient large-scale model training, which in turn will help reduce the carbon footprint of deep learning.

• DFG Programme: Research Grants