Structured Code Generation for Deep-Learning Computations via Multi-Dimensional Homomorphisms (MDH.DL)

• Applicant: Professor Dr. Sergei Gorlatch
• Subject Area: Computer Architecture, Embedded and Massively Parallel Systems
• Term: since 2022
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 470527619

Deep Learning (DL) has revolutionized the world. Started in 2012, the AlexNet deep neural network was able to significantly outperform existing methods to image classification. A fundamental requirement for the success of DL is the software that implements DL: to make DL practicable, it is essential to achieve high performance for DL computations, and the software needs to be (performance) portable over different kinds of hardware architectures, ranging from mobile devices to high-performance computing clusters. Moreover, to make DL software attractive for the common DL domain scientist (who is generally not familiar with hardware details and low-level code optimizations), DL software must also be productively usable by the scientist – by offering easy-to-use programming abstractions for implementing DL programs, agnostic of hardware and optimization details. Furthermore, to meet the complex and increasing real-world demands on DL, DL software must be easily adaptable to new demands, due to the continuously increasing complexity of DL algorithms and the frequently changing landscape of computer systems in order. During our predecessor DFG project PPP-DL, we have designed and developed the algebraic formalism of Multi-Dimensional Homomorphisms (MDH) – a formal representation for DL computations, such as Matrix Multiplication and Convolution. We demonstrated that MDH achieves significantly higher performance, portability, and/or productivity for DL computations on GPUs and CPUs as compared to major DL approaches, including hand-optimized vendor libraries provided by NVIDIA and Intel which are viewed as gold standards the research and application communities. This new project proposal aims at designing and implementing a novel code generation approach for DL that relies on our formalism of MDH developed in our predecessor project PPP-DL. The ultimate goal of our envisaged code generation approach is to simultaneously achieve high performance, portability, and productivity for DL: 1. Performance: by optimizing entire DL programs (which consist of multiple DL computations, such as Matrix Mult. and Convolution); for this, we aim to use our MDH formalism as foundation, because it allows expressing DL computations uniformly in the same formalism and consequently to reason about optimizations across the computations. 2. Portability: by elaborating a generalized approach for supporting for DL-specific hardware (such as NVIDIA’s popular Tensor Cores) for our intended code generation approach. For this, we aim to exploit the algebraic high-level semantic information captured within our MDH representation for DL computations to check if DL-specific hardware can be (legally) used or not. 3. Productivity: by expressing DL computations conveniently via the optimization-agnostic high-level language of MDH; we plan in this project to significantly widen the expressivity of MDH, e.g., to express entire DL programs instead of individual DL computations only.

• DFG Programme: Research Grants
• International Connection: Denmark, France, United Kingdom, USA
• Cooperation Partners: Professor Albert Cohen, Ph.D.; Professor Tobias Grosser, Ph.D.; Vinod Grover, Ph.D.; Professorin Dr. Mary Hall, Ph.D.; Professor Dr. Cosmin Oancea, Ph.D.
• Co-Investigator: Professor Dr. Michel Steuwer