Source Separation and Restoration of Sound Components in Music Recordings

• Applicant: Professor Dr. Meinard Müller
• Subject Area: Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing ; Data Management, Data-Intensive Systems, Computer Science Methods in Business Informatics
• Term: since 2016
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 328416299

This project aims at the development of techniques for separating and restoring sound events as occurring in complex music recordings. In the first phase (initial proposal), we focused on percussive sound sources, where we decomposed a drum recording into individual drum sound events. Using Non-Negative Matrix Factor Deconvolution (NMFD) as our central methodology, we studied how to generate and integrate audio- and score-based side information to guide the decomposition. We tested our approaches within concrete application scenarios, including audio remixing (redrumming) and swing ratio analysis of jazz music. In the second phase of the project, our goals will be significantly extended. First, we want to go beyond the drum scenario by considering other challenging music scenarios, including piano music (e.g., Beethoven Sonatas, Chopin Mazurkas), piano songs (e.g., Klavierlieder by Schubert), and string music (e.g., Beethoven String Quartets). In these scenarios, our goal is to decompose a music recording into individual note-related sound events. As our central methodology, we plan to develop a unifying audio decomposition framework that combines classical signal processing and machine learning with recent deep learning (DL) approaches. Furthermore, we want to adopt generative DL techniques for improving the perceptual quality of restored sound events. As a general goal, we will investigate how prior knowledge, such as score information can be integrated into DL-based learning to improve the interpretability of the trained models.

• DFG Programme: Research Grants