Improved Music Appreciation in Cochlear Implant Listeners via Optimization and Learning of Parametric Teacher-Student Models

• Applicants: Professor Dr.-Ing. Rainer Martin; Dr.-Ing. Anil Nagathil
• Subject Area: Acoustics
• Term: since 2024
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 544277892

Cochlear implants (CI) have been remarkably successful in restoring speech perception in severely hearing-impaired or even deaf people by bypassing the auditory periphery and stimulating the auditory nerve electrically via an electrode array. Nevertheless, CIs entail a number of technical and physiological restrictions for the accurate transmission of acoustic signals. For instance, the low number of stimulation electrodes used in current CI devices allows only a coarse representation of the spectral content, which is additionally distorted by a spread of electrical fields within the cochlea. This implicates that CI users have difficulties to identify melodies or to distinguish music instruments. The spectral and temporal complexity of polyphonic music impedes the access to music and results in degraded music enjoyment. To improve music perception in CI users, different music pre-processing strategies have been proposed recently, which aim at reducing a music piece to its most essential characteristics and thus facilitate a more faithful transmission through the electrode-nerve interface. However, the performance of these strategies, as rated by CI users, strongly depends on the musical genre and on various characteristics of the music signal. This indicates the need for more flexible and customizable music processing algorithms. The core objective of this project is to design novel music processing schemes which help CI users to access essential characteristics of music and derive enjoyment from listening to it. We will develop a parametric music processing model which can be optimized for specific musical styles and complexity levels in a signal-adaptive and user-controllable way. To this end, we will develop and explore a tool chain of music processing methods which first decomposes and enhances individual music tracks (i.e. separate instruments or voices) and then composes a remix of all processed tracks, yielding a teacher source mixture model. The optimization of the teacher model parameters will be steered by different cost functions. These rely on existing auditory models of normal and electric hearing and on feedback from normal-hearing listeners and CI users. The proposed methodology aims at providing optimized parameter settings while taking music characteristics such as genre, complexity, or tempo into account. The optimized teacher models will then be approximated by DNN-based student systems which, unlike the teacher models, do not require separated instrument tracks but can generate the desired remix for CI listeners solely based on the original music signal.

• DFG Programme: Research Grants