Eidetic Representations of Natural Language

• Applicant: Professor Dr.-Ing. Alan Akbik
• Subject Area: Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing
• Term: since 2021
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 448414230

Neural language modeling is currently one of the main drivers of progress in natural language processing (NLP). These models are trained using very large collections of unlabeled text data and so automatically learn a wide array of semantic and syntactic linguistic knowledge. Numerous models were developed in the last two years (such as ELMo, FLAIR, BERT and GPT-3) leading to breakthroughs across many NLP tasks, and revolutionizing the field. Our own prior work FLAIR is one of the first works of this kind and achieved new state-of-the-art scores for dozens of classic NLP tasks.However, current approaches for neural language modeling suffer from three fundamental limitations that we have begun to investigate:(1.) Current models require gigantic collections of text data and huge GPU-clusters. The recent GPT-3 model for instance was trained on text of 500 billion tokens and required a custom GPU-supercluster to train. Such resources lie far outside of what most universities and companies have available. With this proposal we address this issue by conducting research in "Sample-Efficient Learning". Our goal is to create language models that require significantly fewer data than current approaches.(2.) Current models internally use purely latent representations of semantics, making it difficult to analyse their inner workings. With this proposal, we conduct research in structured embedding spaces to create language models with a partial structure. Our goal is to facilitate error analysis and ultimately make these models more robust.(3.) Current models are fully trained only once and then applied to downstream tasks. However, this means that the knowledge contained in neural language models remains unchanged after training is complete. Based on our prior work we instead argue that similar to how humans continuously learn and improve their understanding of the world, we require language models that never stop learning, even when applied to downstream tasks. We thus research novel approaches for never-ending learning.With this proposal we build on our prior work and conduct research in Sample-Efficient Learning, Structured Representations and Never-Ending Learning to address these three fundamental issues and so create "Eidetic Representations of Natural Language". We expect this research to yield far-reaching improvements across many NLP tasks.

• DFG Programme: Independent Junior Research Groups