Hallucinations in Neural Text Generation: A Theoretical Perspective

• Applicant: Professor Dr. Damien Garreau
• Subject Area: Methods in Artificial Intelligence and Machine Learning ; Theoretical Computer Science
• Term: since 2026
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 573859924

Neural text generation has become widespread, especially with the rise of large language models and their applications as chatbots such as ChatGPT. Although the text they produce is often fluent, that is, free from grammatical or syntactic errors, its meaning can sometimes be entirely disconnected from the input. This phenomenon, known as hallucination, is the central focus of our project. Our goal is to lay the groundwork for a theoretical understanding of hallucinations in neural text generation. We ask: (1) Can hallucinations be reliably detected? and (2) Are hallucinations an unavoidable consequence within the current model frameworks? To address the first research question, we propose to initiate the theoretical study of existing detection methods. In simplified settings, we aim to analyze uncertainty-based detection methods, via the characterization of the distribution of the random outputs. In a second phase, we set our sight on proving that hallucinations are indeed unavoidable in the classical encoder / decoder transformer-based architecture. We plan to focus on two scenarios. First, token insertion, a somewhat artificial way of increasing the hallucination rate. Second, memorized examples, which are empirically more prone to generating hallucinations.

• DFG Programme: Research Grants