Analysis and automatic estimation of the quality of machine-generated texts

• Applicant: Professor Dr.-Ing. Sebastian Möller
• Subject Area: General and Comparative Linguistics, Experimental Linguistics, Typology, Non-European Languages ; Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing
• Term: since 2020
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 436813723

This research project deals with the Quality of Experience of machine-generated texts. The focus within the project lies on the two text types machine translation and automatic summarization. The project aims to identify perceptual quality dimensions, provide subjective methods to quantify these quality dimensions, automatically determine extractable parameters which correlate with the text quality, and thereby develop predictive models which can provide an estimation of the overall quality of a machine-generated text. The predictive models are being developed with the help of various corpora for German and English. Based on these corpora, dimensions that are relevant for the quality estimation are being defined and verified in tests with study participants (through crowdsourcing and in the laboratory). For each identified quality dimension, relevant text parameters are automatically being extracted from the texts which are then correlated with the subjective evaluations of the dimension. Predictive models based on the parameters are being constructed for both text types – for separate quality dimensions as well as for the overall quality – and analyzed concerning the differences between the text types. Finally, the predictive models are being tested regarding their generalizability in an independent test series.

• DFG Programme: Research Grants