Relevance Assessment Tool (RAT): Development and enhancement of a software for conducting studies based on results from search systems

• Applicant: Professor Dr. Dirk Lewandowski
• Subject Area: Data Management, Data-Intensive Systems, Computer Science Methods in Business Informatics
• Term: since 2021
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 460676551

The project's overall goal is to continue to develop and enhance the Relevance Assessment Tool (RAT) so that it will become a stable, flexible and sustainable tool for conducting studies that deal with the collection and analysis of data from search engines and other information retrieval systems.The Relevance Assessment Tool (RAT) is a software toolkit that allows researchers to conduct large-scale studies based on results from (commercial) search engines and other information retrieval systems. It consists of modules for (1) designing studies, (2) collecting data from search systems, (3) collecting judgments on the results, (4) downloading/analysing the results. A starting point to developing RAT was the fact that retrieval effectiveness studies usually require much manual work in designing the test, collecting search results, finding jurors and collecting their assessments, and in analysing the test results, as well. RAT addresses all these issues and aims to offer help in making retrieval studies more efficient and effective.The design of the RAT prototype has been guided by the requirement of researchers to query external information retrieval systems in general, and search engines in particular. This need derives from researchers' interest in the quality of their search results, the composition of their results (lists or pages), and potential biases in the results, to name but a few. This is a unique approach that separates RAT from software developed to aid retrieval evaluation where researchers have complete control over the systems evaluated.RAT allows studies to be conducted under the two major evaluation paradigms, namely system-centred and user-centred evaluations. RAT will also be useful to any other researchers aiming to use results from search engines for further analysis. While the basis of RAT lies in information retrieval evaluation, its use goes well beyond this research area.Already in the prototype phase, we found that there is a significant need for a software tool like RAT in the search engine and information retrieval communities, respectively. Some projects have already demonstrated this.In the dissemination of the software, we will first focus on the search engine research/information retrieval communities, and then roll out the software further. To address these communities, we will (1) have the software development accompanied by some pilot test users; (2) promote the software through conferences, an advisory board, and publications. To target the intended researchers, we aim to make the software well usable also to researchers who are not in the field of information science or computer science. All software developed will be made available open source.We will ensure the sustainability of the project results through measures in three areas: (1) Establishing and distributing the software, (2) establishing and maintaining a user and developer community, (3) publishing the software open source.

• DFG Programme: Research data and software (Scientific Library Services and Information Systems)