Abstract:
Experimental evaluation has always been central to Information Retrieval research. The field is increasingly moving towards online evaluation, which involves experimenting with real, unsuspecting users in their natural task environments. Essentially, the production search engine operates as a "living lab." For a long time, this type of evaluation was only available to those working within organizations that operate a search engine. This, however, is about to change as pioneering efforts to realize the idea of living labs in practice are now in place and are available to the community. Specifically, with the recent introduction of the Living Labs for IR Evaluation initiative at CLEF and the OpenSearch track at TREC, researchers can now have direct access to such labs. In this talk, I will provide an overview of the underlying theory, introduce the living labs for IR evaluation platform, and discuss the specific use-cases studied at CLEF and TREC.

Bio:
Krisztian Balog is a full professor at the University of Stavanger where he leads the Information Access & Interaction research group. He received his PhD from the University of Amsterdam, and worked as a postdoc at the Norwegian University of Science and Technology (NTNU), before joining the University of Stavanger. His general research interests lie in the use and development of information retrieval, information extraction, and machine learning techniques for intelligent information access tasks. His current research concerns entity-oriented and semantic search, and novel evaluation methodologies. He serves as a senior programme committee member at SIGIR, CIKM and ECIR, as an Associate Editor of the ACM Transactions on Information Systems, and as a current and former coordinator of information retrieval benchmarking efforts at TREC and CLEF.