The TREC files: the (ground) truth is out there

Abstract

Traditional tools for information retrieval (IR) evaluation, such as TREC’s trec_eval, have outdated command-line interfaces with many unused features, or ‘switches’, accumulated over the years. They are usually seen as cumbersome applications by new IR researchers, steepening the learning curve. We introduce a platformindependent application for IR evaluation with a graphical easy-touse interface: the TREC_Files Evaluator. The application supports most of the standard measures used for evaluation in TREC, CLEF, and elsewhere, such as MAP, P10, P20, and bpref, as well as the Averaged Normalized Modified Retrieval Rank (ANMRR) proposed by MPEG for image retrieval evaluation. Additional features include a batch mode and statistical significance testing of the results against a pre-selected baseline.