The Best is Yet to Come: A Reproducible Analysis of CLEF eHealth TAR Experiments

The CLEF eHealth Technology Assisted Reviews (TAR) in Empirical Medicine Tasks focused on evaluating the effectiveness of various technology-assisted review systems in assisting healthcare professionals in retrieving relevant information from vast amounts of medical literature. It ran for three years, from 2017 until 2019, giving the opportunity to research groups to conduct experiments and share results on automatic methods to retrieve relevant studies with high precision and high recall. In this paper, we perform a reproducibility study of one of the top-performing systems of both the years 2018 and 2019 by rerunning the original code that was provided by the authors of the paper. The goals of this paper are 1) to document the pitfalls in the description of the code, 2) to reorganize the code using a better reproducibility approach (R markdown), 3) to propose some minor changes to the code that would improve the performances of the system.