Systematic reviews are vital for evidence-based decision-making but remain resource-intensive due to the volume of literature requiring expert screening. Technology-Assisted Review (TAR) systems offer a solution by ranking documents for review, yet questions remain about how best to allocate limited human effort across multiple review topics. In this paper, we explore the problem of effort distribution by comparing alternative screening policies under fixed effort constraints. Using real-world data from the CLEF eHealth 2017-2019 TAR tasks, we evaluate both baseline and adaptive policies that account for topic size, screening depth, and residual uncertainty. We introduce effort-aware evaluation metrics to measure trade-offs between review effectiveness and resource use. Our results show that simple, topic-sensitive policies can significantly improve the yield of relevant documents discovered, offering practical insights for scalable and equitable systematic review workflows.

POLAR: Policy Optimization for Literature Analysis under Review Constraints

Di Nunzio G. M.
2025

Abstract

Systematic reviews are vital for evidence-based decision-making but remain resource-intensive due to the volume of literature requiring expert screening. Technology-Assisted Review (TAR) systems offer a solution by ranking documents for review, yet questions remain about how best to allocate limited human effort across multiple review topics. In this paper, we explore the problem of effort distribution by comparing alternative screening policies under fixed effort constraints. Using real-world data from the CLEF eHealth 2017-2019 TAR tasks, we evaluate both baseline and adaptive policies that account for topic size, screening depth, and residual uncertainty. We introduce effort-aware evaluation metrics to measure trade-offs between review effectiveness and resource use. Our results show that simple, topic-sensitive policies can significantly improve the yield of relevant documents discovered, offering practical insights for scalable and equitable systematic review workflows.
2025
CIKM 2025 - Proceedings of the 34th ACM International Conference on Information and Knowledge Management
34th ACM International Conference on Information and Knowledge Management, CIKM 2025
File in questo prodotto:
File Dimensione Formato  
polar_cikm.pdf

accesso aperto

Tipologia: Published (Publisher's Version of Record)
Licenza: Creative commons
Dimensione 1.03 MB
Formato Adobe PDF
1.03 MB Adobe PDF Visualizza/Apri
Pubblicazioni consigliate

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11577/3591806
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 1
  • ???jsp.display-item.citation.isi??? 1
  • OpenAlex ND
social impact