A Real-World Evaluation of the Implementation of NLP Technology in Abstract Screening of a Systematic Review

Autor/inn/en	Perlman-Arrow, Sara; Loo, Noel; Bobrovitz, Niklas; Yan, Tingting; Arora, Rahul K.
Titel	A Real-World Evaluation of the Implementation of NLP Technology in Abstract Screening of a Systematic Review
Quelle	In: Research Synthesis Methods, 14 (2023) 4, S.608-621 (14 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Perlman-Arrow, Sara) ORCID (Bobrovitz, Niklas)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	1759-2879
DOI	10.1002/jrsm.1636
Schlagwörter	Natural Language Processing; Screening Tests; COVID-19; Pandemics; Accuracy; User Satisfaction (Information); Efficiency; Scores; Evidence; Synthesis + Suchen Sie Ihr Suchwort? Natürliche Sprache; Screening-Verfahren; Benutzerfreundlichkeit; Effectiveness; Effektivität; Wirkungsgrad; Evidenz
Abstract	The laborious and time-consuming nature of systematic review production hinders the dissemination of up-to-date evidence synthesis. Well-performing natural language processing (NLP) tools for systematic reviews have been developed, showing promise to improve efficiency. However, the feasibility and value of these technologies have not been comprehensively demonstrated in a real-world review. We developed an NLP-assisted abstract screening tool that provides text inclusion recommendations, keyword highlights, and visual context cues. We evaluated this tool in a living systematic review on SARS-CoV-2 seroprevalence, conducting a quality improvement assessment of screening with and without the tool. We evaluated changes to abstract screening speed, screening accuracy, characteristics of included texts, and user satisfaction. The tool improved efficiency, reducing screening time per abstract by 45.9% and decreasing inter-reviewer conflict rates. The tool conserved precision of article inclusion (positive predictive value; 0.92 with tool vs. 0.88 without) and recall (sensitivity; 0.90 vs. 0.81). The summary statistics of included studies were similar with and without the tool. Users were satisfied with the tool (mean satisfaction score of 4.2/5). We evaluated an abstract screening process where one human reviewer was replaced with the tool's votes, finding that this maintained recall (0.92 one-person, one-tool vs. 0.90 two tool-assisted humans) and precision (0.91 vs. 0.92) while reducing screening time by 70%. Implementing an NLP tool in this living systematic review improved efficiency, maintained accuracy, and was well-received by researchers, demonstrating the real-world effectiveness of NLP in expediting evidence synthesis. (As Provided).
Anmerkungen	Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01