Status : Verified
| Personal Name | Descallar, Juwaln Diego |
|---|---|
| Resource Title | Evaluating the Performance of Large Language Models for Automated Abstract Screening in Scoping Reviews: A Prompt-Based Simulation Study |
| Date Issued | May 2026 |
| Abstract | Scoping reviews are a form of evidence synthesis that map the extent, range, and nature of research activity on a broad topic, providing a foundation for identifying knowledge gaps and informing future research and policy. Abstract screening, typically conducted by two or more independent reviewers, is one of the most resource-intensive phases of scoping reviews, motivating interest in automation through large language models (LLMs). This study evaluated LLMs as zero-shot binary classifiers for screening abstracts for a scoping review. The review consisted of 1,169 abstracts on LGBTIQ inclusion in the Philippines screened by three human experts. A prompt-based simulation design was implemented, with 2,300 records sampled with replacement per scenario across ten configurations: a benchmark, a model comparison across five models from an early snapshot of the GPT-5 family (GPT-5, GPT-5.1, GPT-5.2, GPT-5-mini, and GPT-5-nano), a reasoning effort comparison evaluating GPT-5.2 across three effort levels (low, medium, and high), and a prompt design variant evaluating whether removing the uncertain output category and enforcing a strictly binary classification constraint alters screening performance. Performance was evaluated using accuracy, precision, recall, and F1-score. Findings inform whether LLM-assisted screening achieves performance suitable for integration into evidence synthesis workflows across model architectures, reasoning mode configurations, and prompt output granularity. |
| Degree Course | Professional Master in Data Science (Analytics) |
| Language | English |
| Keyword | large language models, scoping review, abstract screening, zero-shot classification, evidence synthesis |
| Material Type | Thesis/Dissertation |
Preliminary Pages
5.61 Mb
Category : F - Regular work, i.e., it has no patentable invention or creation, the author does not wish for personal publication, there is no confidential information.
Access Permission : Open Access