PROBAST+AI or QUADAS-3? Choosing the appropriate risk-of-bias assessment tool for systematic reviews of AI-based prediction models and diagnostic tests

Abstract

The rapid expansion of artificial intelligence (AI) and machine learning (ML) in healthcare has been accompanied by a growing number of diagnostic systematic reviews and meta-analyses. However, selecting an appropriate risk-of-bias framework remains challenging, particularly when studies combine prediction-model development and validation with diagnostic accuracy evaluation. The aim of this narrative review is to examine the respective scopes of PROBAST+AI and QUADAS-3 and develop practical guidance for selecting and applying these frameworks in systematic reviews of AI-based healthcare research. Two most recently updated appraisal tools used for AI-themed diagnostic systematic reviews were qualitatively assessed, namely PROBAST+AI and QUADAS-3. The assessment includes the scope of both frameworks, which was later used to synthesize practical guidance in selecting and applying them. PROBAST+AI is appropriate when the primary objective is to assess the methodological quality of prediction-model development or the risk of bias and applicability in model evaluation. In contrast, QUADAS-3 is appropriate when the objective is to assess the risk of bias and applicability of diagnostic accuracy estimates from a prespecified AI system evaluated against an appropriate reference standard. The proposed guidance provides a decision pathway for identifying the relevant unit of assessment, distinguishing prediction-model evaluation from diagnostic test-accuracy assessment, and determining when separate assessments using both frameworks are warranted. When a publication includes distinct model-development and diagnostic accuracy components, each component should be assessed using the framework corresponding to its methodological purpose. This guidance provides a conceptual basis for transparent and methodologically appropriate framework selection, although its proposed decision pathway has not yet been empirically validated.