Interrater agreement in dementia diagnosis: A systematic review and meta-analysis.

Objectives:  Dementia remains a clinical diagnosis with a degree of subjective assessment and potential for interrater disagreement. We described interrater agreement of clinical dementia diagnosis for various diagnostic criteria. 

Methods:  We conducted a PROSPERO-registered (CRD42020168245) systematic review and meta-analysis. We searched multiple cross-disciplinary databases from inception until April 2020 for relevant papers, extracted data and described study quality in duplicate. Study quality was assessed using the Guidelines for Reporting Reliability and Agreement Studies. We used random-effects models to obtain summary estimates of interrater agreement using kappa and, where possible, Gwet's AC1/2 coefficients. 

Results: We found 7577 titles and 22 eligible studies. Meta-analysis was possible for all-cause dementia using the Diagnostic and Statistical Manual of Mental Disorders third edition revised (DSM-III-R) criteria (kappa = 0.66, 95% CI = [0.53,0.78]), Alzheimer's disease using the National Institute of Neurological and Communicative Disorders and Stroke and the Alzheimer's disease and Related Disorders Association (NINCDS-ADRDA) criteria (kappa = 0.71, 95% CI = [0.65,0.77] for presence/absence and AC2 = 0.61, 95% CI = [0.53,0.70] when distinguishing probable/possible cases), and vascular dementia using the International Classification of Diseases version 10 (ICD-10) criteria kappa = 0.79 (95% CI = [0.70,0.87]). Data was more limited for other criteria and dementia types. AC1/2 coefficients generally indicated higher agreement. One study was rated as high quality. 

Conclusions:  Diagnostic criteria for clinical dementia may have good but imperfect agreement. This has important implications for clinical practice and research studies, which frequently assume these criteria are perfect tests, such as diagnostic test accuracy studies frequently conducted for biomarkers and neuropsychological tests, and for trials where incident dementia is the outcome.