Agreement between Emergency Medicine Residents and Radiologists in RSNA-Based Chest CT Interpretation

Abstract

Background Standardized interpretation of chest computed tomography (CT) is essential for clinical decision-making in patients presenting to the emergency department with suspected coronavirus disease 2019 (COVID-19). Objectives This study evaluated interobserver agreement between emergency medicine residents (EMRs) and radiologists using the Radiological Society of North America (RSNA) classification and examined the effect of training level. Methods This retrospective single-center study included 1,104 patients with suspected COVID-19 who underwent chest CT between December 1, 2020, and January 1, 2021. CT findings were classified as typical, indeterminate, atypical, or negative for pneumonia according to RSNA criteria. A total of 23 EMRs independently evaluated 48 CT examinations each. Resident interpretations were compared with routine radiology reports. Interobserver agreement among radiologists was additionally assessed by independent re-evaluation of 100 CT scans. Agreement was measured using Cohen's kappa (kappa) coefficient. Diagnostic performance of the typical category was assessed using real time polymerase chain reaction (RT-PCR) as the reference standard. Results The mean patient age was 59.2 years, and 50.9% were male. RT-PCR positivity was 79.1%. Overall agreement between EMRs and radiologists was good (kappa = 0.705; 95% CI, 0.671-0.740; p < 0.001). Agreement was very good for typical (kappa = 0.827; 95% CI, 0.793-0.859) and negative (kappa = 0.804; 95% CI, 0.765-0.847) categories, but lower for indeterminate (kappa = 0.340; 95% CI, 0.253-0.428) and atypical (kappa = 0.341; 95% CI, 0.216-0.456) categories. Agreement among radiologists was also good (kappa = 0.730; 95% CI, 0.610-0.840; p < 0.001). Agreement improved with increasing training level. Conclusion RSNA-based chest CT classification shows good overall agreement. Lower agreement in indeterminate and atypical patterns highlights the need for targeted training and closer radiology collaboration.