김남국 (울산대학교 의과대학/서울아산병원) | 한빛사논문 > 한빛사

한빛사논문

조회36

울산대학교 의과대학/서울아산병원

CV File

npj Digit. Med., 07 July 2026 | https://doi.org/10.1038/s41746-026-02978-8 Human-in-the-loop validation of a sequential multi-LLM medical education pipeline

Yoojin Nam^1,2; Taein An³; Sung Il Hwang⁴; Chawoong Jeon⁵; Ji Won Jeong⁶; Juhyun Jeong⁷; Dong Yeong Kim^2,8; Pyeong Hwa Kim⁸; So Yeon Kim⁸; Sungkwan Kim¹; Yangwon Kim¹; Kwang Hwi Lee⁹; Taehee Lee¹⁰; Hee Sang Oh¹¹; Joon Hyung Park^2,6; Seunghyeon Shin¹²; Yongsik Sim¹³; Jeong Min Song^2,8; Seunghyun Song¹¹; Pa Hong^1,*; Namkug Kim^2,8,*

¹Department of Radiology, Samsung Changwon Hospital, Sungkyunkwan University School of Medicine, Changwon, Republic of Korea
²Department of Convergence Medicine, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Republic of Korea
³Department of Radiology, Hallym University Dongtan Sacred Heart Hospital, Hwaseong, Republic of Korea
⁴Department of Radiology, Seoul National University Bundang Hospital, Seongnam, Republic of Korea
⁵Department of Radiology, Yeouido St. Mary’s Hospital, College of Medicine, The Catholic University of Korea, Seoul, Republic of Korea
⁶Department of Radiology, Kyung Hee University Hospital, Kyung Hee University College of Medicine, Seoul, Republic of Korea
⁷Department of Radiology, Ansan Hospital, Korea University School of Medicine, Ansan, Republic of Korea
⁸Department of Radiology and Research Institute of Radiology, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Republic of Korea
⁹Centum Breast and Thyroid Clinic, Busan, Republic of Korea
¹⁰Department of Radiology, Seoul National University Hospital, Seoul, Republic of Korea
¹¹Department of Radiology and the Research Institute of Radiological Science, Gangnam Severance Hospital, Yonsei University College of Medicine, Seoul, Republic of Korea
¹²Department of Nuclear Medicine, Samsung Changwon Hospital, Sungkyunkwan University School of Medicine, Changwon, Republic of Korea
¹³Department of Radiology, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Republic of Korea

^*Co-corresponding authors
Corresponding Authors: Pa Hong, Namkug Kim

Abstract

Large language models (LLMs) can generate medical educational content at scale, but multi-dimensional human validation of multi-LLM pipelines has not been performed at scale. We evaluated a 7-stage sequential multi-LLM pipeline implemented exclusively with the Gemini family (Gemini 3 Flash and Pro for text; Gemini 3 Pro Image, the Nano Banana Pro preview, for images), producing 6000 flashcards and 833 infographics for radiology board preparation. Nine residents and eleven attending radiologists evaluated 1284 flashcards across 11 subspecialties in a two-phase design around fifth-stage (S5) feedback. Among 1100 evaluations of 980 text-level PASS cards, the evaluation-level false-negative rate (FNR) for blocking errors was 1.00% (95% CI 0.50–1.78%), exceeding the pre-specified 0.3% safety threshold; card-level (1.12%) and majority-vote (0.82%) analyses converged. S5 feedback was associated with increased blocking-error flags (39–54 evaluation-level observations, McNemar p = 0.003) and decreased technical accuracy and educational quality scores (both p < 0.001). In unadjusted analyses, attending radiologists identified more errors than residents (odds ratio 4.52, 95% CI 2.38–8.57), although workload-matched sensitivity analyses attenuated this gap. A preliminary image pilot of 19 items showed critical errors in both tables and infographics, with between-format comparisons underpowered and not significant after Holm correction; this image-related observation should be interpreted as exploratory. The conditional safety estimates, low-reliability reference standard, and rater- and workload-dependent patterns do not support fully automated deployment; automated feedback was associated with sharpened evaluator scrutiny, though the single-arm design cannot isolate useful cueing from anchoring or re-review effects.

논문소개전체 설명

이 연구는 영상의학 전문의 시험용 플래시카드와 인포그래픽을 멀티모달 LLM(Gemini 3 Flash and Pro for text; Gemini 3 Pro Image, the Nano Banana Pro preview, for images)을 이용하여 대규모로 생성하여서, 이를 Human reader study 한 것 입니다. 9 명의 전공의와 11명의 전문의가 평가한 결과, evaluation-level false-negative rate (FNR) (1.00%)이 안전 기준치(0.3%)를 초과했으며 피드백 후 교육 품질 평가 점수가 하락했습니다. 따라서 안전성과 평가자별 불일치 등을 고려할 때, 완전 자동배포는 권장되지 않으며, 인간의 검증이 필요하다는 결론입니다.

논문정보