이정은 (충남대학교 의과대학, 충남대학교병원) | 한빛사논문 > 한빛사

한빛사논문

조회501

이정은Jeong Eun Lee

충남대학교 의과대학, 충남대학교병원

Med. Image Anal. , Nov 07 2023, 91 S1361-8415(23)00281-5 | https://doi.org/10.1016/j.media.2023.103021 Self-supervised multi-modal training from uncurated images and reports enables monitoring AI in radiology

Sangjoon Park ^a, Eun Sun Lee ^b, Kyung Sook Shin ^c, Jeong Eun Lee ^c, Jong Chul Ye ^d

^aDepartment of Radiation Oncology, Yonsei College of Medicine, Seoul, Republic of Korea

^bChung-Ang University Hospital, Seoul, Republic of Korea

^cDepartment of Radiology, Chungnam National University Hospital, Chungnam National University College of Medicine, Daejeon, Republic of Korea

^dKim Jaechul Graduate School of AI, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea

Corresponding authors : Eun Sun Lee, Jeong Eun Lee, Jong Chul Ye

Abstract

The escalating demand for artificial intelligence (AI) systems that can monitor and supervise human errors and abnormalities in healthcare presents unique challenges. Recent advances in vision-language models reveal the challenges of monitoring AI by understanding both visual and textual concepts and their semantic correspondences. However, there has been limited success in the application of vision-language models in the medical domain. Current vision-language models and learning strategies for photographic images and captions call for a web-scale data corpus of image and text pairs which is not often feasible in the medical domain. To address this, we present a model named medical cross-attention vision-language model (Medical X-VL), which leverages key components to be tailored for the medical domain. The model is based on the following components: self-supervised unimodal models in medical domain and a fusion encoder to bridge them, momentum distillation, sentencewise contrastive learning for medical reports, and sentence similarity-adjusted hard negative mining. We experimentally demonstrated that our model enables various zero-shot tasks for monitoring AI, ranging from the zero-shot classification to zero-shot error correction. Our model outperformed current state-of-the-art models in two medical image datasets, suggesting a novel clinical application of our monitoring AI model to alleviate human errors. Our method demonstrates a more specialized capacity for fine-grained understanding, which presents a distinct advantage particularly applicable to the medical domain.

논문정보

형식Research article
게재일2023년 11월 (BRIC 등록일 2023-11-14)
연구진국내 연구진
분야 바이오·의료융합 > 의료기기

댓글 작성 로그인

CV 열람하기

연락처 보기