Multi-Modal Fusion Techniques for Improved Diagnosis in Medical Imaging

Abstract

Identifying diverse disease states is crucial for prompt and efficient clinical management. Complementary data from many medical imaging modalities, including MRI, CT, and PET, can be integrated to improve diagnostic performance. This work aims to assess how well multi-modal fusion methods work to enhance medical picture diagnosis. A multicenter study was conducted with 150 patients with different clinical conditions (mean age 58.2 ± 12.4 years, 52% female). After gathering data from MRI, CT, and PET scans, structural, functional, and textural characteristics were removed from each modality. The three fusion strategies studied were fusion through concatenation, fusion through kernels, and fusion through attention. The fused features were used to train classification models such as Convolutional Neural Networks (CNNs), ensemble techniques, and Support Vector Machines (SVMs). ROC analysis was utilized to assess the diagnostic performance. The multi-modal fusion techniques outperformed the single-modality methods in diagnosing performance. Attention-based fusion yielded the top AUCs of 0.92, 0.89, and 0.91 for brain tumors, neurodegenerative diseases, and cardiovascular conditions, respectively. This significantly improved (p<0.05) compared to the AUC of the best single-modality models. Multi-modal fusion methods are powerful for combining data from various imaging modalities to improve diagnostic accuracy for various medical conditions. These findings highlight the advantages of combining information sources to improve clinical judgment and patient care.