Cross-Modal Deep Learning Framework for Multi-Disease Risk Assessment Using Clinical Data and Medical Images

Abstract

Cardiovascular disease, diabetes mellitus, and chronic kidney disease constitute the cardiometabolic triad — three conditions that actively amplify one another through established clinical pathways including diabetic nephropathy, cardiorenal syndrome, and cardiovascular-metabolic disease progression. Existing clinical decision support systems assess these conditions in isolation, failing to model the compounded risk faced by comorbid patients. This paper presents MedFuse Lite, a comorbidity-aware multi-disease risk prediction system integrating five heterogeneous model branches - three XGBoost tabular classifiers, an EfficientNet-B0 ECG classifier, and a ResNet50 diabetic retinopathy detector - with a Comorbidity Fusion Network, a PyTorch multi-layer perceptron trained on 5,856 real NHANES 2017–2018 patients. The fusion network accepts a 15-dimensional input vector comprising individual model probabilities, shared clinical biomarkers, and multiplicative comorbidity interaction terms, producing comorbidity-adjusted risk estimates for all three diseases simultaneously. The heart disease model achieved a test ROC-AUC of 79.93% on 68,633 cardiovascular records; the retinal model achieved validation AUC of 90.27%; and the fusion network converged to a best validation loss of 11.770%. The system provides SHAP feature attribution, GradCAM spatial explainability, disease correlation analysis, and rule-based clinical report generation, deployed as a Flask RESTful web application with a four-section sequential workflow.

Introduction

Chronic non-communicable diseases account for approximately 74% of global deaths, with cardiovascular disease (CVD), diabetes mellitus, and chronic kidney disease (CKD) forming a closely related comorbidity cluster. These diseases interact clinically—for example, diabetes can lead to diabetic nephropathy, CKD increases cardiovascular risk through cardiorenal syndrome, and high blood glucose accelerates atherosclerosis. Despite these well-established interactions, existing clinical risk assessment tools evaluate each disease independently without considering their combined effects.

To address this limitation, the study proposes MedFuse Lite, a multimodal, comorbidity-aware clinical decision support system that simultaneously predicts the risk of cardiovascular disease, diabetes, and chronic kidney disease. The framework integrates structured clinical data and medical imaging using machine learning and deep learning models. Disease-specific predictions are combined through a Comorbidity Fusion Network (CFN) trained on real NHANES population data to model interactions among the three diseases. The system also incorporates SHAP and Grad-CAM explainability techniques, disease interaction analysis, and automatically generated plain-English clinical reports aligned with international clinical guidelines.

Objectives and Contributions

The major contributions of MedFuse Lite include:

• Development of a heterogeneous multi-branch architecture capable of simultaneously assessing cardiovascular disease, diabetes, and chronic kidney disease.
• Design of a Comorbidity Fusion Network trained on real NHANES data to learn population-level interactions among multiple diseases.
• Integration of multimodal explainable AI using SHAP for tabular clinical models and Grad-CAM for retinal image interpretation.
• Generation of disease interaction alerts and evidence-based clinical reports to improve interpretability and support clinical decision-making.

Literature Review

Previous studies have demonstrated the benefits of machine learning and deep learning for disease prediction:

• Streamlit-based systems successfully combined prediction of multiple diseases but relied only on tabular data and lacked multimodal analysis, explainability, and comorbidity modeling.
• Deep learning models such as ResNet have achieved ophthalmologist-level performance in diabetic retinopathy detection from retinal images.
• Multi-disease deep learning frameworks have demonstrated high diagnostic accuracy but generally treated diseases independently without modeling interactions.
• Deep neural networks trained on ECG signals have achieved cardiologist-level arrhythmia detection, supporting the use of convolutional neural networks for cardiac diagnosis.

These studies provided the foundation for developing MedFuse Lite, which extends previous work by explicitly modeling disease interactions and integrating multimodal data.

Proposed System

MedFuse Lite is a comorbidity-aware clinical decision support framework designed for integrated prediction of cardiovascular disease, diabetes mellitus, and chronic kidney disease.

The system combines:

• Structured clinical parameters analyzed using XGBoost classifiers.
• ECG images analyzed using EfficientNet-B0.
• Retinal fundus images analyzed using ResNet50.

Outputs from these specialized models are fused using a Comorbidity Fusion Network, which models clinically relevant interactions among diseases and shared biomarkers. The framework also provides:

• SHAP feature attribution.
• Grad-CAM visual explanations.
• Disease correlation analysis.
• Rule-based clinical report generation.

System Architecture

The architecture consists of six sequential stages:

1. User Interface
   • Flask and Bootstrap-based web application for entering clinical information and uploading ECG or retinal images.
2. Input Acquisition
   • Validation and preprocessing of clinical parameters and optional medical images.
3. Disease Prediction Models
   • XGBoost models for heart disease, diabetes, and CKD.
   • EfficientNet-B0 for ECG image classification.
   • ResNet50 for diabetic retinopathy detection.
4. Individual Disease Assessment
   • Generation of probability scores for each disease.
5. Comorbidity-Aware Fusion
   • Integration of disease probabilities, shared biomarkers, and interaction terms using the Comorbidity Fusion Network.
6. Explainability and Clinical Insights
   • SHAP explanations.
   • Grad-CAM heatmaps.
   • Disease interaction summaries.
   • Plain-English clinical reports based on international guidelines.

Disease Prediction Models

Heart Disease

The heart disease branch combines:

• XGBoost classifier
  • Trained on 68,633 cardiovascular records.
  • Optimized using Optuna hyperparameter tuning.
  • Enhanced with engineered features such as BMI and pulse pressure.
  • Probability calibration performed using Platt scaling.
• EfficientNet-B0
  • Fine-tuned for ECG image classification into:
    • Normal
    • Abnormal heartbeat
    • Myocardial infarction
    • Post-myocardial infarction

The ECG abnormality probability complements tabular heart disease prediction.

Diabetes

The diabetes branch includes:

• XGBoost classifier
  • Trained on 100,000 patient records.
  • Optimized for precision-recall performance.
• ResNet50
  • Fine-tuned for diabetic retinopathy detection from retinal images.
  • Uses class-weighted loss to handle data imbalance.
  • Generates Grad-CAM activation maps for visual explanation.

Chronic Kidney Disease

The kidney disease model uses:

• XGBoost classifier
  • Trained on NHANES CKD dataset.
• Multiple Imputation by Chained Equations (MICE)
  • Handles missing clinical variables.
• Standardization using StandardScaler before prediction.

Datasets Used

Framework Deployment

The complete system is implemented as a Flask-based web application with a Bootstrap 5 frontend. All trained machine learning and deep learning models are loaded during startup to minimize prediction latency. Separate APIs are provided for each disease module and the Comorbidity Fusion Network, allowing modular yet integrated prediction.

Comorbidity Fusion Network (CFN)

The core innovation of MedFuse Lite is the Comorbidity Fusion Network, implemented as a five-layer PyTorch multilayer perceptron.

The network receives:

• Five disease prediction probabilities.
• Five normalized shared biomarkers:
  • Age
  • Systolic blood pressure
  • BMI
  • Serum creatinine
  • Albumin-to-creatinine ratio
• Diabetes diagnosis flag.
• Three clinically meaningful interaction terms representing:
  • Diabetic nephropathy
  • Elevated creatinine in diabetes
  • Cardiorenal syndrome

The network was trained using 5,856 NHANES patients for 150 epochs with Adam optimization and binary cross-entropy loss.

After prediction, a clinically motivated amplification layer increases risk estimates when multiple diseases coexist:

• 12% increase for diabetic nephropathy.
• 10% increase for cardiorenal syndrome.
• 8% increase for cardiovascular-metabolic interactions.

Risk values are constrained within predefined limits to maintain clinical credibility.

Explainability Module

To improve transparency, MedFuse Lite incorporates:

• SHAP TreeExplainer
  • Identifies the three most influential clinical features contributing to each prediction.
• Grad-CAM
  • Highlights important retinal regions responsible for diabetic retinopathy classification.
• Disease correlation analysis describing active comorbidity pathways.
• Rule-based clinical reports using recommendations from:
  • American Diabetes Association (ADA)
  • European Society of Cardiology (ESC)
  • KDIGO CKD guidelines

Results

The performance of individual models is summarized below:

The kidney disease model achieved the highest predictive performance, while both ECG and retinal image models demonstrated reliable classification accuracy, validating the effectiveness of multimodal disease prediction.

User Interface

The developed web application provides:

• Individual heart disease risk assessment.
• Diabetes prediction with Grad-CAM retinal visualization.
• Comorbidity-aware integrated risk assessment.
• Explainable AI outputs and clinical recommendations.

Conclusion

This paper presented MedFuse Lite, a comorbidity-aware multi-disease risk prediction system for the cardiometabolic triad. The system integrates five heterogeneous model branches with a Comorbidity Fusion Network trained on real NHANES population data, addressing the documented gap in clinical tools that assess cardiovascular, metabolic, and renal risk independently. Experimental results demonstrate clinically meaningful performance across all model branches, while the proposed Comorbidity Fusion Network effectively captures clinically relevant interactions among heart disease, diabetes, and chronic kidney disease. The complete explainability stack — SHAP attribution, GradCAM visualisation, disease correlation analysis, and guideline-sourced clinical reporting — makes the system interpretable and actionable in a clinical context. Future work will prioritise prospective clinical validation on real patient records, multi-centre dataset evaluation to assess demographic generalisation, and formal ablation studies comparing the NHANES-trained fusion network against rule-based and no-fusion baselines to precisely quantify the marginal value of the learned comorbidity adjustment

References

[1] T. Chen and C. Guestrin, “XGBoost: A Scalable Tree Boosting System,” in Proc. 22nd ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining (KDD), San Francisco, CA, USA, Aug. 2016, pp. 785–794. [2] S. F. Weng, J. Reps, J. Kai, J. M. Garibaldi, and N. Qureshi, “Can Machine Learning Improve Cardiovascular Risk Prediction Using Routine Clinical Data?” PLoS One, vol. 12, no. 4, Art. no. e0174944, Apr. 2017. [3] Q. Zou, K. Qu, Y. Luo, D. Yin, Y. Ju, and H. Tang, “Predicting Diabetes Mellitus With Machine Learning Techniques,” Front. Genet., vol. 9, Art. no. 515, Oct. 2018. [4] V. Gulshan et al., “Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs,” JAMA, vol. 316, no. 22, pp. 2402–2410, Dec. 2016. [5] A. Y. Hannun et al., “Cardiologist-Level Arrhythmia Detection and Classification in Ambulatory Electrocardiograms Using a Deep Neural Network,” Nat. Med., vol. 25, no. 1, pp. 65–69, Jan. 2019. [6] S. M. Lundberg and S. I. Lee, “A Unified Approach to Interpreting Model Predictions,” in Adv. Neural Inf. Process. Syst. (NeurIPS), vol. 30, 2017, pp. 4765–4774. [7] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-CAM: Visual Explanations From Deep Networks via Gradient-Based Localization,” in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), Venice, Italy, Oct. 2017, pp. 618–626. [8] K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Las Vegas, NV, USA, Jun. 2016, pp. 770–778. [9] M. Tan and Q. V. Le, “EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks,” in Proc. 36th Int. Conf. Mach. Learn. (ICML), Long Beach, CA, USA, Jun. 2019, pp. 6105–6114. [10] Centers for Disease Control and Prevention (CDC), National Center for Health Statistics, “National Health and Nutrition Examination Survey (NHANES), 2017–2018 and 2021–2023.” [11] American Diabetes Association, “Standards of Care in Diabetes—2023,” Diabetes Care, vol. 46, Suppl. 1, pp. S1–S291, Jan. 2023. [12] European Society of Cardiology, “2021 ESC Guidelines on Cardiovascular Disease Prevention in Clinical Practice,” Eur. Heart J., vol. 42, no. 34, pp. 3227–3337, 2021. [13] Kidney Disease: Improving Global Outcomes (KDIGO), “KDIGO 2022 Clinical Practice Guideline for Diabetes Management in Chronic Kidney Disease,” Kidney Int., vol. 102, no. 5S, pp. S1–S127, Nov. 2022. [14] World Health Organization, \"Noncommunicable Diseases,\" Fact Sheet, 2023. [15] C. Ronco, M. Haapio, A. A. House, N. Anavekar, and R. Bellomo, \"Cardiorenal Syndrome,\" J. Am. Coll. Cardiol., vol. 52, no. 19, pp. 1527–1539, Nov. 2008. [16] [16] R. B. D\'Agostino Sr., R. S. Vasan, M. J. Pencina, P. A. Wolf, M. Cobain, J. M. Massaro, and W. B. Kannel, \"General Cardiovascular Risk Profile for Use in Primary Care: The Framingham Heart Study,\" Circulation, vol. 117, no. 6, pp. 743–753, Feb. 2008. [17] J. Hippisley-Cox, C. Coupland, and P. Brindle, \"Development and Validation of QRISK3 Risk Prediction Algorithms to Estimate Future Risk of Cardiovascular Disease: Prospective Cohort Study,\" BMJ, vol. 357, p. j2099, May 2017. [18] A. S. Levey, L. A. Stevens, C. H. Schmid, Y. L. Zhang, A. F. Castro III, H. I. Feldman, J. W. Kusek, P. Eggers, F. Van Lente, T. Greene, and J. Coresh, \"A New Equation to Estimate Glomerular Filtration Rate,\" Ann. Intern. Med., vol. 150, no. 9, pp. 604–612, May 2009.

Copyright

Copyright © 2026 Gandla Sai Dheeraj Rao, N. Naveen Kumar . This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.