A Multimodal AI-Based Healthcare System for Pneumonia and Tuberculosis Detection Using Clinical Data and Chest X-Ray Analysis

Abstract

Pneumonia and tuberculosis (TB) together account for millions of deaths every year, and the problem is especially severe in places where access to quality diagnostics is limited. Our paper presents a multimodal AI-based healthcare system that brings together three independent diagnostic modules a custom CNN for pneumonia detection from chest X-rays, a MobileNetV2 based model for TB screening, and a Random Forest classifier for cardiovascular risk assessment using structured clinical data. Each module operates separately and produces its own confidence-based output. Grad-CAM is used to generate visual explanations for the image-based predictions, making the system more transparent. The whole thing is packaged as a Flask web application with user authentication, real-time analytics, and automated PDF report generation. In terms of results, the Random Forest model performed best with 97.11% accuracy and an AUC of 0.9967. The CNN for pneumonia also did well, reaching 87.98% accuracy and an AUC of 0.9454. The TB model, however, showed weaker performance (AUC of 0.6622), which we traced back to a dataset mismatch the model was trained on TB-specific images but evaluated on a general chest X-ray set. Overall, the system provides a scalable and explainable framework for AI-assisted multi-disease screening.

Introduction

This work presents a multimodal AI-based medical diagnostic system designed to improve early detection of respiratory diseases such as pneumonia and tuberculosis (TB), along with cardiovascular disease, by combining medical imaging and clinical data analysis. The motivation stems from real-world healthcare challenges, including limited access to radiologists, diagnostic delays, and the limitations of existing AI systems that typically focus on only a single disease or data type.

The system integrates deep learning for chest X-ray analysis (CNN and MobileNetV2) with a Random Forest model for structured clinical data. Pneumonia detection is handled by a custom four-block CNN, while TB detection uses transfer learning with MobileNetV2. Cardiovascular risk prediction is performed using ten clinical parameters from the UCI Heart Disease dataset. All models operate independently but are combined at the interface level to provide a unified diagnostic view.

A key feature of the system is explainability through Grad-CAM, which generates heatmaps showing which regions of an X-ray influenced the model’s decision, improving transparency and clinical trust. The system also automatically generates diagnostic reports using ReportLab and provides a Flask-based web application for image upload, form-based input, and real-time result visualization.

The literature review highlights that most existing systems either rely on image data or clinical data alone, while few integrate both in a practical, interpretable way. The proposed approach addresses this gap using a simple but effective multimodal design where outputs are presented together rather than fused into a single complex model.

Data preprocessing includes resizing and normalization of X-ray images, augmentation during training, and one-hot encoding of clinical features. The datasets used include the Kaggle Chest X-ray dataset, a TB-specific dataset, and the UCI Heart Disease dataset.

Experimental results show strong performance, with the pneumonia model achieving 87.98% accuracy, the cardiovascular model reaching 97.11% accuracy, and effective TB classification through transfer learning. Overall, the system demonstrates that combining image-based deep learning, structured data models, and explainable AI can produce a practical and deployable clinical decision-support tool that enhances diagnostic efficiency and interpretability.

Conclusion

This project set out to build a practical, multi-disease AI diagnostic system that integrates both image data and clinical parameters, explains its decisions visually, and is accessible enough to deploy without specialized infrastructure. For two of the three diagnostic modules, those goals were achieved. The Random Forest cardiovascular model performs excellently on structured clinical data, and the custom CNN demonstrates clinically meaningful sensitivity for pneumonia detection. The TB module is an honest limitation. Its poor evaluation scores are not a model failure per sethey are a data pipeline failure. The mismatch between training data and test data produced misleading results, and this needed to be acknowledged clearly rather than glossed over. That kind of transparency is part of what makes AI systems trustworthy in medical contexts. What this work demonstrates most clearly is that a modular multimodal architecture is a viable and practical design choice for healthcare AI. Each component can be developed, evaluated, and updated independently. The system as a whole is more useful than any single module would be in isolation. The Flask-based deployment keeps things simple without sacrificing functionality, and Grad-CAM makes the predictions interpretable enough for clinical review. There is clearly more work to be done particularly on the TB module but the foundation built here is solid. The architecture is extensible, the codebase is clean, and the system has already proven two out of three modules work reliably. That is a meaningful step toward AI-assisted multi-disease screening that is actually deployable in the real world.

References

[1] World Health Organization, \"Global Health Estimates 2020: Deaths by Cause, Age, Sex, by Country and by Region, 2000–2019,\" WHO, Geneva, 2020. [2] P. Patel et al., \"Radiologist shortage in sub-Saharan Africa: A systematic review,\" EClinicalMedicine, vol. 46, 2022, doi: 10.1016/j.eclinm.2022.101338. [3] Z. Che et al., \"Recurrent neural networks for multivariate time series with missing values,\" Sci. Rep., vol. 8, no. 1, Apr. 2018, doi: 10.1038/s41598-018-24271-9. [4] P. Rajpurkar et al., \"CheXNet: Radiologist-level pneumonia detection on chest X-rays with deep learning,\" arXiv:1711.05225, 2017. [5] G. Aceto, V. Persico, and A. Pescapé, \"A survey on deep learning in medicine: Why, how and when?,\" Inf. Fusion, vol. 66, pp. 111–137, Feb. 2021, doi: 10.1016/j.inffus.2020.09.006. [6] S. Hochreiter and J. Schmidhuber, \"Long short-term memory,\" Neural Comput., vol. 9, no. 8, pp. 1735–1780, Nov. 1997, doi: 10.1162/neco.1997.9.8.1735. [7] Y. Hannun et al., \"Cardiologist-level arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network,\" Nature Medicine, vol. 25, pp. 65–69, 2019, doi: 10.1038/s41591-018-0268-3. [8] F. Lahat, T. Adali, and C. Jutten, \"Multimodal data fusion: An overview of methods, challenges, and prospects,\" Proc. IEEE, vol. 103, no. 9, pp. 1449–1477, Sep. 2015, doi: 10.1109/JPROC.2015.2460697. [9] M. Lauritsen et al., \"Explainable artificial intelligence model to predict acute critical illness from electronic health records,\" Nature Commun., vol. 11, no. 1, p. 3852, 2020, doi: 10.1038/s41467-020-17431-x. [10] K. Simonyan and A. Zisserman, \"Very deep convolutional networks for large-scale image recognition,\" in Proc. ICLR, San Diego, CA, USA, May 2015. [11] T. Rahman et al., \"Exploring the effect of image enhancement techniques on COVID-19 detection using chest X-rays,\" Comput. Biol. Med., vol. 132, May 2021, doi: 10.1016/j.compbiomed.2021.104319. [12] R. R. Selvaraju et al., \"Grad-CAM: Visual explanations from deep networks via gradient-based localization,\" Int. J. Comput. Vis., vol. 128, no. 2, pp. 336–359, Feb. 2020, doi: 10.1007/s11263-019-01228-7. [13] V. Nair et al., \"Exploring uncertainty measures in deep networks for multiple sclerosis lesion detection and segmentation,\" Med. Image Anal., vol. 59, p. 101557, Jan. 2020, doi: 10.1016/j.media.2019.101557. [14] G. Hinton, O. Vinyals, and J. Dean, \"Distilling the knowledge in a neural network,\" arXiv:1503.02531, 2015.

Copyright

Copyright © 2026 Allikanti Saikiran, Gaikwad Pranay, G. Sridhar Goud, Mr. Shaik Rasool. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.