Resi-View 360: Multimodal AI Triage System

Abstract

Introduction

Respi-View 360: Multimodal AI-Assisted Respiratory Disease Assessment System is a healthcare decision-support project that uses Artificial Intelligence (AI), Machine Learning (ML), and Deep Learning (DL) to improve the early detection and assessment of respiratory diseases. The system integrates Chest X-Ray (CXR) images, respiratory audio recordings (cough and breathing sounds), and patient clinical history to generate comprehensive, explainable disease assessments.

Respiratory diseases such as pneumonia, bronchitis, asthma, COPD, and tuberculosis are major global health concerns. Traditional diagnosis depends on skilled radiologists, pulmonologists, laboratory tests, and specialized equipment, which are often unavailable in rural or resource-limited areas. Existing AI systems typically focus on only one type of medical data, reducing diagnostic accuracy.

To address these challenges, Respi-View 360 employs advanced foundation models, including Google’s Health Acoustic Representations (HeAR) for respiratory sound analysis and MedGemma for medical reasoning and Chest X-Ray interpretation. The system combines multiple data sources to provide preliminary disease assessment, severity evaluation, emergency risk alerts, and explainable clinical reports, supporting healthcare professionals rather than replacing them.

Key Objectives

• Develop a multimodal AI system combining Chest X-Ray analysis, respiratory audio processing, and symptom evaluation.
• Use advanced deep learning and foundation models for accurate disease assessment.
• Generate explainable AI reports with clinical findings, severity, confidence scores, and recommendations.
• Create a user-friendly web application for uploading patient data and viewing assessment reports.
• Implement an emergency risk alert system for critical respiratory conditions.

Scope

The project includes:

• Chest X-Ray analysis for detecting lung abnormalities.
• Respiratory sound analysis using HeAR.
• Integration of patient symptoms and medical history.
• Explainable AI reporting with downloadable PDF reports.
• A Gradio-based web interface.
• Emergency risk detection for severe cases.

The system is intended as a clinical decision-support tool for preliminary screening and does not replace professional medical diagnosis.

Project Plan

The project follows a structured 12-week timeline covering:

1. Requirement analysis and literature review.
2. Technology selection and system design.
3. Data preprocessing.
4. AI model integration.
5. Multimodal reasoning development.
6. Explainable AI and risk assessment.
7. Dashboard development.
8. Testing, optimization, documentation, and deployment.

Literature Survey

The literature review examines existing AI applications in healthcare, including computer vision, acoustic signal processing, multimodal AI, and explainable AI for respiratory disease diagnosis. It identifies the limitations of single-modal systems and highlights the need for integrated multimodal approaches, providing the foundation for developing Respi-View 360.

Conclusion

The Respi-View 360 Multimodal AI Respiratory Disease Assessment System was successfully developed by integrating respiratory audio analysis, chest X-ray interpretation, patient clinical history evaluation, and AI-assisted report generation into a unified platform. The system utilizes Google\'s HeAR model for extracting respiratory acoustic biomarkers and MedGemma for chest X-ray analysis and clinical reasoning. The developed Gradio-based dashboard provides an interactive environment for uploading respiratory audio recordings and chest X-ray images, entering patient information, generating respiratory disease assessments, and downloading structured PDF reports. The implementation of image validation, emergency risk alerts, report history management, and explainable assessment features further enhances the reliability and usability of the system. The experimental results demonstrate that the proposed multimodal approach can effectively combine information from multiple medical sources to generate meaningful respiratory disease assessment reports. The system provides a scalable and user-friendly AI-assisted healthcare solution while maintaining transparency through clinical explanations and recommendations. Overall, the project successfully achieves its objectives and demonstrates the potential of multimodal artificial intelligence in supporting respiratory disease assessment and clinical decision support.

Copyright

Copyright © 2026 Pallavi Marbe, Laxmi Sanganabasappa Soddi , Prajwal , Rohit , Prof. Anupama Pattanasetty . This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.