AI-Powered Medical Assistant with peech to Vision Fusion

Abstract

The convergence of artificial intelligence with healthcare systems has significantly enhanced the delivery of accessible and intelligent medical services. This paper proposes a novel AI-powered medical assistant integrating both voice recognition and image processing to replicate a real-time doctor-patient interaction. Utilizing speech-to-text systems and multimodal transformers, the assistant interprets voice queries and clinical visuals such as ECGs to generate diagnostic feedback. The response is further synthesized into speech for interactive communication. The system is modular, efficient, and operates in real-time through a web-based Gradio interface, offering substantial utility for telemedicine and health education applications.

Introduction

The paper presents an AI-powered multimodal healthcare assistant (AI Doctor) that combines speech recognition, medical image analysis, and natural language generation to simulate a realistic doctor–patient consultation. The system allows users to speak their symptoms and upload medical images (such as ECGs or X-rays). It uses Whisper for speech-to-text transcription, LLaMA-based large language models for medical reasoning, vision-language models for image interpretation, and gTTS/ElevenLabs for generating natural spoken responses. The entire workflow is implemented as an interactive Gradio web application.

The study highlights that existing medical chatbots are largely rule-based, lack contextual understanding, and generally cannot process both voice and images simultaneously. The proposed framework overcomes these limitations by integrating multimodal inputs, enabling context-aware diagnosis and natural conversational interaction.

The system workflow includes:

• Collecting voice queries and medical images.
• Transcribing speech using Whisper.
• Encoding and analyzing medical images.
• Generating a diagnosis with a LLaMA-based language model.
• Converting the response into natural speech using text-to-speech.
• Displaying the transcription, diagnosis, and audio response to the user.

Experimental demonstrations showed that the AI Doctor could successfully interpret common symptom descriptions and relevant medical images, generating clinically relevant, grammatically correct, and empathetic responses in less than 10 seconds.

Conclusion

This paper presents a multimodal AI-driven system designed to simulate a medical consultation using voice input, visual data, and human-like output. By integrating state-of-the-art speech-to-text transcription, medical image analysis via foundation models, and realistic text-to-speech synthesis, the system effectively mimics the behavior of a virtual doctor. The proposed approach offers a seamless pipeline that transforms spoken patient queries into intelligent diagnostic responses, optionally enhanced through the interpretation of uploaded images like ECGs or scans. Our modular design—consisting of audio transcription, vision-based analysis, and AI-generated dialogue—has proven effective in producing timely, medically relevant, and empathetic responses. With potential applications in telemedicine, health education, and rural diagnostics, this project sets the foundation for more immersive and accessible AI-assisted healthcare platforms. Future work will focus on improving medical accuracy through fine-tuning and expanding multilingual support for broader accessibility.

References

[1] Abdelrahman Mohamed, ‘Dimitri Kanevsky’, ‘Audrey G.’, “Whisper: Robust Speech Recognition via Large-Scale Weak Supervision,” in OpenAI Blog, 2022. [2] Ronald J. Williams and David Zipser, “A Learning Algorithm for Continually Running Fully Recurrent Neural Networks,” Neural Computation, 1989. [3] ‘Google Text-to-Speech API (gTTS)’ Documentation: https://pypi.org/project/gTTS/ [4] ‘Pydub’ Audio Processing Library: https://github.com/jiaaro/pydub [5] ‘‘Simpleaudio’ Cross-Platform Audio Library for Python: https://simpleaudio.readthedocs.io/en/latest/ [6] ‘Gradio’ Interface Documentation: https://www.gradio.app/docs/ [7] ElevenLabs API Documentation: https://docs.elevenlabs.io/ [8] Meta AI, “LLaMA 2: Open Foundation and Fine-Tuned Chat Models,” Meta AI Research Blog, 2023. [9] OpenAI, “Multimodal Capabilities of GPT Models,” OpenAI Technical Report, 2024. [10] Hugging Face Transformers: https://huggingface.co/docs/transformers

Copyright

Copyright © 2026 Aayush ., Manju B H. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.