Real Time Translation and Emotional Intelligent Voice Model

Abstract

Current speech-to-speech translation systems often struggle to capture the original speaker’s vocal identity and emotional tone, resulting in robotic and unnatural conversations. To address this, we introduce the Real Time Translation and Emotional Intelligent Voice Model. Our system fuses automatic speech recognition (ASR) and neural machine translation (NMT) with a hybrid emotion detection mechanism that looks at both how a person sounds and what their words mean. By leveraging the zero-shot voice cloning capabilities of XTTS v2, our model generates translated speech that sounds like the original speaker and dynamically shifts its prosody based on their current emotional state. We built the system using a responsive Flutter mobile interface and a PyTorch-accelerated FastAPI backend. Experimental testing shows our approach significantly improves Mean Opinion Scores (MOS) for naturalness and emotion retention when compared to traditional, cascaded translation setups.

Introduction

The text describes a real-time speech-to-speech (S2S) translation system designed to improve over traditional pipelines that convert speech → text → translation → speech. While conventional systems produce accurate translations, they often sound robotic and fail to preserve the speaker’s voice, emotion, and prosody.

To address this, the proposed system focuses on end-to-end, emotionally intelligent translation that preserves voice identity and emotional tone. It uses modern deep learning models such as Translatotron-style architectures and neural speech synthesis systems to directly or efficiently convert speech between languages while maintaining natural expressiveness.

A key feature is hybrid emotion recognition, which combines:

• Acoustic analysis (voice tone, pitch, rhythm using wav2vec 2.0)
• Semantic analysis (meaning of text using DistilRoBERTa)

These are fused to determine the final emotional state, which then guides speech synthesis so the output reflects both the correct language and the speaker’s emotion.

The system is built using a modular architecture:

• Backend: FastAPI with GPU-accelerated AI models for speech cleaning, transcription (ASR), machine translation, emotion detection, and voice synthesis.
• Frontend: Flutter mobile app with real-time waveform visualization, audio capture, and playback features.

For voice generation, it uses zero-shot voice cloning (XTTS v2), which can replicate a speaker’s voice using only a few seconds of input audio, enabling natural multilingual speech output without retraining.

Conclusion

Our Real Time Translation and Emotional Intelligent Voice Model showcases what is possible when you merge modern transformer architectures with affective computing. By combining zero-shot cross-lingual cloning with our bimodal emotion fusion, the system completely outpaces traditional cascading translators in terms of empathy and realism. It delivers a highly natural, low-latency communication experience that preserves both who the speaker is and how they are feeling. Additionally, splitting the architecture between Flutter and FastAPI creates a scalable, easily deployable framework. While a 2-second delay works fine for turn-based chat, our future goal is to push this into continuous streaming territory. If we can transition the XTTS v2 model from a chunk-based generator to a true token-streaming setup, we believe we can drop the perceived latency below 500ms. Moving forward, we also plan to experiment with model quantization (like 4-bit integer quantization) to reduce the heavy VRAM requirements, hopefully allowing the backend to run entirely on edge devices rather than relying on dedicated cloud GPUs.

References

[1] S. Popuri, K. Vaswani, and J. Li, “End-to-End Speech-to-Speech Translation with Latency Control,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 32, no. 4, pp. 1234–1247, 2024. [2] H. Zhang, X. Chen, and J. Yao, “Translatotron 3: Unsupervised Direct Speech-to-Speech Translation from Monolingual Speech–Text Datasets,” IEEE Signal Processing Letters, vol. 31, pp. 215–219, 2024. [3] T. Kano, C. Lu, and S. Nakamura, “Hibiki: A Decoder-Only Model for Simultaneous Speech Translation,” in Proceedings of Interspeech 2024, pp. 2158–2162. [4] Y. Li, X. Huang, and T. Fang, “Emotional Intelligence Multi-Lingual Voice Translation Model,” Irish Interdisciplinary Journal of Science and Research, vol. 8, no. 3, pp. 72–84, 2024. [5] S. Kim, P. Wang, and M. Lee, “Real-Time Speech Translation between Indian Languages Using Transformer-based Architecture,” in International Conference on Computational Linguistics (COLING), 2023. [6] L. Wang and J. Su, “Simultaneous Speech-to-Speech Translation with Reinforcement Learning Policies,” IEEE Transactions on Neural Networks and Learning Systems, vol. 35, no. 2, pp. 910–923, 2024. [7] Y. Tanaka, N. Kato, and H. Miyazaki, “A Comprehensive Review on End-to-End Speech-to-Speech Translation Systems,” ACM Computing Surveys, vol. 56, no. 4, Article 85, 2025. [8] K. Xu, J. Han, and Y. Luo, “Emotional Voice Conversion Using Conditional Variational Autoencoders,” Speech Communication, vol. 158, pp. 112–125, 2022. [9] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations,” in Advances in Neural Information Processing Systems (NeurIPS), 2020. [10] V. Sanh, L. Debut, J. Chaumond, and T. Wolf, “DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter,” arXiv preprint arXiv:1910.01108, 2019. [11] M. Chen and K. Li, “Emotional Intensity-Aware Network (EINet) for Controllable Speech Emotion Conversion,” IEEE Transactions on Affective Computing, 2024. [12] E. Casanova et al., “XTTS: a Massively Multilingual Zero-Shot Textto-Speech Model,” Coqui AI Technical Report, 2024. [13] J. Lee, D. Cho, and C. Park, “ClapFM-EVC: Flexible Emotional Voice Conversion Driven by Language Prompts,” Neural Processing Letters, vol. 56, pp. 893–908, 2024. [14] D. Park and M. Kim, “Streaming Speech Translation with Adaptive Wait-K Policy,” in ICASSP 2023, pp. 135–139, 2023. [15] A. Ashraf, B. M. S., H. T. B., N. C. H., and L. Prakash, “A Review On Real Time Translation And Emotional Intelligent Voice Model,” Preprint / Under Review, 2024. [16] E. Salesky et al., “The IWSLT 2021 Evaluation Campaign,” in Proceedings of the 18th International Conference on Spoken Language Translation (IWSLT), 2021. [17] A. Vaswani et al., “Attention is All you Need,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 30, 2017. [18] S. Poria, E. Cambria, R. Bajpai, and A. Hussain, “A review of affective computing: From unimodal analysis to multimodal fusion,” Information Fusion, vol. 37, pp. 98–125, 2017. [19] A. Radford et al., “Robust Speech Recognition via Large-Scale Weak Supervision,” Proceedings of the 39th International Conference on Machine Learning (ICML), 2022. [20] T. Kudo and J. Richardson, “SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing,” arXiv preprint arXiv:1808.06226, 2018.

Copyright

Copyright © 2026 Nooh C. H., Anan Ashraf, Basil M. S., Haneena T. B., Liya Prakash. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.