CampusBuddy: A Multimodal NLP-Powered Chatbot for Malayalam Voice and Text-Based College Queries

Abstract

CampusBuddy is a chatbot powered by AI that helps Malayalam speakers talk and write to each other. The system uses Natural Language Processing technologies like OpenAI’s GPT-4o Mini for conversational intelligence, Whisper for recognizing Malayalam speech accurately, and Silero TTS for making speech sound real. The chatbot makes it easy and natural for students and staff to ask academic and administrative questions, and it is designed especially for Malayalam-speaking users in schools. The system operates using the following order: the system receives Malayalam speech input, utilises Whisper to convert it into English text, relies on Retrieval-Augmented Generation (RAG) with OpenAI’s model and ChromaDB to ensure the responses are relevant to the context, and finally is translated back into Malayalam and output as speech or text.CampusBuddy is built on Streamlit for scalability and supports multimodal interactions, giving users both voice and text outputs to make it easier to use.

Introduction

The paper presents a voice-based chatbot system designed specifically for the Malayalam language, addressing the lack of AI tools for regional languages. While most current systems like Alexa or Rasa X focus on widely spoken languages such as English, this project bridges the gap by providing a localized, scalable, and user-friendly solution for Malayalam-speaking users—particularly in educational settings.

Problem & Motivation:

• Existing voice chatbots lack support for regional languages, making them inaccessible to Malayalam speakers.

• Traditional speech recognition models (e.g., HMMs) struggle with regional accents.

• There is a need for multilingual, voice-enabled, and domain-specific AI assistants for applications such as college information systems.

Proposed Solution:

The project introduces a modular, multilingual voice chatbot using:

• OpenAI’s GPT-4o mini for natural language understanding.

• Whisper ASR for accurate Malayalam speech recognition.

• Silero TTS for natural-sounding Malayalam speech synthesis.

• RAG (Retrieval-Augmented Generation) with OpenAI embeddings and Chroma vector database for domain-specific (college-related) responses.

• Streamlit interface for voice-based interactions and conversation history with audio and text outputs.

The full voice interaction pipeline includes:

1. Malayalam speech input via Whisper.

2. Translation to English for query processing.

3. Answer generation via GPT-4o mini.

4. Translation back to Malayalam.

5. Text-to-speech synthesis with Silero for spoken output.

Key Features & Achievements:

• Real-time Malayalam voice interaction.

• Cross-lingual pipeline enables Malayalam input and output with English-based processing.

• Scalable and adaptable architecture with separate UI and backend layers.

• College-specific knowledge base using embedded data for accurate answers.

• User-friendly interface that requires no technical expertise.

Literature Review Highlights:

• Transformer architecture (e.g., BERT, GPT) is central to modern NLP.

• TTS systems like Tacotron and WaveNet enable accent-specific voice synthesis with limited data.

• NLU benchmarks show IBM Watson excels in intent classification but struggles with entity recognition.

• Chatbots like Alexis and Rasa X perform well but lack support for regional languages and scalability in educational environments.

• Other systems focus on healthcare or general college inquiries but remain limited to English.

Conclusion

CampusBuddy shows how AI-powered chatbots can help Malayalam speakers by using speech recognition, natural language processing, and text-to-speech technologies together. The system can understand voice queries in Malayalam, get the right information from a structured knowledge base, and give answers in both spoken and written forms. Its modular design makes it easy to add new features in the future, and its focus on making it easy to use in different languages sets the stage for AI applications in education that are open to everyone. Going forward, possible improvements could include better handling of background noise, support for regional dialects, and the ability to connect to real-time data sources. Using the chatbot for more than just education, such as in healthcare or government services, could have an even bigger effect on society. Also, using lighter versions for mobile devices would make it easier to get to in areas with poor internet access. These improvements would make AI help more useful and available to people who don’t speak English, making it easier for people to use technology.

References

[1] Sangeeta Kumari, Zaid Naikwadi, Akshay Akole, and Purushottam Darshankar. Enhancing college chat bot assistant with the help of richer human computer interaction and speech recog- nition. In Proceedings of the International Conference on Electronics and Sustainable Com- munication Systems (ICESC 2020). IEEE Xplore, 2020. [2] Meet Popat, Aayush Doshi, Yashraj Rai, Deep Vakharia, and Grinal Tuscano. Alexis: A voice- based chatbot using natural language processing. International Journal of Engineering Re- search & Technology (IJERT), 11(04), April 2022. [3] Nguyen TT, Le AD, Hoang HT, and Nguyen T. Neu-chatbot: Chatbot for admission of national economics university. Computers and Education: Artificial Intelligence, 2:100036, 2021. [4] G. S. Ramesh, G. Nagaraju, Vemula Harish, and P. Kumaraswamy. Chatbot for college web- site. In International Conference on Advances in Computer Engineering and Communication Systems, Learning and Analytics in Intelligent Systems. Springer Nature, 2021. [5] Praveen Prasannan, Stephy Joseph, and Rajeev R R. A chatbot in malayalam using hybrid ap- proach. In Proceedings of the 17th International Conference on Natural Language Processing (ICON): System Demonstrations, December 2020. [6] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017. [7] Xue Hao Zhou, Ming Yang Zhang, Zhizheng Wu, Yi Zhou, and Haizhou Li. Accented text-to- speech synthesis with limited data. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 32, 2024. [8] Xingkun Liu, Arash Eshghi, Pawel Swietojanski, and Verena Rieser. Benchmarking natural language understanding services for building conversational agents. In Increasing Naturalness and Flexibility in Spoken Dialogue Interaction. Springer Nature, 2021. [9] Akshada Phalle, Saniya Kadam, Sakshi Sonphule, and Ila Savant. Ai and web-based interac- tive college enquiry chatbot. International Research Journal of Engineering and Technology (IRJET), 8, November 2021. [10] Soufyane Ayanouz, Boudhir Anouar Abdelhakim, and Mohammed Benhmed. A smart chatbot architecture based on nlp and machine learning for health care assistance. In Proceedings of the 3rd International Conference on Networking, Information Systems & Security, 2020.

Copyright

Copyright © 2025 Rejimoan R, Gnanapriya B, Jayasudha J S. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.