Design and Development of an AI-Based Voice-Controlled Robot for Intelligent Human-Robot Interaction

Abstract

Most low-cost voice-controlled robots accept only a handful of fixed words such as “forward” and “stop”, and depend on a smartphone app for speech recognition. This paper presents the design and development of an AI-based voice-controlled mobile robot that holds natural spoken conversations and turns free-form requests into safe movements. The robot is built around an ESP32 microcontroller with an INMP441 I²S MEMS microphone, a MAX98357A I²S class-D amplifier and speaker, an L298N dual H-bridge driving two DC gear motors with wheels, a servo for head and gesture movements, and a 2S pack of 18650 lithium-ion cells with a main switch. A hybrid architecture is used: a compact keyword-spotting model on the ESP32 detects the wake word and a local “stop” command without any network, while the spoken request is streamed to an AI server where speech-to-text, a large language model and text-to-speech produce a structured JSON action and a spoken reply. A command validator on the ESP32 checks every action against a whitelist, speed limits and a time-out before the motors move, so the language model can never drive the robot outside its safe envelope. A review of 19 related studies shows that earlier robots offered either fixed-vocabulary control on cheap hardware or natural-language interaction on expensive platforms, but rarely both. Design calculations show that a 4-second request needs 128 kB of audio, well within the ESP32\'s memory, and that the battery pack supports an estimated 13.4 h of mixed use. An experimental protocol with metrics for recognition accuracy, latency, safety response and user experience is provided for prototype validation.

Introduction

1. Introduction

Human–Robot Interaction (HRI) refers to the communication and cooperation between humans and robots. Voice communication is one of the most natural methods of interaction because it does not require physical controllers or screens and is useful for people of different ages and abilities.

Traditional voice-controlled robots generally use mobile applications and Bluetooth connections to operate through a limited set of commands. However, they cannot understand complex instructions or respond verbally. Modern AI-based robots use Large Language Models (LLMs) to understand natural language and communicate with humans, but they often require expensive hardware.

The proposed project develops a low-cost, AI-based voice-controlled robot that supports two-way communication using an ESP32 microcontroller and an AI server.

2. Objectives of the Project

The main objectives of the proposed system are:

1. To develop a voice-controlled robot using an ESP32, microphone, speaker, motor driver, DC motors, and servo motor.

2. To enable natural voice communication between humans and the robot.

3. To use AI and Large Language Models (LLMs) to understand spoken instructions.

4. To provide spoken responses through a speaker.

5. To implement safety features such as emergency voice-stop detection and automatic motor stopping.

6. To develop a low-cost robotic system with reliable performance and wireless connectivity.

3. Literature Review

Previous research on voice-controlled robots can be divided into four main categories:

• Human–Robot Interaction: Studies have explored how humans communicate and cooperate with robots through speech and other interaction methods.

• Low-Cost Voice-Controlled Robots: Arduino-based robots commonly use Bluetooth and Android applications to recognize predefined voice commands and control movement.

• On-Device Speech Recognition: Researchers have developed lightweight speech-recognition models that operate directly on microcontrollers such as ESP32.

• AI and LLM-Based Robot Control: Recent studies use Large Language Models to understand natural-language instructions and generate robotic actions.

Existing systems have limitations, including restricted voice commands, dependence on smartphones or internet connectivity, lack of verbal responses, and the possibility of unsafe actions generated by AI models.

4. Research Gap

The proposed project addresses four major limitations of existing systems:

1. Limited voice commands: Traditional robots recognize only predefined commands. The proposed system uses AI-based speech recognition and language understanding to process natural-language instructions.

2. Lack of verbal communication: Existing robots generally cannot respond to users. The proposed robot provides spoken responses through a speaker.

3. Safety issues: AI-generated commands may contain incorrect or undefined actions. The proposed system uses JSON validation, predefined action limits, and safety checks.

4. High hardware cost: Advanced AI robots often require expensive computing equipment. The proposed design uses affordable electronic modules and performs complex AI processing on an external server.

5. Proposed System Architecture

The proposed system consists of two main parts:

• Robot Unit: Captures voice commands, processes basic safety instructions, controls motors and the servo motor, and plays spoken responses.

• AI Server: Performs speech recognition, natural-language understanding, and text-to-speech conversion.

Both units communicate through a Wi-Fi connection using the WebSocket protocol.

The ESP32 divides its processing tasks between two cores:

• Core 0 handles audio capture and wake-word detection.

• Core 1 manages wireless communication, motor control, and safety operations.

6. Hardware Components

The main hardware components used in the project are:

• ESP32 Microcontroller: Acts as the main controller and manages communication, motor movement, and safety functions.

• INMP441 Microphone: Captures the user's voice commands.

• MAX98357A Amplifier: Amplifies audio signals to produce sound through the speaker.

• Speaker: Provides spoken responses from the robot.

• L298N Motor Driver: Controls the direction and movement of the DC motors.

• DC Motors and Wheels: Provide movement and navigation.

• Servo Motor: Performs additional mechanical movements.

• 18650 Rechargeable Batteries: Supply electrical power to the robot.

• Wi-Fi Connectivity: Enables communication between the robot and the AI server.

7. Working Principle

The robot operates through voice commands and AI-based processing.

1. The microphone captures the user's spoken instructions.

2. The ESP32 detects the wake word and emergency stop command.

3. The captured audio is transmitted to the AI server through Wi-Fi.

4. The server converts speech into text using speech recognition.

5. The Large Language Model understands the instruction and generates an appropriate response.

6. The AI server returns a JSON-formatted action and a spoken reply.

7. The ESP32 validates the action and controls the motors or servo motor accordingly.

8. The speaker plays the robot's verbal response.

The robot also includes an independent safety mechanism that stops the motors when a stop command is detected, a connection is lost, or a timeout occurs.

8. Safety Features

Safety is an important part of the proposed system. The following features are included:

• Voice-based emergency stopping.

• Validation of AI-generated commands.

• Restricted predefined robotic actions.

• Automatic motor stopping during communication failure.

• Timeout protection to prevent continuous unwanted movement.

• Local safety control independent of the AI server.

Conclusion

This paper presented the design of an AI-based voice-controlled robot built from an ESP32, an INMP441 microphone, a MAX98357A amplifier with speaker, an L298N driver with two DC motors and wheels, a servo and a 2S 18650 battery pack with a switch. A hybrid architecture keeps wake-word and stop detection on the robot and sends free-form requests to a server for speech recognition, language understanding and speech synthesis. A JSON action interface and an on-robot validator let the robot hold natural conversations while every movement stays within fixed safety limits. A review of 19 studies showed that this combination of low cost, natural interaction and enforced safety has not been addressed together before. Future work will complete the prototype measurements, add wheel encoders and an ultrasonic or time-of-flight sensor for obstacle avoidance, replace the L298N with a MOSFET driver, run speech recognition on an ESP32-S3 or a small local server for fully offline operation, and study long-term interaction with users such as children and elderly people.

References

[1] M. A. Goodrich and A. C. Schultz, “Human–robot interaction: A survey,” Foundations and Trends in Human–Computer Interaction, vol. 1, no. 3, pp. 203–275, 2008, doi: 10.1561/1100000005. [2] M. Marge et al., “Spoken language interaction with robots: Recommendations for future research,” Computer Speech & Language, vol. 71, Art. no. 101255, 2022, doi: 10.1016/j.csl.2021.101255. [3] S. P. Pleshkova, Z. Zahariev, and A. B. Bekiarski, “Development of speech recognition algorithm and LabVIEW model for voice commands control of mobile robot motion,” in Proc. International Conference on High Technology for Sustainable Development (HiTech), Sofia, Bulgaria, 2018. [4] A. Umar, M. A. Giwa, A. Y. Kassim, M. U. Ilyasu, I. Abdulwahab, E. E. Agbon, and M. T. Ogedengbe, “Development of an Android-based, voice-controlled autonomous robotic vehicle,” Engineering Proceedings, vol. 58, Art. no. 48, 2023, doi: 10.3390/ecsa-10-16026. [5] A. Azahar, K. Rosda, N. Mustafa, and A. Alias, “Development of voice command control prototype for vehicle mobility,” Enhanced Knowledge in Sciences and Technology, vol. 3, no. 2, pp. 250–254, 2023, doi: 10.30880/ekst.2023.03.02.029. [6] D. S. Pratama, Y. C. H. Yuwono, and M. R. Sufandi, “Prototipe mobil robot pemindah barang dengan voice control berbasis ESP32,” Jurnal LITEK: Jurnal Listrik Telekomunikasi Elektronika, vol. 22, no. 2, pp. 89–93, 2025, doi: 10.30811/litek.v22i2.76. [7] M. Bakouri, M. Alsehaimi, H. F. Ismail, K. Alshareef, A. Ganoun, A. Alqahtani, and Y. Alharbi, “Steering a robotic wheelchair based on voice recognition system using convolutional neural networks,” Electronics, vol. 11, no. 1, Art. no. 168, 2022, doi: 10.3390/electronics11010168. [8] J. Mendoza-Montoya, K. E. Angulo-Oviedo, E. Y. Turpo-Apaza, and R. Alfonte-Zapana, “Design of a prototype mechatronic wheelchair for people with motor disabilities,” in Proc. CITIE 2023: International Congress on Education and Technology in Sciences, CEUR Workshop Proceedings, vol. 3691, 2023. [9] O. E. Adetoyi and O. P. Folami, “Spoken command recognition system for hands-free door control,” Journal of Science, Technology and Innovation Research, vol. 1 (Special Issue), pp. 413–420, 2025, doi: 10.51459/jostir.2025.1.Special-Issue.067. [10] Y. Zhang, N. Suda, L. Lai, and V. Chandra, “Hello Edge: Keyword spotting on microcontrollers,” arXiv:1711.07128, 2017. [11] P. Warden, “Speech Commands: A dataset for limited-vocabulary speech recognition,” arXiv:1804.03209, 2018. [12] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” in Proc. 40th International Conference on Machine Learning (ICML), PMLR, vol. 202, pp. 28492–28518, 2023. [13] B. Ichter, A. Brohan, Y. Chebotar, C. Finn et al., “Do as I can, not as I say: Grounding language in robotic affordances,” in Proc. 6th Conference on Robot Learning (CoRL), PMLR, vol. 205, pp. 287–318, 2023. [14] J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng, “Code as Policies: Language model programs for embodied control,” in Proc. IEEE International Conference on Robotics and Automation (ICRA), 2023, pp. 9493–9500. [15] S. Vemprala, R. Bonatti, A. Bucker, and A. Kapoor, “ChatGPT for robotics: Design principles and model abilities,” Microsoft Research, Tech. Rep. MSR-TR-2023-8, 2023 (arXiv:2306.17582). [16] A. Koubaa, “ROSGPT: Next-generation human–robot interaction with ChatGPT and ROS,” Preprints, 2023, doi: 10.20944/preprints202304.0827.v1. [17] C. Y. Kim, C. P. Lee, and B. Mutlu, “Understanding large-language model (LLM)-powered human–robot interaction,” in Proc. ACM/IEEE International Conference on Human–Robot Interaction (HRI), 2024 (arXiv:2401.03217). [18] C. Wang, S. Hasler, D. Tanneberg, F. Ocker, F. Joublin, A. Ceravola, J. Deigmoeller, and M. Gienger, “LaMI: Large language models for multi-modal human–robot interaction,” in Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA ’24), Honolulu, HI, USA, 2024 (arXiv:2401.15174). [19] Y. Lakhnati, M. Pascher, and J. Gerken, “Exploring a GPT-based large language model for variable autonomy in a VR-based human–robot teaming simulation,” Frontiers in Robotics and AI, vol. 11, 2024, doi: 10.3389/frobt.2024.1347538. [20] Espressif Systems, “ESP32 series datasheet,” technical documentation. [21] InvenSense (TDK), “INMP441 omnidirectional microphone with bottom port and I²S digital output,” datasheet. [22] Analog Devices (Maxim Integrated), “MAX98357A/MAX98357B tiny, low-cost, PCM class D amplifier with class AB performance,” datasheet. [23] STMicroelectronics, “L298 dual full-bridge driver,” datasheet. [24] Philips Semiconductors, “I²S bus specification,” Feb. 1986, revised Jun. 1996. [25] I. Fette and A. Melnikov, “The WebSocket protocol,” IETF RFC 6455, Dec. 2011.

Copyright

Copyright © 2026 Rajnish Kumar, Md. Sahil, Akash Kumar, Sonu Kr. Chaudhary. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.