Prototypic Hand Talk Assistive Technology for Physically Challenged

Abstract

The Flex Sensor–Based Hand Gesture Detection System is designed to assist deaf and dumb individuals in communicating their needs effectively with others This system provides a simple and efficient solution by converting hand gestures into meaningful commands that can be understood by nearby people or caregivers. In this system, flex sensors are used to detect the bending motion of the fingers. When a user performs a specific hand gesture, the flex sensors measure the change in resistance caused by finger movement. These signals are then sent to an Arduino microcontroller, which acts as the main processing unit of the system. The Arduino interprets the sensor data and identifies the corresponding gesture based on predefined commands Once the gesture is recognized, the system generates both text and voice outputs. The LCD display shows the message visually so that it can be read easily, while the voice module converts the detected gesture into a spoken message. For example, gestures can represent needs such as water, food, medicine, emergency help, or general assistance. The generated voice message is then played through a speaker, allowing nearby people, caregivers, or authorities to immediately understand the user’s request. This helps ensure quick response and timely support. Overall, this system improves communication, enhances independence, and provides a reliable assistive technology for deaf and dumb individuals in their daily lives.

Introduction

This work proposes an assistive communication and health-support system for people with disabilities, especially paralyzed, deaf, and dumb individuals, using IoT and flex-sensor-based gesture recognition.

The main problem addressed is the lack of continuous healthcare monitoring and effective communication tools for patients who cannot easily express their needs. Conventional hospital-based monitoring is insufficient for detecting sudden health issues, while communication barriers further increase risk for disabled users.

To solve this, the system integrates IoT sensors and an Arduino-based gesture glove. Health monitoring uses devices like an ADXL335 accelerometer (fall detection), MAX30100 (heart rate and SpO?), LM35 (temperature), and flex sensors, with an ESP32 microcontroller for data processing and real-time transmission to caregivers. This enables continuous tracking and emergency alerts.

In parallel, a flex sensor–based gesture system allows deaf and dumb users to communicate basic needs (water, food, medicine, help, emergency). Finger bending patterns are converted into signals, processed by Arduino, matched with predefined commands, and displayed on an LCD screen while also being converted into voice output via a speaker and voice module.

The literature review shows similar systems using CNNs, MediaPipe, and wearable sensors, but highlights limitations such as poor accuracy under real conditions, high computation, or lack of simplicity.

Conclusion

The proposed system employs flex sensor–based hand gesture detection to assist deaf and dumb individuals by converting their hand movements into meaningful communication. Finger bending movements are captured using flex sensors attached to a specially designed glove, which detect variations in resistance corresponding to the degree of finger flexion. These signals are then transmitted to an Arduino microcontroller, which acts as the central processing unit of the system. The microcontroller continuously monitors the sensor data and applies programmed algorithms to recognize specific hand gestures. Each detected gesture is mapped to a predefined command representing essential needs, such as requesting water, food, medicine, help, or indicating an emergency situation. This mapping allows individuals with speech and hearing impairments to communicate their requirements efficiently and intuitively with the people around them .To enhance user interaction and accessibility, the system provides both visual and auditory feedback. The recognized gesture is displayed on a 16×2 LCD screen, allowing nearby individuals to read the message even in noisy environments or when audio is not feasible. Simultaneously, the voice module converts the gesture command into a pre-recorded speech output, which is played through an integrated speaker. This ensures that caregivers, family members, or authorities can immediately understand the user’s request, facilitating timely assistance. The combination of visual and audio outputs strengthens communication reliability, reduces misunderstandings, and promotes independence for the user. Overall, the system provides an effective, user-friendly, and real-time communication platform that bridges the interaction gap for deaf and dumb individuals, demonstrating the practical integration of sensor technology and embedded system design in assistive applications.

References

[1] Xiao, Feiyun, Zhen Zhang, Changhai Liu, and Yong Wang. ?Human motion intention recognition method with visual, audio, and surface electromyography modalities for a mechanical hand in different environments.? Biomedical Signal Processing and Control 79 ( 2023 ): 104089. [2] Bora, Jyotishman, Saine Dehingia, Abhijit Boruah, Anuraag Anuj Chetia, and Dikhit Gogoi. ?Real-time Assamese Sign Language Recognition using MediaPipe and Deep Learning.? Procedia Computer Science 218 ( 2023 ): 1384 - 1393. [3] Muneeb, Muhammad, R. Hammad, and Ahmad Jalal. ?Automate appliances via gestures recognition for elderly living assistance.? In IEEE Conference on Advancements in Computational Sciences. 2023. [4] Khalaf, Luay Ibrahim, Salma Abdullah Aswad, Saadaldeen Rashid Ahmed, Bassma Makki, and Mohammed Rashid Ahmed. ?Survey On Recognition Hand Gesture By Using Data Mining Algorithms.? In 2022 International Congress on Human-Computer Interaction, Optimization and Robotic Applications (HORA), pp. 1 - 4. IEEE, 2022. [5] Saxena, Rohan, Romy Garg, Bhoomi Gupta, and Narinder Kaur. ?Deep Learning Based Hand Sign Recognition in the Context of Indian Greetings and Gestures.? In IoT Based Control Networks and Intelligent Systems: Proceedings of 3rd ICICNIS 2022, pp. 287 - 297. Singapore : Springer Nature Singapore, 2022. [6] Zhang, Fan, Valentin Bazarevsky, Andrey Vakunov, Andrei Tkachenka, George Sung, Chuo-Ling Chang, and Matthias Grundmann. ?Mediapipe hands: On-device real-time hand tracking.? arXiv preprint arXiv :2006.10214( 2020 ). [7] Amit, Maricel L., Arnel C. Fajardo, and Ruji P. Medina. ?Recognition of Real-Time Hand Gestures using Mediapipe Holistic Model and LSTM with MLP Architecture.? In 2022 IEEE 10th Conference on Systems, Process and Control (ICSPC), pp. 292 - 295. IEEE, 2022. [8] Deep, A. Litoriya, A. Ingole, V. Asare, S. M. Bhole and S. Pathak,?Realtime Sign Language Detection and Recognition,? 2022 2nd Asian Conference on Innovation in Technology (ASIANCON), Ravet, India, 2022,pp. 1 - 4, doi: 10.1109/ASIANCON55314.2022.9908995. [9] Atif, Bilawal, Neil Buckley, and Emanuele Lindo Secco. ?Hand Gesture and Human-Drone Interaction.? In Intelligent Systemsand Applications: Proceedings of the 2022 Intelligent Systems Conference (IntelliSys) Volume 3, pp. 299 - 308. Cham : Springer International Publishing, 2022. [10] Harris, Moh, and Ali Suryaperdana Agoes. ?Applying hand gesture recognition for user guide application using MediaPipe.? In 2nd International Seminar of Science and Applied Technology (ISSAT 2021), pp. 101 - 108. Atlantis Press, 2021.

Copyright

Copyright © 2026 Pavana M, Satha S, Siva Sankari K, Gayathri R, P Padmaloshani. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.