Hand Gesture Recognition for Specially Impaired People Using IoT

Abstract

Communication barriers faced by individuals with speech and hearing impairments limit effective interaction, es- pecially in critical situations. This paper presents a real-time, standaloneembeddedIoT-basedhandgesturerecognitionsystem that operates without machine learning, reducing computational complexity and cost while improving portability. The systemuses a wearable glove equipped with five flex sensors to detect finger movements and an MPU6050 sensor to capture hand orientation, with processing handled by an ESP32 microcon- troller. A hysteresis-based threshold mapping technique encodes gestures using combined finger and orientation states, allowing multiple predefined gesture representations to be stored directly in memory. Experimental results from simulation and prototype testing demonstrate 96% accuracy with an average responsetime of 78 ms, satisfying real-time requirements and providing a practical, low-cost solution for assistive communication.

Introduction

This paper presents a wearable smart glove system for real-time gesture-to-text communication designed to help individuals with speech and hearing impairments communicate more easily without relying on interpreters or cloud-based systems.

The main problem it addresses is that existing solutions are either vision-based (sensitive to lighting and limited by camera view) or machine-learning/sensor-heavy systems (require training data, high memory, and external processing), making them unsuitable for low-cost embedded devices like the ESP32.

To solve this, the authors propose a fully embedded, deterministic system using an ESP32, five flex sensors, and an MPU6050. The system detects finger bends and hand orientation, then uses a simple threshold-based algorithm (no machine learning) to map gestures to predefined messages stored in flash memory (PROGMEM). The output is shown on an OLED display.

Key technical features include:

• Hysteresis-based thresholds to prevent signal flickering
• A 5-bit gesture encoding system combined with 6 orientation states (total 192 possible messages)
• Zero dynamic RAM usage for message storage
• Fully offline operation with no wireless dependency

The system is designed for applications such as healthcare communication, workplace inclusion, emergency messaging, education, and IoT control.

In experiments (about 127 trials), the system achieved:

• 96% overall accuracy
• ~78 ms average response latency

Compared to previous ML-based and vision-based systems, it is faster, cheaper, and more suitable for embedded wearable use, though it is limited to a fixed gesture vocabulary and lacks the flexibility of learning-based approaches.

Conclusion

This paper presented a real-time, standalone embedded gesture recognition system for individuals with speech and hearing impairments. The primary contribution is a low-cost IoT wearable achieving deterministic gesture-to-text commu- nication across 192 unique gesture-orientation combinations, without dependency on machine learning, wireless communi- cation, or external computing hardware. Threecoredesigndecisionsjointlyenablereliable,real-time operation within the ESP32’s resource constraints: hysteresis- based dual-threshold flex sensor detection for flicker-free fin- ger state classification; MPU6050 dominant-axis gravity pro- jection for 3D-aware orientation classification; and PROGMEM flash indexing for zero-RAM-cost message storage. These yield O(1) gesture lookup with deterministic sub-100ms la- tency. Experimental evaluation demonstrated 96% overall recog- nition accuracy with a 78ms average response latency across 100testtrials,validatedonbothasimulationplatformand a physical prototype. Future work will prioritize per-user calibration,text-to-speechoutput,IoTconnectivityviaMQTT, and expanded gesture vocabulary validation.

References

[1] J.Ughade,S.Jadhav,S.Jakkan,andS.Bhorge,“HandGestureVocaliserusing IoT,” International Journal for Research in Applied Science &Engineering Technology (IJRASET), vol. 12, no. 5, pp. –, May 2024. [2] S.B.Shroke,P.Agrawal,A.Koli,andS.Kamble,“Assistivetranslatorsfor the deaf and dumb people,” Int. J. Adv. Res. Comput. Sci. Softw.Eng., vol. 4, no. 3, pp. 1–5, Mar. 2014. [3] G. Vasanti and N. Ramesh Babu, “Visual assistance systems for labeldetection via voice output,” Int. J. Innov. Res. Electr. Electron. Instrum.Control Eng., vol. 2, no. 9, pp. 1959–1963, Sep. 2014. [4] R.Patel,V.Patil,andH.Shah,“Recognitionofhandges-tures using accelerometer sensor,” in Proc. Int. Conf. Electron.Commun.Syst.(ICECS),Coimbatore,India,Feb.2015,pp.1–5,doi: 10.1109/ECS.2015.7124928. [5] K. Kadam and N. Joshi, “Gesture recognition using accelerometer andflexsensor,”Int.J.Adv.Res.Comput.Commun.Eng.,vol.6,no.4, pp.221–225,Apr.2017. [6] B.G.LeeandS.M.Lee,“Smartwearablehanddeviceforsignlanguageinterpretationsystemwithsensorsfusion,”IEEESensorsJ.,vol.18,no. 3, pp. 1224–1232, Feb. 2018, doi: 10.1109/JSEN.2017.2779466. [7] A.SankarandT.Hnat,“Realtimehandgesturerecognitionusingaccelerometers,”inProc.11thACMConf.EmbeddedNetw.SensorSyst.(SenSys),Roma,Italy,Nov.2013,pp.1–2,doi: 10.1145/2517351.2517406. [8] M. Elmahgiubi, F. Bensaali, and N. Ramzan, “Sign language trans-latorandgesturerecognition,”inProc.IEEE15thInt. Conf.Bioinf.Bioeng. (BIBE),Belgrade,Serbia,Nov.2015,pp.1–6,doi: 10.1109/BIBE.2015.7367698. [9] L. J. Kau and B. Zhuo, “A real-time portable sign language translationsystem,” in Proc. IEEE Int. Symp. Circuits Syst. (ISCAS), Baltimore,MD, USA, May 2017, pp. 1–4, doi: 10.1109/ISCAS.2017.8050870. [10] M. A. Koshi, “A survey on advanced technology communication be-tween deaf/dumb people using eye blink and flex sensor,” Int. Res. J.Eng. Technol. (IRJET), vol. 5, no. 2, pp. 1462–1466, Feb. 2018. [11] S.Ahmed,K.A.Alam,andM.Masud,“Electronicspeakingsystemfor speech impaired people: speak up,” in Proc. IEEE Int. WIE Conf.Electr. Comput. Eng. (WIECON-ECE), Dhaka, Bangladesh, Dec. 2015,pp.499–502,doi:10.1109/WIECON-ECE.2015.7590148. [12] P. Pandey and V. Jain, “Deaf and dumb people sign language as acommunicationtool,”Int.J.Comput.Appl.,vol.112,no.3,pp.10–14, Feb. 2015. [13] C.H.Hung,Y.W.Bai,andH.Y.Wu,“Homeappliancecontrolby a hand gesture recognition belt,” in Proc. IEEE 4th Glob. Conf.Consum.Electron.(GCCE),Osaka,Japan,Oct.2015,pp.599–600,doi: 10.1109/GCCE.2015.7398595. [14] Y.Sheetal.,“Areal-timehandgesturerecognitionapproachbasedon motion features of feature points,” in Proc. IEEE 17th Int. Conf.Comput.Sci.Eng.(CSE),Chengdu,China,Dec.2014,pp.1–6,doi: 10.1109/CSE.2014.273. [15] P. Vijayalakshmi and M. Aarthi, “Sign language to speech conversion,”inProc.IEEEInt.Conf.RecentTrendsInf.Technol.(ICRTIT),Chennai,India, Apr. 2016, pp. 1–6, doi: 10.1109/ICRTIT.2016.7569547.

Copyright

Copyright © 2026 Mr. Balakrishna Kancherla , P. Guru Manikanta, P. Phaneendra Kumar, P. Sai, P. Kavitha. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.