AI Hand Sign Detection and Recognition

Abstract

Communication plays a vital role in everyday life by allowing people to exchange ideas, information, and emotions. For individuals with hearing and speech disabilities, sign language is one of the most effective forms of communication, relying on hand gestures, facial expressions, and body movements to convey meaningThis paper presents an AI-based Hand Sign Detection and Recognition System using Convolutional Neural Networks (CNNs) to improve communication between individuals who use sign language and those who do not. Sign language is an essential communication method for people with hearing or speech impairments, but its limited understanding among the general population creates barriers in education, healthcare, employment, and social interaction. Advances in artificial intelligence, computer vision, and deep learning have enabled the development of automated Sign Language Recognition (SLR) systems that translate hand gestures into text or speech. The proposed system employs CNNs to automatically learn important visual features such as hand shape, finger positions, gesture contours, edges, and spatial patterns, eliminating the need for handcrafted feature extraction. The objective is to achieve high recognition accuracy while maintaining robustness under different lighting conditions, backgrounds, and user variations. The model is also designed to generalize effectively across diverse users and environments. The study reviews recent literature on sign language recognition, highlighting the success of CNNs, hybrid CNN–Graph Neural Network (GNN) models, Long Short-Term Memory (LSTM) networks for dynamic gestures, and transfer learning techniques. While previous approaches have achieved recognition accuracies above 90%, challenges remain in real-time performance, computational complexity, hardware requirements, and robustness to environmental variations. The proposed architecture uses the Sign-MNIST dataset, which contains labeled grayscale hand-sign images stored in CSV format. The system workflow includes: Image acquisition from the Sign-MNIST dataset. Image preprocessing through normalization, reshaping, and noise reduction. Setting CNN hyperparameters such as learning rate, batch size, and training epochs. CNN training to learn discriminative spatial features. Continuous error evaluation until acceptable performance is achieved. Model testing using an 80:20 train-test split to assess generalization on unseen gesture images. Feature extraction plays a crucial role in improving classification accuracy. The study focuses on: Edge and contour features, which capture finger outlines, palm boundaries, and gesture geometry using operators such as Sobel, Prewitt, or Canny. Edge contrast features, which emphasize intensity differences along hand boundaries, enabling the model to distinguish between visually similar gestures. By combining efficient preprocessing, edge-based feature enhancement, and CNN-based classification, the proposed system provides accurate and reliable static hand gesture recognition. The framework contributes to the development of intelligent assistive technologies that support inclusive communication and can be extended to real-time sign language translation and broader accessibility applications. Despite its importance, sign language is not widely understood by the general population, which often makes communication between sign language users and non-signers difficult. This communication gap can create obstacles in education, employment, healthcare services, and social interactions, highlighting the need for technologies that can facilitate more accessible and inclusive communication.

Introduction

This paper presents an AI-based Hand Sign Detection and Recognition System using Convolutional Neural Networks (CNNs) to improve communication between individuals who use sign language and those who do not. Sign language is an essential communication method for people with hearing or speech impairments, but its limited understanding among the general population creates barriers in education, healthcare, employment, and social interaction. Advances in artificial intelligence, computer vision, and deep learning have enabled the development of automated Sign Language Recognition (SLR) systems that translate hand gestures into text or speech.

The proposed system employs CNNs to automatically learn important visual features such as hand shape, finger positions, gesture contours, edges, and spatial patterns, eliminating the need for handcrafted feature extraction. The objective is to achieve high recognition accuracy while maintaining robustness under different lighting conditions, backgrounds, and user variations. The model is also designed to generalize effectively across diverse users and environments.

The study reviews recent literature on sign language recognition, highlighting the success of CNNs, hybrid CNN–Graph Neural Network (GNN) models, Long Short-Term Memory (LSTM) networks for dynamic gestures, and transfer learning techniques. While previous approaches have achieved recognition accuracies above 90%, challenges remain in real-time performance, computational complexity, hardware requirements, and robustness to environmental variations.

The proposed architecture uses the Sign-MNIST dataset, which contains labeled grayscale hand-sign images stored in CSV format. The system workflow includes:

• Image acquisition from the Sign-MNIST dataset.
• Image preprocessing through normalization, reshaping, and noise reduction.
• Setting CNN hyperparameters such as learning rate, batch size, and training epochs.
• CNN training to learn discriminative spatial features.
• Continuous error evaluation until acceptable performance is achieved.
• Model testing using an 80:20 train-test split to assess generalization on unseen gesture images.

Feature extraction plays a crucial role in improving classification accuracy. The study focuses on:

• Edge and contour features, which capture finger outlines, palm boundaries, and gesture geometry using operators such as Sobel, Prewitt, or Canny.
• Edge contrast features, which emphasize intensity differences along hand boundaries, enabling the model to distinguish between visually similar gestures.

By combining efficient preprocessing, edge-based feature enhancement, and CNN-based classification, the proposed system provides accurate and reliable static hand gesture recognition. The framework contributes to the development of intelligent assistive technologies that support inclusive communication and can be extended to real-time sign language translation and broader accessibility applications.

Conclusion

The development of a robust Convolutional Neural Network-based Sign Gesture Recognition system marks a significant step toward enhancing accessible communication for individuals with hearing and speech impairments. The proposed CNN model achieved an accuracy of 99.2%, an F1-score of 95.53%, precision of 91.5%, and recall of 99.94% on the Sign-MNIST dataset, validating the effectiveness of the multi-feature extraction pipeline comprising edge and contour features, edge contrast, centroid-based intensity, spatial bandwidth, texture flatness, spatial intensity mapping, and pixel intensity features. Despite the promising performance of the current model, further improvements can considerably expand its applicability in real-world scenarios. Future work may focus on: 1) Incorporating larger and more diverse datasets, ensuring that gesture variations across different users, skin tones, angles, and environmental conditions are effectively addressed. 2) Employing advanced data augmentation strategies to enrich dataset variability, enabling the model to generalize more efficiently. 3) Integrating transfer learning and lightweight deep learning architectures such as MobileNet, VGG16, or EfficientNet to enhance accuracy while reducing computational overhead, making the system suitable for mobile and embedded platforms. 4) Exploring temporal models like GRU, or transformerbased architectures to further improve the recognition of dynamic gestures that involve sequential hand movements. 5) Developing real-time deployment pipelines that can operate on edge devices, enabling on-the-fly gesture translation for the deaf and mute community.

References

[1] Pattern Recognition Letters, vol. 150, pp. 85–102, 2021. [2] J. P. K. Pisharady and M. Saerbeck, “Recent methods and databases in vision-based hand gesture recognition: A review,” Computer Vision and Image Understanding, vol. 141, pp. 152–165, 2015. [3] M. N. Islam, M. A. Hossain, and M. A. H. Akhand, “Bangla sign language recognition using convolutional neural network,” in Proc. IEEE ICIEV, Fukuoka, Japan, 2018, pp. 307–312. [4] A. K. Bheda and D. Radpour, “Using deep convolutional networks for gesture recognition in American Sign Language,” arXiv preprint arXiv:1710.06836, 2017. [5] K. Sahoo, S. Bhoi, and P. K. Sa, “Hybrid sensor and vision-based Indian sign language recognition using deep learning,” IEEE Access, vol. 8, pp. 11765–11777, 2020. [6] T. Starner and A. Pentland, “Real-time American sign language recognition from video using hidden Markov models,” in Proc. IEEE ISCV, Coral Gables, USA, 1995, pp. 265–270. [7] M. P. Kadam and S. R. Todkar, “Real-time sign language recognition using deep learning and computer vision,” Int. J. Advanced Research in Computer and Communication Engineering, vol. 9, no. 4, pp. 1–6, 2020. [8] C. Pigou, S. Dieleman, P. Kindermans, and B. Schrauwen, “Sign language recognition using convolutional neural networks,” in Proc. ECCV Workshops, Zurich, Switzerland, 2014, pp. 572–578. [9] R. Deepika, R. Rajeswari, and V. Divya, “Hand gesture recognition using a CNN-GNN hybrid deep learning architecture,” IEEE, 2024. [10] S. Snehalatha et al., “Action recognition based sign detection using models,” IEEE, 2024. [11] A. Teran-Quezada, J. Hernandez-Vela, and E. Sucar, “ConvNet-based hand gesture recognition for educational applications,” Computers & Education, vol. 182, pp. 104–121, 2022. [12] T. Saini and N. Kumari, “Sign Spectrum: Dynamic sign language interpretation using AI,” IEEE, 2024.

Copyright

Copyright © 2026 Abdul Razzak Surve, Yash Harish Somani, Mayuri More, Payal Bhadikar, Jyoti Ramteke. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.