Touchless Gesture-Based Human-Machine Interaction for Industrial Safety and Automation

Abstract

Recent research explores the integration of computer vision to advance industrial automation, specifically focusing on touchless gesture control and safety monitoring. The sources describe systems designed to replace physical interfaces with hand gestures for controlling Programmable Logic Controllers (PLCs) and collaborative robots, thereby enhancing hygiene and worker safety in hazardous zones [1][4]. These technological advancements utilize deep learning models like LSTM and 3D Convolutional Neural Networks to ensure real-time recognition accuracy [1][4]. Nevertheless, user-centered evaluations highlight barriers such as the absence of tactile feedback and social concerns regarding the professional perception of gesturing [2]. Complementing these control interfaces, the sources also propose action-aware safety frameworks that automatically verify Personal Protective Equipment (PPE) compliance based on the specific task being performed [3]. Collectively, these contributions illustrate a shift toward more intuitive, hands-free industrial environments that prioritize both operational efficiency and proactive risk mitigation.

Introduction

The text explores the development and application of touchless gesture control systems in Industry 4.0 environments, highlighting their role in improving safety, ergonomics, and operational efficiency in industrial and hygiene-sensitive sectors. Traditional physical interfaces, such as buttons and switches, are limited by wear, maintenance, and contamination risks. Gesture-based systems provide hands-free interaction, reducing injury risk, improving hygiene, and enabling operators with limited mobility to control machinery from a safe distance.

Key Points:

• Methodology:

  • Data acquisition includes MediaPipe hand landmarks, Kinect 3D skeleton tracking, and 320 hours of surveillance footage.

  • Preprocessing isolates motion from background noise and filters human activity.

  • Recognition employs hybrid deep learning architectures, combining 3D CNNs, ConvLSTM2D, MLPs, and LSTMs to capture spatio-temporal dependencies.

  • Safety-focused frameworks integrate SlowFast networks with YOLOv9 for action-aware PPE verification.

  • Systems interface with Mitsubishi iQ-R PLCs via Python-based real-time controls, providing immediate actuation and feedback.

• Literature Insights:

  • MediaPipe + LSTM/MLP: Achieves 92% gesture recognition accuracy for direct PLC-based control.

  • 3D CNN + ConvLSTM: Enables real-time collaborative robot interaction with robust spatio-temporal modeling.

  • SlowFast + YOLOv9: Provides task-specific safety monitoring for PPE compliance, improving F1-score by 23%.

  • Kinect-based HMI: Evaluates human factors, identifying barriers like lack of tactile feedback and perceptions of unprofessionalism.

• Strengths:

  • High technical accuracy and task-specific performance.

  • Effective preprocessing for robust gesture detection in complex industrial environments.

  • Human-centered design enhances usability and ergonomics, particularly in collaborative robotics and factory loading operations.

• Weaknesses and Limitations:

  • Lack of tactile feedback reduces operator confidence.

  • 2D vision limitations hinder detection of small objects in large fields of view.

  • Social acceptability of gestures remains a barrier in professional settings.

  • Most datasets are single-actor and lab-controlled, failing to capture real-world industrial complexity.

  • Gesture-to-machine mapping is often limited to simple commands, lacking multi-input or analog controls.

• Research Gaps:

  1. Need for 3D/depth sensing to accurately measure distances in hazardous environments.

  2. Alternative feedback mechanisms to compensate for absence of tactile cues.

  3. More realistic, multi-actor datasets representing chaotic industrial scenarios.

  4. Enhanced detection for small objects in large fields of view.

  5. Addressing social and professional acceptability of gesture interfaces.

  6. Expanding gesture-to-machine mapping for complex, multi-input control.

Conclusion

Current research highlights a transformative shift toward touchless industrial interaction, demonstrating the efficacy of hybrid deep learning architectures—specifically LSTM, 3D CNN, and SlowFast networks—in achieving high-precision recognition for PLC and cobot control [1][3][4]. Key findings indicate that landmark-based models can achieve 92% accuracy in machine actuation, while action-aware frameworks improve safety compliance by reducing false alarms through task-specific PPE verification, yielding recall rates up to 93% [1][3]. However, field evaluations reveal significant barriers to adoption, as the absence of tactile feedback correlates with operator frustration and a perceived lack of professional self-efficacy [2]. To address these limitations, future research must prioritize the development of alternative feedback mechanisms to restore a sense of control without physical contact. Furthermore, expanding system functionality to support multi-input gestures and analog value configurations is essential for complex operations [1]. Finally, transitioning toward depth-aware monitoring and utilizing more realistic, multi-actor datasets will be critical to ensuring environmental robustness within the chaotic field of view typical of real-world industrial shop floors [3][4].

References

[1] Yashika, A. Patra, and N. S. Das, \"Human-Machine Interaction in Industrial Automation: Gesture-Based PLC Control,\" in Proc. 15th Int. Conf. Comput., Commun. Netw. Technol. (ICCCNT), 2024, pp. 1–13. [2] T. Heimonen, J. Hakulinen, M. Turunen, J. P. P. Jokinen, T. Keskinen, and R. Raisamo, \"Designing Gesture-Based Control for Factory Automation,\" in Human-Computer Interaction – INTERACT 2013, Part II, LNCS 8118, P. Kotzé et al., Eds. Heidelberg, Germany: Springer, 2013, pp. 202–209. [3] S. N. Reddy, V. Kurrey, M. Nagar, and G. R. Gupta, \"Action Recognition based Industrial Safety Violation Detection,\" in Proc. 8th Int. Conf. Data Sci. Manage. Data (CODS-COMAD), Jodhpur, India, Dec. 2024, pp. 1–10. doi: 10.1145/3703323.3703722. [4] W. Dumoulin, N. Thiry, and R. Slama, \"Real Time Hand Gesture Recognition in Industry,\" in Proc. 3rd Int. Conf. Video, Signal Image Process. (VSIP), Wuhan, China, Nov. 2021, pp. 1–11. doi: 10.1145/3503961.3503962

Copyright

Copyright © 2026 Ms. Rutuja Gharat, Mr. Neej Patel, Ms. Rashmi Chaudhari, Mrs. Hiral Patel. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.