Embedded AI-Based Gesture Control System

Abstract

This paper presents the design and implementation of an Embedded AI-Based Gesture Control System for Greenhouse Automation. The system provides a contactless and intelligent method to control greenhouse equipment using hand gestures. Conventional control methods, such as switches, remotes, or internet-based systems, may be inconvenient in humid agricultural environments and often require continuous connectivity. To address these issues, the proposed system uses a motion-based gesture recognition approach that operates completely offline. The system uses an MPU6050 accelerometer and gyroscope sensor to capture real-time hand motion data. This data is processed using a lightweight Artificial Intelligence (AI) model trained with sensor datasets and deployed on an ESP32 microcontroller using TensorFlow Lite. The trained model recognizes predefined gestures and converts them into control commands to operate greenhouse devices such as irrigation pumps, ventilation fans, lighting systems, and servo mechanisms through a relay interface. Additional environmental monitoring is carried out using sensors such as DHT11 for temperature and humidity, soil moisture sensors, and water level sensors, which help maintain suitable greenhouse conditions. The system works locally without requiring internet connectivity, ensuring reliability, low latency, and improved data privacy. Experimental implementation and testing show that the system can accurately recognize gestures and control greenhouse devices with fast response time and stable performance. The proposed system provides a low-cost, energy-efficient, and user-friendly solution for smart greenhouse automation and modern precision agriculture.

Introduction

This work presents an Embedded AI-based gesture control system for greenhouse automation that replaces manual control with a contactless, intelligent approach.

The system is designed to improve greenhouse management by automating irrigation, ventilation, and lighting using hand gestures. Instead of camera-based systems (which are costly, power-hungry, and raise privacy issues), it uses a sensor-based approach for better efficiency and suitability in embedded environments.

A motion sensor, the MPU6050, captures hand movement data using accelerometer and gyroscope readings. This data is processed by a lightweight AI model deployed on the ESP32. The model recognizes predefined gestures (such as UP, DOWN, LEFT, RIGHT) and converts them into control commands.

These commands operate greenhouse devices through relay modules, controlling systems like pumps, fans, lights, and servo motors. Additional environmental monitoring is done using sensors such as the DHT11 along with soil moisture and water level sensors to maintain optimal plant conditions.

The AI model is implemented using a lightweight framework such as TensorFlow Lite, enabling offline, energy-efficient operation without internet dependency. Sensor data is also visualized during experimentation using tools like the Arduino Serial Plotter to identify gesture patterns.

From literature and methodology, the system addresses limitations of earlier approaches by being low-cost, portable, privacy-safe, and suitable for smart agriculture. Experimental results show that different gestures produce distinct motion patterns, which the AI model successfully learns for reliable control of greenhouse systems.

Conclusion

This paper presented the design and implementation of an Embedded AI-Based Gesture Control System for Greenhouse Automation. The proposed system uses an MPU6050 accelerometer and gyroscope sensor to capture hand motion data and recognize gestures using a lightweight AI model deployed on the ESP32 microcontroller [2]. Based on the recognized gesture, the system controls greenhouse devices such as irrigation pumps, ventilation fans, lighting systems, and servo mechanisms through a relay module [11]. The system operates offline without requiring internet connectivity, which improves reliability, reduces latency, and ensures data privacy. Experimental results obtained from sensor data visualization and gesture training demonstrate that different hand gestures produce distinct motion patterns, allowing the AI model to classify gestures effectively [2], [4], [6]. The proposed system provides a low-cost, energy-efficient, and user-friendly solution for smart greenhouse automation. By integrating gesture recognition with environmental monitoring sensors, the system improves convenience and efficiency in greenhouse management. This work highlights the potential of embedded artificial intelligence and gesture-based control systems in advancing modern agriculture and smart farming technologies [1], [4], [8].

References

[1] R. Tchantchane, H. Zhou, S. Zhang, and G. Alici, “A Review of Hand Gesture Recognition Systems Based on Noninvasive Wearable Sensors,” Advanced Intelligent Systems, vol. 5, 2023, doi: 10.1002/aisy.202300207. [2] V. Viswanatha, A. C. Ramachandra, R. P. Raghavendra, P. C. Kakarla, V. S. PJ, and N. Mohan, “Implementation of Tiny Machine Learning Models on Arduino 33 – BLE for Gesture and Speech Recognition,” Journal of Xi’an University of Architecture & Technology, vol. XIV, no. 7, pp. 160–168, 2022. [3] S. Zhao, H. Cai, W. Li, Y. Liu, and C. Liu, “Hand Gesture Recognition on a Resource-Limited Interactive Wristband,” Sensors, vol. 21, no. 17, p. 5713, Aug. 2021, doi: 10.3390/s21175713. [4] D. Lauss, F. Eibensteiner, and P. Petz, “A Deep Learning Based Hand Gesture Recognition on a Low-Power Microcontroller Using IMU Sensors,” Embedded Systems Lab Research Paper, UAS Upper Austria, 2022. [5] V. Sideridis, A. Zacharakis, G. Tzagkarakis, and M. Papadopouli, “GestureKeeper: Gesture Recognition for Controlling Devices in IoT Environments,” Proceedings of the Institute of Computer Science, Foundation for Research and Technology-Hellas, 2023. [6] C. K. Mummadi, F. P. P. Leo, K. D. Verma, S. Kasireddy, P. M. Scholl, J. Kempfle, and K. Van Laerhoven, “Real-Time and Embedded Detection of Hand Gestures with an IMU-Based Glove,” Informatics, vol. 5, no. 2, p. 28, Jun. 2018, doi: 10.3390/informatics5020028. [7] S. Vadlamudi, D. N. Kumar, and G. S. Kumar, “Gesture Controlled Robot Using Arduino and MPU6050,” International Journal of Recent Technology and Engineering, vol. 9, no. 1, pp. 777–783, May 2020. [8] E. Fertl, E. Castillo, G. Stettinger, M. P. Cuéllar, and D. P. Morales, “Hand Gesture Recognition on Edge Devices: Sensor Technologies, Algorithms, and Processing Hardware,” Sensors, vol. 25, 2025, doi: 10.3390/s25061687. [9] N. Prasanth, K. Shrivastava, A. Sharma, A. Basu, R. A. Sinha, and S. P. Raja, “Gesture-Based Mouse Control System Based on MPU6050 and Kalman Filter Technique,” International Journal of Intelligent Systems Technologies and Applications, vol. 21, no. 1, pp. 56–71, 2023. [10] A. N. M. Anaki, N. A. Zakaria, M. A. Hamka, and A. Z. Azizi, “Gesture Video Control Using ESP32,” APS Proceedings, vol. 25, pp. 115–120, 2025, doi: 10.5281/zenodo.17678606. [11] Ch. Vandhana and K. Srinivasulu, “Design and Implementation of an Intelligent Gesture Recognition System Using MPU6050 With ESP32 for Real-Time Omni-Directional Robotic Vehicle Control,” International Journal of Engineering & Science Research, vol. 15, no. 3, pp. 592–600, 2025.

Copyright

Copyright © 2026 Vivek Ghanekar, Aniket Gangawane, Prathamesh Parmale, Nikhil Patil, Pranav Rathod. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.