Emotion Based AI Assistant with Dynamic System Control

Abstract

This project presents the design and development of an Emotion-Based AI Assistant with Dynamic System Control and Hand Gesture Interaction, implemented entirely on a standard laptop environment using Python. Unlike conventional AI assistants that rely solely on voice commands, the proposed system integrates voice recognition, hand gesture recognition, contextual natural language understanding, real-time operating system control, and emotion-aware interaction into a unified and intelligent desktop platform. User interaction is enabled through speech input and visual hand gestures, allowing seamless and contactless control of the system. Voice commands are interpreted using the Gemini 2.5 Flash AI model, while the system further enhances human-computer interaction through an emotion detection module implemented using OpenCV and the DeepFace library. Continuous emotion re-evaluation ensures real-time responsiveness and personalized interaction, such as triggering adaptive actions like playing calming background music during negative emotional states.

Introduction

The rapid advancement of Artificial Intelligence has transformed human–computer interaction by moving beyond traditional input devices toward more natural, contactless, and intelligent interfaces. While existing voice assistants represent progress, they are limited on desktop systems due to restricted OS control, reliance on cloud services, and lack of emotional awareness. These limitations motivate the development of more responsive, privacy-preserving, and emotionally intelligent desktop assistants.

The proposed Emotion-Based AI Assistant integrates hand gesture recognition, voice interaction, and facial emotion detection to enable intuitive and adaptive system control. Using real-time webcam input, the system identifies hand gestures through landmark detection and interprets facial expressions to recognize user emotions. Validated gestures and emotional states are then mapped to operating system actions such as application control, media management, and system adjustments, with secure two-hand gestures preventing accidental mode changes.

The system is modular, combining generative AI–based voice recognition, MediaPipe-powered gesture tracking, DeepFace-based emotion analysis, and an adaptive response engine that adjusts system behavior based on the user’s emotional state. Implemented in a multi-threaded Python architecture, the assistant ensures real-time responsiveness, local processing for privacy, and seamless OS-level automation. Overall, the project demonstrates a shift toward affective, context-aware, and human-centric human–computer interaction on personal computers.

Conclusion

The Emotion-Based AI Assistant successfully demonstrates the application of affective computing and real-time computer vision to streamline human-computer interaction and enhance desktop productivity. The platform improves efficiency by providing a touchless interface for system control, minimizes interaction friction through contextual reasoning, and encourages an engaging user experience. By integrating empathetic responses into the core architecture, the assistant addresses the psychological needs of the user, creating a more responsive and human-centric digital environment.

References

[1] Kapoor and S. R. Gupta, \"Designing Emotion-Aware AI Assistants for Desktop Environments,\" International Conference on Artificial Intelligence and Human-Computer Interaction (ICAHCI), pp. 12-18, 2022. [2] R. Patel and M. Singh, \"Voice-Controlled Personal Assistants: A Survey on AI Integration and System Automation,\" International Journal of Intelligent Systems, vol. 35, no. 4, pp. 250-260, 2021. [3] L. Chen and J. Zhao, \"Real-Time Emotion Detection Using Facial Recognition in AI Applications,\" IEEE Transactions on Affective Computing, vol. 13, no. 2, pp. 450-462, 2022. [4] H. Kim and T. Lee, \"Dynamic OS Control through Natural Language Processing in Personal Assistants,\" International Journal of Software Engineering and Knowledge Engineering, vol. 31, no. 5, pp. 715-730, 2021. [5] Smith et al., \"Advances in Voice-Activated AI Assistants,\" Contextual Reasoning Survey, 2021. [6] Brown et al., \"Emotion Recognition in Human-Computer Interaction,\" Affective Computing Research, 2021.

Copyright

Copyright © 2026 Kamatchi T P, Suganya J, Pranav A, Surya Kesavan, Vivek D, Nishanth B. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.