Designing an Image-Based System for Deepfake Detection and Facial Expression Recognition

Abstract

With the widespread use of digital image editing and AI-based generation tools, it has become increasingly easy to manipulate images for deceptive purposes. Fake or altered images can spread misinformation, damage reputations, and mislead the public. In addition to authenticity concerns, analyzing human facial expressions from images has become important for applications such as user behavior analysis and human-computer interaction. This research focuses on the developmentof adeep learning based image analysissystem capableofperforming two key tasks:classifying imagesasreal or fake and recognizing facial expressions. The proposed system uses Convolutional Neural Network (CNN) models trained on publicly available datasets containing both genuine and manipulated images, as well as facial emotion datasets. The system analyzes subtle pixel-level inconsistencies, lighting variations, and structural distortions for forgery detection, while also identifying facial features to determine emotions. The system is implemented as a web-based platform that allows users to choose betweendeepfakedetection and emotionrecognition.Itsupportsmultiplefaceswithin a singleimageand provides individualemotionpredictions along withconfidencescores.Testingonbenchmarkdatasetsdemonstrateshighaccuracyand reliability. The approach provides a practical and efficient solution for image authenticity verification and facial expression analysis, with applications in digital media, surveillance, and interactive systems.

Introduction

This research presents a CNN-based intelligent image analysis system designed to perform two tasks within a single web-based platform: image authenticity detection (real/fake classification) and facial expression recognition. The study addresses the growing problem of digitally manipulated and AI-generated images, which can contribute to misinformation, fraud and fabricated evidence. Traditional approaches such as Error Level Analysis (ELA), metadata inspection and handcrafted image features have limited effectiveness against sophisticated modern manipulations.

The proposed system uses deep learning and Convolutional Neural Networks (CNNs) to automatically learn pixel-level, textural and facial features. ResNet50 with transfer learning, pretrained on ImageNet, is fine-tuned for both real-vs-fake classification and facial emotion recognition. Public datasets including CASIA v2, Columbia Image Splicing Dataset, Kaggle Forgery Dataset, FER-2013 and CK+ were used. Images were resized to 224×224 pixels, normalized and augmented using rotation, flipping, scaling, brightness variation and Gaussian noise. The data were divided into 70% training, 20% validation and 10% testing sets.

For image forgery detection, the system identifies manipulation types such as splicing, copy-move, object removal, cloning and retouching. For emotion recognition, it detects facial expressions such as happiness, sadness, anger and surprise and can analyze multiple faces in a single image, providing individual predictions and confidence scores. The models were trained using binary and categorical cross-entropy losses with the Adam optimizer (learning rate 0.0001). Dropout, batch normalization and early stopping were applied to reduce overfitting.

The system was implemented as a Flask-based web application. Users can upload an image and select either authenticity detection or emotion recognition. Results are generated within a few seconds, making the platform suitable for practical and real-time applications. Privacy and security were considered by processing uploaded images temporarily and using encrypted communication, while a disclaimer emphasizes that predictions should not be treated as definitive legal evidence without expert validation.

Key Results

The optimized CNN showed significant improvement in image authenticity detection:

The model achieved an AUC of 0.96, indicating excellent discrimination between real and manipulated images. False positives decreased from 10% to 4%, while false negatives decreased from 12% to 5% after optimization.

Dataset-wise performance was also strong, with approximately 95% accuracy on CASIA v2, 93% on Columbia and 92% on the Kaggle forgery dataset. The model performed particularly well on structured manipulations but experienced some difficulty with highly compressed images and subtle manipulations.

For facial expression recognition, the system successfully handled multiple faces and different facial orientations, although errors occurred with occlusion, poor lighting, ambiguous emotions and partially visible facial features.

User testing with 40 participants showed positive usability results: 94% rated ease of use positively, 91% were satisfied with detection speed, 89% found the results clear, and 93% expressed confidence in the outputs.

Research Gap and Contribution

The study identifies several limitations in existing systems, including poor cross-dataset generalization, high computational requirements, lack of manipulation localization, limited real-time accessibility, and insufficient attention to explainability and ethics. Unlike many existing systems that focus on only forgery detection or emotion recognition, the proposed framework integrates both functionalities into one accessible web application.

Conclusion

The rapid advancement of imageediting and AI-based generation technologies has made digital image authenticity verification a critical concern in today’s technology-driven world. From social media misinformation to manipulated visual evidence, the impact of altered images extends across multiple domains. In this study, a deep learning-based image analysis system was successfully developed toperform two key tasks: classifying images as real or fake and recognizingfacial expressions. The system is designed to automatically analyze subtle inconsistencies introducedduring manipulation—such as changes inlighting,texture, and pixel correlation—while also identifying facial features to determine human emotions using Convolutional NeuralNetworks (CNNs). The proposedmodel,builtusingtransferlearning with architectures like ResNet50, demonstrated strong performance across multiple benchmark datasets for both image authenticity detection and facialexpressionrecognition. Through effectivetrainingand optimization, the system achieved high accuracy and reliability, ensuring robust performance in real-world scenarios.Theabilitytohandlemultiplefaceswithina single image and generate individual emotion predictions further enhances its practical applicability. One of the key contributions of this project is its web-based implementation, which provides a user-friendly interface for selecting between deepfake detectionand emotion recognition. Unlike traditional academic models, the developed system offers an interactive platform where users can upload an imageand receive meaningful insights along with confidence scores.This improves transparency and allows users to better interpret the results rather than relying solely onbinary outputs. Despite its effectiveness, certain limitations remain. The model’s performance may vary when exposed to unseen datasets or highly complex manipulations. Additionally, while the system provides accurate classification and emotion detection, it does not currently highlight specific regions responsible for predictions, which could further improve interpretability. Addressing these limitations requires continuous dataset expansion and the integration of explainable AI techniques. Future enhancements may include region-based visualization for forgery detection, improved emotion classification accuracyunderchallengingconditions, and deployment optimizations for mobile and cloud-based platforms. Expanding dataset diversity will furtherimprove the system’s generalization capability across varied real-world scenarios. In conclusion, the proposed system effectively combines image authenticity detection and facial expression recognition into a unified framework. By integrating deep learningaccuracywithusabilityandethicalconsiderations, it contributes to building a more reliable and intelligent image analysis system, supporting efforts to reduce misinformation while enhancing human-centric understanding in digital environments.

References

[1] Mirsky,Y.,&Lee,W.(2020).TheCreationand Detection of Deepfakes: A Survey. arXiv preprint. https://arxiv.org/abs/2004.11138 [2] Almars,A.(2021).DeepfakesDetectionTechniques Using Deep Learning: A Survey. Journal of Computer and Communications, 9, 20–35. https://www.scirp.org/journal/paperinformation?paperid=109149 [3] Mushtaq,S.,Saeed,A.,Almotiri,S.H.,Ahmad,F., & Al Ghamdi, M. A. (2023). Deepfake Forensics: A Survey of Digital Forensic Methods for Multimodal Deepfake Identification on Social Media. PeerJ Computer Science. https://peerj.com/articles/cs-2037/ [4] Zhang,L.,etal.(2024).AContemporarySurveyon Deepfake Detection: Datasets, Methods, and Challenges. Electronics, 13(3), 585. https://www.mdpi.com/2079-9292/13/3/585 [5] Goodfellow,I.,Bengio,Y.,&Courville,A.(2016). Deep Learning. MIT Press. https://www.deeplearningbook.org/ [6] Li, S., & Deng, W. (2020). Deep Facial Expression Recognition:ASurvey.IEEETransactionsonAffective Computing. [7] https://arxiv.org/pdf/1804.08348 [8] Xu,H.,etal.(2023).DeepfakeDetection:A Comprehensive Survey from the Reliability Perspective. ACM Digital Library. https://dl.acm.org/doi/10.1145/369971

Copyright

Copyright © 2026 Nehal Luhadia Jain, Varnima Gupta, Priyanshi Joshi, Prof. Nidhi Nigam. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.