Detection of Face-Swap Based Deepfake Videos Using Hybrid CNN-LSTM Architecture

Abstract

The development of deepfake technologies due to breakthroughs in AI and deep learning allows producing highly realistic manipulated videos and audio, thus posing a threat to misinformation and digital security. Despite deepfake technology having several legitimate uses, including use in the media industry, its inappropriate use for distributing fake news, impersonation, and cyber attacks demands the development of reliable detection techniques. The current state-of-the-art approaches of detecting deepfakes mostly utilize spatial or temporal analysis based on CNNs and RNNs; however, most existing approaches fail to generalize well and are unable to recognize more complicated manipulations on a variety of different data sets. This paper aims at developing a novel multimodal approach to deepfake detection, integrating spatial, temporal, and audio features. The proposed system makes use of CNN-based architectures to extract spatial information from the input images, transformers to capture the temporal information, and Mel-frequency cepstral coefficients (MFCC) to analyze the audio data. These heterogeneous features are combined using attention-based learning to improve the classification accuracy. The proposed method was tested on various benchmarking datasets, including FaceForensics++, DeepFake Detection Challenge (DFDC), and Celeb-DF, yielding higher accuracy than current methods. Experimental results indicate that the combination of multimodal features enhances the detection capacity. The proposed model is highly efficient in addressing deepfake challenges in the digital forensic field.

Introduction

The text presents a deepfake video detection system designed to identify fake or manipulated videos using advanced deep learning techniques. Deepfake technology can realistically alter faces, expressions, identities, and speech, creating serious concerns such as misinformation, identity theft, political manipulation, and loss of trust in digital media.

Background

Traditional deepfake detection methods mainly use CNNs to identify visual abnormalities in individual video frames. However, these methods may miss inconsistencies that occur across multiple frames, such as unnatural facial movements or expressions. CNN-LSTM systems improve temporal analysis, but they can be computationally expensive and may struggle with long-term relationships between frames.

The proposed approach aims to improve detection by combining spatial, temporal, and audio information.

Proposed System

The system follows a pipeline:

Input Video → Pre-processing → Face Detection & Cropping → Feature Extraction → Temporal/Audio Analysis → Feature Fusion → Classification → REAL/FAKE

• Pre-processing: Videos are divided into frames, faces are detected using methods such as MTCNN, and unnecessary background areas are removed.
• Spatial analysis: CNN models such as ResNeXt or EfficientNet identify visual artifacts, texture problems, distortions, and other manipulation clues.
• Temporal analysis: LSTM or transformer models analyze relationships between video frames to identify unnatural facial movements and expressions.
• Audio analysis: Audio is converted into MFCC features to detect unnatural speech and audio-visual inconsistencies such as lip-sync problems.
• Feature fusion: Visual, temporal, and audio features are combined using an attention-based approach.
• Classification: The system classifies the video as REAL or FAKE and provides a confidence level.

Methodology

The study proposes using benchmark datasets such as FaceForensics++, DFDC, and Celeb-DF. Data is preprocessed through frame extraction, face detection, cropping, normalization, noise reduction, and augmentation.

The model is trained using binary cross-entropy loss and the Adam optimizer, with techniques such as dropout used to reduce overfitting.

Evaluation

The system is evaluated using:

• Accuracy
• Precision
• Recall
• F1-score
• ROC-AUC
• Confusion matrix

The results indicate that the CNN-LSTM hybrid approach performs better than using CNN or LSTM alone, because it can analyze both spatial and temporal inconsistencies. Multimodal analysis involving audio is expected to further improve reliability.

Challenges

The major limitation is high computational complexity. Other challenges include poor generalization across different datasets, the rapidly changing nature of deepfake generation techniques, and the need for standardized evaluation methods.

Conclusion

In this paper, we present a combined CNN-LSTM model that can detect face-swap deepfake videos using spatial and temporal analysis techniques. The proposed approach demonstrates superior accuracy in identifying the manipulated videos and performs better than conventional approaches. The combination of CNN and LSTM models makes the proposed detection method more accurate and robust. It is evident from the paper that there is a need for development of advanced detection techniques to deal with the challenges of deepfake technologies.

References

[1] Mulani, C. R., et al., “Development of AI/ML Based Solution for Detection of Face Swap Based Deepfake Videos,” International Journal for Multidisciplinary Research, 2025. [2] Malik, A., et al., “DeepFake Detection for Human Face Images and Videos: A Survey,” IEEE Access, 2022. [3] Sunkari, V., et al., “Artificial Intelligence for Deepfake Detection: Systematic Review and Impact Analysis,” IAES International Journal of Artificial Intelligence, 2024. [4] Yuezun Li, Siwei Lyu, “ExposingDF Videos By Detecting Face Warping Artifacts,” in arXiv:1811.00656v3 [5] Yuezun Li, Ming-Ching Chang and Siwei Lyu “Exposing AI Created Fake Videos by Detecting Eye Blinking” in arxiv. [6] Huy H. Nguyen , Junichi Yamagishi, and Isao Echizen “ Using capsule networks to detect forged images and videos ”. [7] Hyeongwoo Kim, Pablo Garrido, Ayush Tewari and Weipeng Xu “Deep Video Portraits” in arXiv:1901.02212v2. [8] Umur Aybars Ciftci, ?Ilke Demir, Lijun Yin “Detection of Synthetic Portrait Videos using Biological Signals” in arXiv:1901.02212v2 [9] I an Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, 2014. [10] David G¨uera and Edward J Delp. Deepfake video detection using recurrent neural networks. In AVSS, 2018. [11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.

Copyright

Copyright © 2026 Suraj S. Pawar, Kaustubh R. Saswade, Nikhil R. Mane, Jay L. Borate, Anurag S. Karande. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.