DeepGuard: A Robust Multimodal Framework for Automated Deepfake Screening and Triage

Abstract

Deepfakes are synthetic images, videos, and voice recordings produced by deep generative models, and their realism now exceeds what human observers can reliably judge. Most existing detectors address a single medium, which leaves them blind to forgeries that live in motion or in sound. This paper presents DeepGuard, a unified multimodal framework that accepts an image, a video, or an audio file through one web interface and returns a Real or Deepfake verdict together with a confidence percentage. Uploaded media are normalized and, for video, separated into frames and an audio track. YOLOv8-Face localizes and crops the facial region, a ResNet-50 convolutional network extracts spatial forgery cues such as texture irregularities and blending artifacts, a bidirectional long short-term memory network models frame-to-frame inconsistencies, and a Wav2Vec2 model examines the speech signal for synthetic or cloned voices. A Bayesian weighted fusion stage combines whichever modality predictions are available into one deepfake probability, which is thresholded and stored with its metadata in a history database. The system is implemented with a Flask backend, OpenCV, and NumPy. The paper details the design rationale, the algorithm of each module, the fusion rule, and a leakage-free evaluation protocol based on accuracy, precision, recall, F1 score, and a confusion matrix. Quantitative results are intentionally not reported here; they will be added after experiments on benchmark datasets.

Introduction

The text presents DeepGuard, a unified multimodal deepfake detection system designed to identify fake images, videos, and audio recordings. Modern deepfake technologies can generate highly realistic faces and voices, making manual detection difficult. Although deepfakes have legitimate applications in filmmaking and dubbing, they are also used for identity theft, voice-cloning scams, and spreading misinformation. A major challenge is that detectors trained on known fakes may fail to recognize new types of manipulations.

DeepGuard addresses three major limitations of existing detection systems: their focus on a single media type, their inability to detect independently manipulated audio and visual streams, and their lack of a simple, unified interface for ordinary users.

Key Features and Methodology

DeepGuard integrates four main technologies:

• YOLOv8-Face: Detects and extracts faces from images and video frames.

• ResNet-50: Extracts visual features and identifies possible signs of facial manipulation.

• Bi-LSTM: Analyzes video frames over time to detect unnatural facial movements and inconsistencies.

• Wav2Vec2: Analyzes audio recordings to identify potential synthetic or cloned voices.

The system follows a sequence of steps: media upload, preprocessing, face detection, visual and temporal feature extraction, audio analysis, probability fusion, and final classification.

A Bayesian weighted fusion mechanism combines the available model predictions, assigning greater weight to more reliable branches based on validation performance. It automatically adjusts the weights according to the modalities available, allowing it to process images, silent videos, videos with sound, and audio-only files.

System Architecture

DeepGuard uses a Flask-based web application with four layers:

1. Presentation layer: Provides a user-friendly interface for uploading media and viewing results.

2. Application layer: Handles file processing and extracts video frames and audio.

3. Inference layer: Runs the deepfake detection models.

4. Data layer: Stores predictions, confidence scores, and analysis history.

The system provides a binary classification of Real or Deepfake, along with a confidence percentage and a history of previous analyses.

Evaluation and Limitations

The proposed evaluation protocol uses accuracy, precision, recall, F1 score, a confusion matrix, and cross-dataset testing. It emphasizes testing on unseen datasets to assess how well the detector generalizes to unfamiliar deepfake techniques.

However, DeepGuard has several limitations. It analyzes uploaded files rather than live streams, does not identify the specific type of manipulation, and treats each file as entirely real or entirely fake. Partially manipulated media remains a challenge for future research. The reported confidence percentage is a model output and should not automatically be interpreted as a guarantee of correctness.

Conclusion

This paper presented DeepGuard, a unified framework that verifies images, videos, and audio through one web interface. A face-guided pipeline of YOLOv8-Face and ResNet-50 captures spatial evidence, a Bi-LSTM captures temporal evidence, Wav2Vec2 captures audio evidence, and a Bayesian weighted fusion rule combines whichever branches apply into a Real or Deepfake label with a confidence score that can be stored for later review. The contribution is the integration of established components into a practical tool, together with an evaluation protocol that stresses leakage-free splits, cross-dataset testing, and full metric reporting. Because new fake-generation methods will keep appearing, no single detection approach can be expected to remain sufficient; checking picture, motion, and sound together is a more durable strategy. Future work will complete the experiments on larger and more diverse datasets, including generators unseen during training, and fill Tables III and IV. Planned improvements are: • comparing the fixed Bayesian weighting with learned fusion and with transformer-based detectors; • highlighting the suspect frame or region to assist human reviewers; • adding lip-to-speech consistency checks for fake voices [18] and handling partially manipulated media; • speeding up and compressing the models for near-real-time and low-cost deployment; • testing robustness under compression, resizing, noise, and poor quality; and • adding login-based access control and secure storage of saved results.

References

[1] L. Verdoliva, “Media forensics and deepfakes: An overview,” IEEE J. Sel. Topics Signal Process., vol. 14, no. 5, pp. 910–932, Aug. 2020. [2] M. Masood, M. Nawaz, K. M. Malik, A. Javed, and A. Irtaza, “Deepfakes generation and detection: State-of-the-art, open challenges, countermeasures, and way forward,” 2021, arXiv:2103.00484. [3] P. Edwards, J.-C. Nebel, D. Greenhill, and X. Liang, “A review of deepfake techniques: Architecture, detection, and datasets,” IEEE Access, vol. 12, pp. 154718–154742, 2024. [4] N. U. R. Ahmed, A. Badshah, H. Adeel, A. Tajammul, A. Daud, and T. Alsahfi, “Visual deepfake detection: Review of techniques, tools, limitations, and future prospects,” IEEE Access, vol. 13, pp. 1923–1961, 2025. [5] S. A. Khan and D.-T. Dang-Nguyen, “Deepfake detection: A comparative analysis,” 2023, arXiv:2308.03471. [6] E. Altuncu, V. N. L. Franqueira, and S. Li, “Deepfake: Definitions, performance metrics and standards, datasets, and a meta-review,” Front. Big Data, vol. 7, Art. no. 1400024, 2024. [7] D. Afchar, V. Nozick, J. Yamagishi, and I. Echizen, “MesoNet: A compact facial video forgery detection network,” in Proc. IEEE Int. Workshop Inf. Forensics Security (WIFS), 2018, pp. 1–7. [8] Y. Li and S. Lyu, “Exposing DeepFake videos by detecting face warping artifacts,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. Workshops (CVPRW), 2019, pp. 46–52. [9] L. Li, J. Bao, T. Zhang, H. Yang, D. Chen, F. Wen, and B. Guo, “Face X-ray for more general face forgery detection,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2020, pp. 5001–5010. [10] F. Chollet, “Xception: Deep learning with depthwise separable convolutions,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2017, pp. 1251–1258. [11] A. Rössler, D. Cozzolino, L. Verdoliva, C. Riess, J. Thies, and M. Nießner, “FaceForensics++: Learning to detect forged facial images,” in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), 2019, pp. 1–11. [12] H. Dang, F. Liu, J. Stehouwer, X. Liu, and A. K. Jain, “On the detection of digital face manipulation,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2020, pp. 5781–5790. [13] X. Zhu, H. Wang, H. Fei, Z. Lei, and S. Z. Li, “Face forgery detection by 3D decomposition,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2021, pp. 2929–2939. [14] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2016, pp. 770–778. [15] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2016, pp. 779–788. [16] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Comput., vol. 9, no. 8, pp. 1735–1780, 1997. [17] M. Schuster and K. K. Paliwal, “Bidirectional recurrent neural networks,” IEEE Trans. Signal Process., vol. 45, no. 11, pp. 2673–2681, Nov. 1997. [18] S. Agarwal, H. Farid, O. Fried, and M. Agrawala, “Detecting deep-fake videos from phoneme-viseme mismatches,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. Workshops (CVPRW), 2020. [19] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” in Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), vol. 33, 2020, pp. 12449–12460. [20] X. Wang et al., “ASVspoof 2019: A large-scale public database of synthesized, converted and replayed speech,” Comput. Speech Lang., vol. 64, Art. no. 101114, 2020. [21] Y. Zhou and S.-N. Lim, “Joint audio-visual deepfake detection,” in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), 2021, pp. 14800–14809. [22] H. Khalid, S. Tariq, M. Kim, and S. S. Woo, “FakeAVCeleb: A novel audio-video multimodal deepfake dataset,” 2021, arXiv:2108.05080. [23] G. Jocher, A. Chaurasia, and J. Qiu, “Ultralytics YOLOv8,” 2023. [Online]. Available: https://github.com/ultralytics/ultralytics [24] Y. Li, X. Yang, P. Sun, H. Qi, and S. Lyu, “Celeb-DF: A large-scale challenging dataset for DeepFake forensics,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2020, pp. 3207–3216. [25] B. Dolhansky et al., “The DeepFake Detection Challenge (DFDC) dataset,” 2020, arXiv:2006.07397. [26] L. Jiang, R. Li, W. Wu, C. Qian, and C. C. Loy, “DeeperForensics-1.0: A large-scale dataset for real-world face forgery detection,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2020, pp. 2889–2898.

Copyright

Copyright © 2026 Rajini G, Sulaksha S, Preetham P, YashwanthRaj T R, Suhas Bharadwaj G R. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.