Automated Toxicity Detection in Online platform

Abstract

The growth of online communication tools has caused a rise in the number of instances of toxic, abusive and harmful content is now a real issue for the safety of users and their digital well-being. Old School moderation practices based on the use of keyword-based filtering are generally not able to identify context and therefore lead to poor identification of content. This paper discusses an automated AI-based toxicity detection system which will be able to accurately identify, in real-time, the nature of what someone is posting, taking context into account and making use of various forms of data to help make this determination possible. The proposed system uses a hybrid model of Natural Language Processing and Deep Learning techniques to analyze multi-modal sources of data; including text, image and voice. Image data and voice data are converted to text format through Optical Character Recognition (OCR) and Speech-to-Text methods. The converted data then undergoes \"preprocessing\" before being evaluated using multiple techniques for text analysis, i.e. REGEX rule based filtering, TextBlob (for sentiment analysis), and BERT (for context analysis). The ability to evaluate the data at multiple layers of evaluation allows for very precise toxic scoring of the data which then allows for the automated moderation of that data through content filtering, warning generation, and user feedback. The results of the experimental analysis indicate that the proposed system has improved accuracy in detecting toxicity, reduced false positives, and improved the quality of user interactions. Therefore, it can be concluded that the proposed system is a scalable, intelligent and efficient way to create safe digital environments while also addressing the shortcomings of previous moderation techniques.

Introduction

The text describes the development of an AI-based Automated Toxicity Detection System designed to improve content moderation on social media platforms. The motivation comes from the increasing amount of toxic, abusive, and harmful content across platforms like social networks, messaging apps, and forums, where traditional moderation methods (keyword filters and manual review) are insufficient. These older systems struggle to detect contextual, subtle, or multi-modal toxicity such as sarcasm, implicit hate speech, and content expressed through images or voice.

To address this, the proposed system uses a multi-modal and hybrid AI approach. It can process text directly, extract text from images using OCR, and convert speech to text using speech recognition. For detection, it combines rule-based methods (regex filtering), sentiment analysis (TextBlob), and deep learning (BERT) to generate a toxicity score. Based on this score, the system can automatically warn users, filter, or block content. It also includes a suggestion module that recommends non-toxic alternative expressions to encourage positive communication.

Prior research shows that traditional machine learning methods (like SVMs) lack contextual understanding, while deep learning models (LSTM, CNN, BERT) improve accuracy but face challenges such as high computational cost, limited scalability, and lack of multi-modal support. The proposed system aims to overcome these limitations by integrating multiple techniques into a unified framework.

The system architecture consists of multiple layers: input processing (text, image, audio), preprocessing (cleaning and tokenization), detection (hybrid AI models), application (moderation actions and suggestions), and data storage

Conclusion

The presented paper describes an AI-based automated toxicity detection system intended to enhance the quality and safety of online communication through intelligent moderation. The system integrates the processing of multi-modal inputs, pre-processing of text, and a hybrid detection framework combining Regex-based filtering with sentiment analysis and BERT-based contextual understanding to facilitate accurate detection of both explicit and implicit toxicity, including contextually dependent and nuanced expressions. In conclusion, the proposed system offers an effective, scalable, and intelligent method for addressing the increasingly complex issue of toxic content in digital environments and emphasizes the value of AI-based monitoring systems in improving user experiences and providing safe online interactions.

References

[1] T. Davidson, D. Warmsley, M. Macy, and I. Weber, “Automated hate speech detection and the problem of offensive language,” Proc. International AAAI Conference on Web and Social Media (ICWSM), 2017. [2] P. Badjatiya, S. Gupta, M. Gupta, and V. Varma, “Deep learning for hate speech detection in tweets,” Proc. World Wide Web Companion (WWW), pp. 759–760, 2017 [3] J. Devlin, M. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” Proc. NAACL-HLT, pp. 4171–4186, 2019. [4] Z. Zhang, D. Robinson, and J. Tepper, “Detecting hate speech on social media using convolutional neural networks,” Proc. European Semantic Web Conference (ESWC), 2018.M [5] M. Mozafari, R. Farahbakhsh, and N. Crespi, “Hate speech detection and racial bias mitigation in social media based on BERT model,” Proc. EMNLP Workshops, 2019. [6] B. Mathew, P. Saha, S. Yimam, C. Biemann, P. Goyal, and A. Mukherjee, “HateXplain: A benchmark dataset for explainable hate speech detection,” Proc. AAAI Conference on Artificial Intelligence, vol. 35, no. 17, pp. 14867–14875, 2020. [7] D. Noever, “Machine learning suites for online toxicity detection,” IEEE Access, vol. 6, pp. 1–10, 2018. [8] F. Del Vigna, A. Cimino, F. Dell’Orletta, M. Petrocchi, and M. Tesconi, “Hate me, hate me not: Hate speech detection on Facebook,” Proc. Italian Conference on Computational Linguistics, 2017.

Copyright

Copyright © 2026 Reshma E, Shalini J, Bharathy G. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.