End-to-End Handwritten Malayalam to English Translation: A Deep Learning Implementation

Abstract

Bridging the gap between handwritten regional language documents and automated English translation remains a genuinely difficult problem—particularly for morphologically complex, low-resource scripts like Malayalam. The challenge goes beyond simple recognition: handwritten Malayalam exhibits tightly coupled ligatures, circular stroke patterns, and high inter-writer variability that together defeat most off-the-shelf OCR tools. This paper describes an end-to-end deep learning pipeline we built and deployed to address exactly this problem. The architecture works in four stages: a fine-tuned YOLOv8 model localizes individual handwritten words, a custom ResNetCRNN with Bidirectional LSTMs and CTC decoding performs character-level recognition, a KenLM language model combined with SymSpell post-processing corrects phonetic ambiguities, and Meta’s NLLB-200 transformer handles the final Malayalam-toEnglish translation. The system is delivered as a containerized web application built on FastAPI and React, supporting real-time inference with asynchronous batch processing. Evaluated on a robust test set of 19,680 handwritten samples, the OCR component achieved a Character Error Rate (CER) of 1.20% and a Word Error Rate (WER) of 7.30%, with 92.7% of predictions being exact matches. These results suggest the pipeline is practically viable for digitizing and translating unconstrained handwritten Malayalam at scale.

Introduction

The text discusses the challenge of digitizing handwritten Malayalam documents and proposes a complete end-to-end system for converting handwritten Malayalam images into English translations. While progress in OCR and machine translation has improved handling of printed text and high-resource languages, handwritten Malayalam remains difficult due to its complex, connected script, variability in handwriting styles, and the presence of ligatures that cannot be easily segmented into individual characters.

Existing OCR systems struggle with these issues, and although separate OCR and translation tools exist, they are rarely integrated into a unified pipeline. To address this gap, the paper introduces an end-to-end framework that directly processes handwritten images and outputs English translations with minimal manual intervention.

The proposed system includes multiple stages: image preprocessing using OpenCV to correct distortion and lighting issues; text localization using a YOLOv8-based detector to identify word regions and reconstruct reading order; OCR using a ResNet-CRNN-BiLSTM model with CTC decoding to recognize handwritten text without explicit character segmentation; linguistic post-processing using KenLM and SymSpell to correct errors; and final translation using Meta’s NLLB-200 neural machine translation model. The system is deployed as a full-stack web application with a React frontend and FastAPI backend.

Conclusion

We have presented an end-to-end pipeline for handwritten Malayalam document recognition and translation that achieves high accuracy while remaining practical to deploy. The modular architecture—YOLOv8 for word detection, ResNet-CRNNBiLSTM-CTC for recognition, KenLM/SymSpell for postprocessing, and NLLB-200 for translation—bridges a significant gap in regional language digitization. Future directions include extending the training corpus to encompass heavily degraded historical palm-leaf manuscripts. Additionally, exploring Vision-Language Models (VLMs) as a unified recognition and translation backbone—potentially skipping the separate OCR stage—is a promising avenue. Finally, optimizing the transformer weights via quantization could reduce the model footprint, making edge deployment on mobile devices feasible.

References

[1] K. B. Baiju, T. S. Sabna, and V. L. Lajish, “Segmentation of Malayalam Handwritten Characters into Pattern Primitives and Recognition using SVM,” Int. J. Eng. Adv. Technol. (IJEAT), vol. 9, no. 3, pp. 1817–1821, Feb. 2020. [2] K. Manjusha, M. A. Kumar, and K. P. Soman, “On Developing Handwritten Character Image Database for Malayalam Language Script,” Engineering Science and Technology, an International Journal, vol. 22, no. 2, pp. 637–645, 2019. [3] V. K. Vaisakh and B. D. Lyla, “Handwritten Malayalam Character Recognition System Using Artificial Neural Networks,” in Proc. IEEE Int. Students’ Conf. Electrical, Electronics and Computer Science (SCEECS), 2020. [4] S. Anish and V. Preeja, “A Novel Method on Malayalam Handwritten Character Recognition based on Texture Extraction,” Int. J. Eng. Adv. Technol. (IJEAT), vol. 4, no. 6, pp. 234–239, Aug. 2015. [5] M. A. Rahiman and M. S. Rajasree, “An Efficient Character Recognition System for Handwritten Malayalam Characters Based on Intensity Variations,” Int. J. Comput. Theory Eng., vol. 3, no. 3, pp. 369–373, 2011. [6] P. Bhise, R. Singh, V. Kulathunkal, S. Shirgaonkar, and N. Mokal, “Leveraging OCR alongside Machine Translation Techniques: Image-toText System Integrating OCR, Translation, Summarization, and Q&A,” Sirjana Journal, vol. 54, no. 3, pp. 191–198, 2021. [7] R. Anitha, R. R. Rajeev, M. Nazeem, and S. Navaneeth, “Open Source OCR Libraries: A Comprehensive Study for Low Resource Language,” ICFOSS, Govt. of Kerala, 2023. [8] D. Sudarsan and D. Sankar, “An Ensemble Neural Network Model for Malayalam Character Recognition from Palm Leaf Manuscripts,” ACM Trans. Asian and Low-Resource Language Information Processing, vol. 23, no. 8, Aug. 2024. [9] E. Lalitha, A. Mondal, and C. V. Jawahar, “Enhancing Accuracy in Indic Handwritten Text Recognition,” in Proc. Conf. Computer Vision for Indic Languages (CVIP), 2024. [10] B. Jose and K. P. Pushpalatha, “Intelligent Handwritten Character Recognition for Malayalam Scripts Using Deep Learning Approach,” IOP Conf. Ser.: Materials Science and Engineering, vol. 1085, 012022, 2021. [11] D. Keysers et al., “The Architecture of a Multi-Script and MultiLanguage Online Handwriting Recognition System,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 39, no. 6, pp. 1180–1195, 2017. [12] S. P. Salim, A. James, P. Simon, and B. N. Divakaran, “Multiscale Residual Network for Recognizing Handwritten Malayalam Characters,” Traitement du Signal, vol. 41, no. 1, pp. 421–430, 2024. [13] H. Choudhary, S. Rao, and R. Rohilla, “Neural Machine Translation for Low-Resourced Indian Languages,” in Proc. 12th Conf. Language Resources and Evaluation (LREC), Marseille, France, 2020. [14] A. Hatami, S. Banerjee, M. Arcan, P. Buitelaar, and J. P. McCrae, “English-to-Low-Resource Translation: A Multimodal Approach for Hindi, Malayalam, Bengali, and Hausa,” in Proc. ACL, 2024. [15] Y. Li, D. Chen, T. Tang, and X. Shen, “HTR-VT: Handwritten Text Recognition with Vision Transformer,” Pattern Recognition, 2024. [16] P. P. Nair, A. James, P. Simon, and B. P. V. Bhagyasree, “Malayalam Handwritten Character Recognition using CNN Architecture,” Indonesian J. Electr. Eng. Informatics (IJEEI), vol. 11, no. 3, pp. 764–777, Sept. 2023. [17] C. Anaswara, C. Swetha, and S. Unnikrishnan, “Scene Image to Text Recognition in Malayalam App,” Int. J. Creative Research Thoughts (IJCRT), vol. 12, no. 5, May 2024. [18] Prathwini, A. P. Rodrigues, P. Vijaya, and R. Fernandes, “Tulu Language Text Recognition and Translation,” IEEE Access, vol. 12, pp. 12734– 12745, Jan. 2024. [19] A. Vaidya, T. Prabhakar, D. George, and S. Shah, “Analysis of Indic Language Capabilities in LLMs,” MLCommons AI Luminate Report, 2025. [20] V. Mujadia et al., “Assessing Translation Capabilities of Large Language Models involving English and Indian Languages,” in Proc. LTRC, IIIT Hyderabad, 2023. [21] J. Joseph and A. Kurian, “Breaking Barriers: Transformer-Based Summarization and Translation of English Legal Documents to Malayalam,” in Proc. IEEE 7th Int. Conf. Contemporary Computing and Informatics (IC3I), pp. 590–595, 2024. [22] P. V. Pearlsy and D. Sankar, “Malayalam Handwritten Character Recognition using Transfer Learning and Fine Tuning of Deep Convolutional Neural Networks,” in Proc. IEEE ACCESS Conf., 2023. [23] A. S. Kolavi, S. P., and V. Jain, “Nayana OCR: A Scalable Framework for Document OCR in Low-Resource Languages,” in Proc. 1st Workshop on Language Models for Underserved Communities (LM4UC 2025), pp. 86–103, May 2025. [24] B. Premjith, M. A. Kumar, and K. P. Soman, “Neural Machine Translation System for English to Indian Language Translation Using MTIL Parallel Corpus,” J. Intell. Syst., vol. 28, no. 3, pp. 387–398, 2019. [25] A. P. G. Anisree and R. K. T. Radhika, “Malayalam to English Machine Translation: A Hybrid Approach,” Int. J. Innovative Research in Science, Engineering and Technology (IJIRSET), vol. 5, no. 7, pp. 12604–12610, July 2016. [26] S. Sreelekha and P. Bhattacharyya, “A Case Study on EnglishMalayalam Machine Translation,” arXiv preprint arXiv:1702.08217, 2017. [27] A. Patil, I. Joshi, and D. Kadam, “PICT@WAT 2022: Neural Machine Translation Systems for Indic Languages,” in Proc. 9th Workshop on Asian Translation (WAT 2022), 2022. [28] A. George, “English to Malayalam Statistical Machine Translation System,” Int. J. Eng. Research and Technology (IJERT), vol. 2, no. 7, pp. 230–234, 2013. [29] L. R. Nair, D. P. S., and R. P. Ravindran, “Design and Development of a Malayalam to English Translator: A Transfer Based Approach,” Int. J. Computational Linguistics, vol. 3, 2012. [30] N. B. Nithya and S. Joseph, “A Hybrid Approach to English to Malayalam Machine Translation,” Int. J. Computer Applications, vol. 81, no. 8, 2013. [31] G. Jocher, A. Chaurasia, and J. Qiu, “Ultralytics YOLOv8,” 2023. [Online]. Available: https://github.com/ultralytics/ultralytics [32] K. Heafield, “KenLM: Faster and Smaller Language Model Queries,” in Proc. WMT, 2011. [33] NLLB Team, “No Language Left Behind: Scaling Human-Centered Machine Translation,” arXiv preprint arXiv:2207.04672, 2022. [34] M. Farhan, M. Nowfal K M, R. Raghav K R, Sabah K J, and S. Haridas, “A Review on Handwritten Malayalam to English Digitization and Translation,” International Journal for Research in Applied Science and Engineering Technology (IJRASET), vol. 13, no. 12, pp. 1908–1914, Dec. 2025.

Copyright

Copyright © 2026 Mohammed Farhan, Mohammed Nowfal K M, Radhesyam Raghav K R, Sabah K J, Riya K Prakash. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.