PACCF: Prescription-Aware Adaptive Confidence Fusion for Robust Medical Prescription OCR

Abstract

The digitization of handwritten medical prescriptions remains a challenging task due to poor handwriting, variations in writing styles, image quality issues, and the presence of specialized pharmaceutical terminology. Although recent Optical Character Recognition (OCR) technologies have achieved significant progress in document understanding, their performance on medical prescriptions is often limited because most systems rely on a single recognition model and are therefore vulnerable to model-specific errors. To address these challenges, this paper proposes a novel Prescription-Aware Adaptive Confidence Fusion (PACCF) framework for accurate and reliable medical prescription text recognition. The proposed approach combines the strengths of multiple OCR engines, namely TrOCR, PARSeq, EasyOCR, and PaddleOCR, to generate diverse recognition hypotheses from prescription text regions. Instead of selecting predictions based solely on confidence scores, PACCF introduces an adaptive fusion strategy that evaluates each candidate using four complementary reliability measures: confidence reliability, inter-model agreement, Gaussian consensus reliability, and medical lexical similarity. These factors are dynamically weighted through a Softmax-based mechanism, enabling data-driven decision making without manually tuned parameters. To further enhance recognition accuracy, a BK-Tree-based medical validation module is employed for efficient approximate matching and correction of pharmaceutical terms. Experimental results on a challenging handwritten prescription dataset demonstrate that the proposed framework significantly improves both character-level and word-level recognition performance compared with individual OCR models and conventional fusion techniques. The findings suggest that integrating ensemble diversity, adaptive confidence modeling, and domain-specific medical knowledge provides an effective and scalable solution for automated prescription digitization and healthcare document processing.

Introduction

The text presents a research framework called Prescription-Aware Adaptive Confidence Fusion (PACCF) for improving medical prescription text recognition. The work addresses the limitations of existing OCR-based prescription digitization systems, which often fail due to handwritten text, poor image quality, inconsistent writing styles, and medical terminology complexity.

Current OCR systems, including deep learning and transformer-based models, have improved document recognition but commonly depend on a single OCR engine or use fixed ensemble strategies. These methods may produce incorrect but highly confident predictions and usually do not integrate medical knowledge during recognition.

The proposed PACCF framework introduces a multi-OCR ensemble approach using TrOCR, PARSeq, EasyOCR, and PaddleOCR to generate multiple recognition candidates. These predictions are combined using an adaptive fusion mechanism based on four reliability factors:

1. Confidence Reliability – evaluates the OCR model’s certainty.
2. Hybrid Agreement – measures similarity between OCR outputs using edit distance and character n-gram comparison.
3. Gaussian Consensus Reliability – estimates how closely a prediction matches overall OCR confidence patterns.
4. Medical Lexical Similarity – checks whether predictions match valid pharmaceutical terms.

Unlike traditional fusion techniques with manually assigned weights, PACCF uses an adaptive Softmax-based weighting mechanism to dynamically determine the importance of each factor. A BK-Tree-based medical validation module is also included to efficiently search and correct medicine names from large pharmaceutical vocabularies.

The framework pipeline consists of:

• Prescription image preprocessing
• Text detection and region extraction
• Multi-OCR candidate generation
• PACCF adaptive confidence fusion
• Medical dictionary validation and correction
• Final corrected prescription output

The major contributions include:

• A heterogeneous multi-OCR architecture combining different OCR models.
• A novel adaptive fusion algorithm combining confidence, agreement, statistical reliability, and medical knowledge.
• A dynamic Softmax weighting strategy without manual parameter tuning.
• Efficient medicine-name correction using BK-Tree search.
• Experimental evaluation showing improved robustness over existing OCR methods.

Conclusion

This paper presented the Prescription-Aware Adaptive Confidence Fusion (PACCF) framework for robust medical prescription text recognition under challenging real-world conditions. The proposed framework addresses the limitations of conventional single-engine OCR systems by integrating a heterogeneous OCR ensemble consisting of TrOCR, PARSeq, EasyOCR, and PaddleOCR within a unified adaptive fusion architecture. Unlike traditional confidence-based voting methods, PACCF incorporates four complementary reliability indicators—Confidence Reliability, Hybrid Agreement, Gaussian Consensus Reliability, and Medical Lexical Similarity—to evaluate candidate predictions and perform adaptive decision fusion. To further enhance domain-specific recognition accuracy, a BK-Tree-based medical validation module was introduced for efficient approximate retrieval and correction of pharmaceutical terminology. The proposed validation strategy enables scalable medicine matching while reducing retrieval complexity from (O(D)) to (O(\\log D)), thereby improving computational efficiency for large pharmaceutical vocabularies. Experimental evaluation conducted on a challenging dataset of 137 handwritten prescription images demonstrated the effectiveness of the proposed framework. PACCF achieved a Character Recognition Accuracy of 93.86% and a Word Recognition Accuracy of 78.52%, substantially outperforming all individual OCR engines. Ablation studies further confirmed the contribution of adaptive fusion and domain-aware validation components, while qualitative analysis demonstrated the framework\'s ability to recover medically plausible predictions from highly inconsistent OCR outputs. Overall, the results indicate that combining ensemble diversity, adaptive confidence modeling, statistical reliability estimation, and medical-domain knowledge provides a practical and effective solution for prescription digitization. The proposed framework offers a promising approach for healthcare document processing applications where recognition accuracy and reliability are critical. Future work will focus on expanding the pharmaceutical knowledge repository, incorporating contextual prescription understanding, and investigating learning-based fusion mechanisms capable of automatically optimizing reliability relationships from data. Although the proposed framework demonstrated strong performance on a dataset of 137 handwritten prescription images, further evaluation on larger and more diverse prescription datasets is required to comprehensively assess its generalization capability across different handwriting styles, imaging conditions, and clinical settings. Additional research may explore lightweight deployment architectures, multilingual prescription recognition, and multimodal healthcare document understanding systems to support large-scale clinical digitization environments.

References

Core OCR and Transformer References [1] M. Li et al., \"TrOCR: Transformer-Based Optical Character Recognition with Pre-Trained Models,\" Proc. AAAI Conf. Artif. Intell., vol. 37, no. 11, pp. 13094–13102, 2023. [2] D. Bautista and R. Atienza, \"Scene Text Recognition with Permuted Autoregressive Sequence Models,\" in Proc. Eur. Conf. Comput. Vis. (ECCV), 2022. [3] A. Vaswani et al., \"Attention Is All You Need,\" in Adv. Neural Inf. Process. Syst. (NeurIPS), vol. 30, 2017. [4] A. Dosovitskiy et al., \"An Image Is Worth 16×16 Words: Transformers for Image Recognition at Scale,\" in Proc. Int. Conf. Learn. Represent. (ICLR), 2021. [5] A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, \"Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks,\" in Proc. ICML, pp. 369–376, 2006. [6] R. Smith, \"An Overview of the Tesseract OCR Engine,\" in Proc. ICDAR, pp. 629–633, 2007. [7] T. Coquenet, C. Chatelain, and T. Paquet, \"End-to-End Handwritten Paragraph Text Recognition Using a Vertical Attention Network,\" IEEE Trans. Pattern Anal. Mach. Intell., vol. 45, no. 1, pp. 508–524, 2023. Medical Prescription OCR [8] U. Ali et al., \"Leveraging Deep Learning with Multi-Head Attention for Accurate Extraction of Medicine from Handwritten Prescriptions,\" arXiv preprint arXiv:2412.18199, 2024. [9] H. Ullah et al., \"Enhancing Handwritten Prescription Recognition with AI-Driven OCR Frameworks,\" J. Comput. Biomed. Informat., 2025. [10] T. Mankash et al., \"MIRAGE: Multimodal Identification and Recognition of Annotations in Indian General Prescriptions,\" arXiv preprint arXiv:2410.09729, 2024. [11] S. R. Gudi, \"Enhancing OCR Accuracy in Healthcare Prescription Processing Using Artificial Neural Networks,\" Eur. J. Artif. Intell. Mach. Learn., vol. 4, no. 6, 2025. OCR Ensemble and Fusion [12] A. Goel and A. Kaushik, \"Prescription-to-Text Conversion Using Ensemble Faster R-CNN and OCR,\" 2023. [13] E. Hassan et al., \"Medical Prescription Recognition Using Machine Learning,\" 2021. [14] J. Kittler, M. Hatef, R. P. W. Duin, and J. Matas, \"On Combining Classifiers,\" IEEE Trans. Pattern Anal. Mach. Intell., vol. 20, no. 3, pp. 226–239, 1998.

Copyright

Copyright © 2026 Manish Joshi. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.