Deep Learning Based Doctors Handwritten Prescription Recognition and Nearby Pharmacy Recommendation System

Abstract

Handwritten prescriptions remain prevalent in healthcare settings, yet their interpretation is often hindered by poor handwriting, abbreviations, incomplete medicine names, and image quality issues. This paper proposes an intelligent healthcare support framework for automated handwritten prescription recognition and nearby pharmacy recommendation. The system preprocesses prescription images, detects text regions using the CRAFT mechanism in EasyOCR, and recognizes handwritten medicine names through the Transformer-based TrOCR model. Extracted text is validated and corrected using a medicine database and RapidFuzz similarity matching. The validated medicines are presented in a structured format, while geolocation services identify nearby pharmacies and provide interactive map-based navigation. The proposed framework integrates prescription digitization, medicine validation, pharmacy recommendation, and route guidance within a unified platform, aiming to reduce prescription interpretation errors, improve healthcare accessibility, and enhance patient convenience in medicine acquisition.

Introduction

This research proposes an AI-powered Handwritten Prescription Recognition and Nearby Pharmacy Recommendation System that automates the interpretation of handwritten medical prescriptions and helps users locate nearby pharmacies. The system addresses the limitations of traditional OCR, which often struggles with illegible handwriting, abbreviations, inconsistent writing styles, and complex medical terminology.

Motivation

Handwritten prescriptions remain common in healthcare, but poor handwriting can lead to medication errors and delays. Existing solutions typically treat prescription recognition and pharmacy search as separate tasks, creating fragmented user experiences. This study aims to integrate both functionalities into a single platform.

Objectives

The system is designed to:

• Recognize handwritten medicine names from prescription images.
• Validate and correct recognized medicine names using fuzzy matching.
• Recommend nearby pharmacies based on the user's location.
• Provide map-based navigation to selected pharmacies.

Proposed Methodology

The framework consists of several AI and location-based components:

1. Image Preprocessing (OpenCV): Enhances prescription images through grayscale conversion, noise removal, thresholding, and contrast enhancement.
2. Text Detection (EasyOCR + CRAFT): Detects handwritten text regions within the prescription.
3. Handwriting Recognition (TrOCR): Uses a Transformer-based OCR model to convert handwritten medicine names into text.
4. Medicine Validation (RapidFuzz): Corrects OCR errors by comparing recognized words with a medicine database using fuzzy string matching and Levenshtein distance.
5. Pharmacy Recommendation (Geoapify API): Finds and ranks nearby pharmacies using the user's current location.
6. Navigation Support: Displays recommended pharmacies on an interactive map with route guidance.

Literature Review

Previous research has shown strong performance in handwritten OCR using Transformer-based models like TrOCR and multimodal frameworks such as MIRAGE. However, most existing studies focus only on prescription recognition or medicine extraction. Few integrate recognition, validation, pharmacy recommendation, and navigation into a unified healthcare platform.

Experimental Results

The system was evaluated on handwritten prescriptions using standard OCR metrics:

• Accuracy: 80.00%
• Precision: 62.59%
• Recall: 50.29%
• F1-Score: 55.77%

The confusion matrix demonstrated relatively few recognition errors, indicating that combining OpenCV preprocessing, EasyOCR, TrOCR, and RapidFuzz improves handwritten medicine recognition.

System Demonstration

The web application supports the complete workflow:

• Upload prescription image.
• Select the prescription region (ROI).
• Recognize and validate medicine names.
• Recommend nearby pharmacies.
• Provide interactive map-based navigation to the selected pharmacy.

Conclusion

This paper presented a Deep Learning-Based Handwritten Prescription Recognition and Nearby Pharmacy Recommendation System that automates the recognition of handwritten medicine names and assists users in locating nearby pharmacies. The proposed framework integrates OpenCV-based image preprocessing, EasyOCR with the CRAFT text detection model, the transformer-based TrOCR recognition model, RapidFuzz similarity matching, and geolocation services to provide an end-to-end healthcare assistance solution. Experimental results demonstrate that the proposed system effectively recognizes handwritten medicine names while reducing OCR errors through medicine validation and correction. Furthermore, the integration of nearby pharmacy recommendation and map-based navigation enhances user convenience by enabling quick access to prescribed medicines. Overall, the proposed system provides an efficient, accurate, and practical solution for handwritten prescription interpretation, improving healthcare accessibility and reducing the effort required for manual prescription reading.

References

[1] Li, M., Lv, T., Chen, J., Cui, L., Lu, Y., Florencio, D., Zhang, C., Li, Z., & Wei, F., TrOCR: Transformer-Based Optical Character Recognition with Pre-trained Models, Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37, No. 11, pp. 13094–13102, 2023, https://doi.org/10.1609/aaai.v37i11.26538 [2] Baek, Y., Lee, B., Han, D., Yun, S., & Lee, H., Character Region Awareness for Text Detection (CRAFT),Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9365–9374, 2019, https://doi.org/10.1109/CVPR.2019.00959 [3] Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., et al., Transformers: State-of-the-Art Natural Language Processing,Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP), pp. 38–45, 2020, https://doi.org/10.18653/v1/2020.emnlp-demos.6 [4] He, K., Zhang, X., Ren, S., & Sun, J., Deep Residual Learning for Image Recognition, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778, 2016, https://doi.org/10.1109/CVPR.2016.90. [5] Hochreiter, S., &Schmidhuber, J., Long Short-Term Memory, Neural Computation, Vol. 9, No. 8, pp. 1735–1780, 1997, https://doi.org/10.1162/neco.1997.9.8.1735 [6] Karatzas, D., Shafait, F., Uchida, S., Iwamura, M., Gomez-Bigorda, L., Robles, S., Mas, J., et al., ICDAR 2013 Robust Reading Competition,Proceedings of the 12th International Conference on Document Analysis and Recognition (ICDAR), pp. 1484–1493, 2013, https://doi.org/10.1109/ICDAR.2013.216

Copyright

Copyright © 2026 Gudumala Meghana, Dr. V Kamakshi Prasad. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.