PhytoSense AR: Augmented-Reality Plant Disease Diagnosis Using a Field-Adapted Vision Transformer and Colour-Based Lesion Segmentation

Abstract

Plant diseases cause substantial crop losses, and smallholder farmers often lack timely access to expert diagnosis. Deep-learning classifiers trained on the PlantVillage dataset report near-perfect accuracy, yet they degrade sharply on photographs captured in the field. This paper presents PhytoSense AR, a free, browser-based diagnosis system that overlays disease information directly on the live camera view. The systemcombinesa colour-spaceleafandlesionsegmentationstage,basedonHSV thresholdsandtheExcess-Green index, witha VisionTransformer(ViT-B/16) classifier covering 38 classes across 14 crops. To narrow the gap between laboratory and field imagery, the classifier was fine-tuned on a mixture of PlantVillage and the PlantDoc field dataset with augmentation that simulates changes in zoom, orientation, illumination and focus, and was then exported to an 8-bit quantised ONNX model of 87 MB that runs on a CPU without PyTorch. On the PlantDoc test set of 236 unseen field images, top-1 accuracy rose from 37.7 % for the originalPlantVillage-onlymodelto76.7%forthedeployedquantisedmodel,whileaccuracyon760held-out PlantVillage images remained at 99.9 %. End-to-end processing, including segmentation and classification, takes about 200 ms per image on a laptop CPU. The system also estimates an infected-area severity and a health score, provides treatment guidance from a curated knowledge base, and offers a multilingual assistant. The results show that domain-adaptive fine-tuning on a modest field dataset, combined with lightweight deployment, makes AR-assisted plant diagnosis practical on commodity hardware.

Introduction

The paper presents PhytoSense AR, an AI-powered plant disease detection system designed to help farmers identify crop diseases quickly using smartphone cameras. It combines deep learning, computer vision, augmented reality (AR), and multilingual assistance to provide disease diagnosis, visual symptom localisation, severity estimation, and treatment recommendations.

1. Background and Problem Statement

Plant diseases significantly affect agricultural productivity, causing approximately 17–30% yield losses in major crops. Traditional disease identification depends on agricultural experts, who may not be readily available in rural areas.

Although existing AI models achieve over 99% accuracy on controlled laboratory images, their performance drops considerably in real agricultural environments because of:

• Complex backgrounds and overlapping leaves.

• Different lighting and camera conditions.

• Variations in disease symptoms.

• Lack of visual explanations and severity measurements.

• Dependence on expensive GPUs or paid cloud services.

An initial prototype achieved only 37.7% accuracy on the PlantDoc field dataset, highlighting the need for better real-world performance.

2. Proposed System – PhytoSense AR

PhytoSense AR is a zero-cost, installable Progressive Web Application (PWA) that identifies 38 healthy and diseased conditions across 14 crops.

Its main objectives are:

1. Identify plant diseases from leaf photographs.

2. Detect and highlight leaves and infected regions using AR overlays.

3. Estimate disease severity and calculate a health score.

4. Provide treatment and prevention recommendations.

5. Offer an interactive assistant supporting nine Indian languages.

6. Run on ordinary CPUs without requiring dedicated GPUs.

3. Literature Review

Previous research established deep learning as an effective approach to plant disease classification.

• PlantVillage: Enabled researchers to train CNN models with laboratory images, achieving accuracies above 99%.

• PlantDoc: Introduced real-world field images and demonstrated the challenges of applying laboratory-trained models in natural environments.

• CNN architectures: Models such as AlexNet, VGG, ResNet, and EfficientNet achieved high classification accuracy.

• Vision Transformers (ViT): Improved image understanding through attention mechanisms.

• MobileNet: Demonstrated the possibility of deploying disease detection on smartphones.

However, existing approaches often lack field reliability, visual localisation, severity estimation, and affordable deployment. PhytoSense AR attempts to combine these capabilities into one application.

4. System Architecture and Methodology

The system follows a three-tier architecture:

• Client Layer: JavaScript-based PWA that captures images, displays AR overlays, and provides diagnosis results.

• Application Layer: Python FastAPI backend that performs image segmentation, disease classification, severity calculation, and knowledge lookup.

• Data Layer: ONNX model, JSON disease knowledge base, and SQLite database for scan history.

The main processing steps are:

1. Image Acquisition: Captures leaf images through a smartphone or computer camera.

2. Leaf Segmentation: Uses HSV colour thresholds, Excess Green (ExG), and background modelling to identify the leaf.

3. Lesion Detection: Identifies diseased regions using colour, brightness, and texture characteristics.

4. Disease Classification: Uses a fine-tuned ViT-B/16 model to classify images into 38 categories.

5. Severity Estimation: Calculates the percentage of infected leaf area.

6. Health Scoring: Combines disease urgency and infection severity to generate a health score.

7. Recommendations: Provides disease information, treatment methods, and prevention strategies.

5. AI Model Training and Deployment

The project uses two datasets:

• PlantVillage for laboratory-based training images.

• PlantDoc for real-world field photographs.

The Vision Transformer is fine-tuned using data augmentation techniques such as rotation, flipping, cropping, colour adjustment, and Gaussian blur.

The trained model is converted into an 8-bit quantised ONNX model to reduce memory requirements and enable CPU-based inference.

Key deployment specifications:

6. Disease Severity and Health Scoring

The system calculates infection severity using the ratio of infected leaf area to total leaf area.

Three severity levels are defined:

• Early: Less than 5% infection.

• Moderate: 5–20% infection.

• Severe: 20% or more infection.

The health score ranges from 5 to 95 for diseased leaves, depending on disease urgency and infection severity. Healthy leaves receive a score of 100, or 90 when significant discolouration is detected.

7. Multilingual Assistant and AR Features

The application provides:

• Live camera-based disease detection with AR overlays.

• Visual highlighting of leaf boundaries and infected regions.

• Disease history and health-trend charts.

• A searchable disease information library.

• Chemical and organic treatment recommendations.

• Voice input and read-aloud functionality.

• An interactive assistant supporting English, Hindi, Marathi, Tamil, Telugu, Kannada, Bengali, Gujarati, and Punjabi.

The assistant uses Google Gemini when configured and falls back to the local knowledge base when internet access or an API key is unavailable.

8. Experimental Results

The fine-tuned model was evaluated using independent PlantDoc field images and held-out PlantVillage laboratory images.

Major improvement

Other important findings:

• Field accuracy increased from 37.7% to 76.7%.

• Quantisation did not cause a measurable accuracy reduction.

• Fine-tuning required approximately 25 minutes on a free Google Colab GPU.

• Average inference time was approximately 200 milliseconds.

• All eight automated software tests passed.

Some diseases, such as corn gray leaf spot, potato early blight, and tomato bacterial spot, remained difficult to identify because their symptoms resemble other diseases.

9. Limitations

The project has several limitations:

• PlantDoc contains relatively few images, limiting field evaluation.

• Visually similar diseases can produce classification errors.

• Colour-based segmentation may struggle with unusual lighting and backgrounds.

• Per-class accuracy estimates have considerable uncertainty because of small test samples.

• The health score and lesion-area estimates depend on segmentation quality.

• The assistant's advanced responses depend on an external API when configured.

Conclusion

ThispaperpresentedPhytoSenseAR,afreeAR-basedsystemforplantdiseasediagnosisthatcombinestraining-free leaf and lesion segmentation, a field-adapted Vision Transformer, severity and health scoring, a treatment knowledge base and a multilingual assistant. Fine-tuning on PlantVillage and PlantDoc with field-style augmentation raised top-1 accuracy on unseen field photographs from 37.7 % to 76.7 % while keeping laboratory accuracy at 99.9 %, and int8 quantisation allowed the whole system to run on a CPU in about 200 ms per image within a free cloud instance. These results show that practical, explainable plant diagnosis does not require expensive hardware or paid services.

References

[1] S. Savary, L. Willocquet, S. J. Pethybridge, P. Esker, N. McRoberts, and A. Nelson, \"The global burden of pathogens and pests on major food crops,\" Nature Ecology & Evolution, vol. 3, no. 3, pp. 430–439, 2019. [2] D. P. Hughes and M. Salathé, \"An open access repository of images on plant health to enable the development of mobile disease diagnostics,\" arXiv preprint arXiv:1511.08060, 2015. [3] S. P. Mohanty, D. P. Hughes, and M. Salathé, \"Using deep learning for image-based plant disease detection,\" Frontiers in Plant Science, vol. 7, art. no. 1419, 2016. [4] K.P.Ferentinos,\"Deeplearningmodelsforplantdiseasedetectionanddiagnosis,\"ComputersandElectronicsin Agriculture, vol. 145, pp. 311–318, 2018. [5] J.G.A.Barbedo,\"Factorsinfluencingtheuseofdeeplearningforplantdiseaserecognition,\"BiosystemsEngineering,vol. 172, pp. 84–91, 2018. [6] D. Singh, N. Jain,P. Jain,P. Kayal, S. Kumawat, andN. Batra,\"PlantDoc: Adatasetforvisualplantdisease detection,\" in Proc. 7th ACM IKDD CoDS and 25th COMAD, Hyderabad, India, 2020, pp. 249–253. [7] A. Dosovitskiyet al., \"An image is worth 16x16 words: Transformers for image recognition at scale,\" in Proc. Int. Conf. Learning Representations (ICLR), 2021. [8] B.Jacobet al.,\"Quantizationandtrainingofneural networksforefficientinteger-arithmetic-onlyinference,\"inProc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), 2018, pp. 2704–2713. [9] E.C.Too,L.Yujian,S.Njuki,andL.Yingchun,\"Acomparativestudyoffine-tuningdeeplearningmodelsforplant disease identification,\" Computers and Electronics in Agriculture, vol. 161, pp. 272–279, 2019. [10] Ü.Atila,M.Uçar,K.Akyol,andE.Uçar,\"PlantleafdiseaseclassificationusingEfficientNetdeeplearningmodel,\" Ecological Informatics, vol. 61, art. no. 101182, 2021. [11] J.G.A.Barbedo,\"Plantdiseaseidentificationfromindividuallesionsandspotsusingdeeplearning,\"Biosystems Engineering, vol. 180, pp. 96–107, 2019. [12] L. Li, S. Zhang, and B. Wang, \"Plant disease detection and classification by deep learning—A review,\" IEEE Access, vol. 9, pp. 56683–56698, 2021. [13] A. Ahmad, D. Saraswat, and A. El Gamal, \"A survey on using deep learningtechniques for plant disease diagnosis andrecommendations for development of appropriate tools,\" Smart Agricultural Technology, vol. 3, art. no. 100083, 2023. [14] A. Ramcharanetal.,\"Amobile-based deeplearning modelfor cassavadisease diagnosis,\" Frontiers in PlantScience, vol. 10, art. no. 272, 2019. [15] Y.Borhani, J.Khoramdel,and E.Najafi,\"A deeplearning basedapproachforautomatedplantdiseaseclassification using vision transformer,\" Scientific Reports, vol. 12, art. no. 11554, 2022. [16] P. S. Thakur, P. Khanna, T. Sheorey, and A. Ojha, \"Explainable vision transformer enabled convolutional neural network for plant disease identification: PlantXViT,\" arXiv preprint arXiv:2207.07919, 2022. [17] GeminiTeam,Google,\"Gemini:Afamilyofhighlycapablemultimodalmodels,\"arXivpreprintarXiv:2312.11805,2023. [18] D.M.Woebbecke,G.E.Meyer,K.VonBargen,andD.A.Mortensen,\"Colorindicesforweedidentificationunder various soil, residue, and lighting conditions,\" Transactions of the ASAE, vol. 38, no. 1, pp. 259–269, 1995. [19] C. Szegedy, V.Vanhoucke,S.Ioffe,J.Shlens,and Z. Wojna,\"Rethinking theInceptionarchitectureforcomputervision,\" in Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2016, pp. 2818–2826. [20] I.LoshchilovandF.Hutter,\"Decoupledweightdecayregularization,\"inProc.Int.Conf.LearningRepresentations (ICLR),2019. [21] Gupta, P., Sharma, V. and Varma, S., 2021. People detection and counting using YOLOv3 and SSD models. Materials Today: Proceedings. [22] Gupta, P., Shukla, M., Arya, N., Singh, U. and Mishra, K., 2022. Let the Blind See: An AIIoT-Based Device for Real-Time Object Recognition with the Voice Conversion. In Machine Learning for Critical Internet of Medical Things (pp. 177-198). Springer, Cham [23] Gupta, P. and Kulkarni, N., 2013. An introduction of soft computing approach over hard computing. International Journal of Latest Trends in Engineering and Technology (IJLTET), 3(1), pp.254-258 [24] Gupta, P., Sharma, V. and Varma, S., 2022. A novel algorithm for mask detection and recognizing actions of human. Expert Systems with Applications, 198, p.116823. [25] Gupta, P., Kulkarni, A. and Sarda, A., 2013. An embedded health care supervisory systems. International Journal of Latest Trends in Engineering and Technology (IJLTET), 3, pp.379-386

Copyright

Copyright © 2026 Sonu Airen, Puja Gupta, Yameesh Nayak , Suyash Gupta , Swapnil Pal , Neha Agrawal. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.