Plant Identification and Query System Using Deep Learning and RAG-Based Search

Abstract

In this project, we present a dual-module system that utilizes a Convolutional Neural Network (CNN) for leaf-based plant identification and integrates a Retrieval-Augmented Generation (RAG) system for answering plant-related queries. The CNN was trained and fine-tuned using the LeafSnap dataset comprising 184 plant classes, achieving a final accuracy of 88%. Additionally, the RAG module leverages LangChain tools incorporating Wikipedia, Arxiv, and a custom PDF retriever for context-aware plant information search. This end-to-end solution is implemented using Flask and Streamlit, offering users a seamless interface to either upload plant leaf images or ask questions.

Introduction

???? Project Overview

This project automates plant species identification using a Convolutional Neural Network (CNN) trained on leaf images and integrates a smart assistant using retrieval-augmented generation (RAG) to answer botanical queries.

???? System Components

1. Plant Identification Engine:

   • Built with Flask

   • Uses a CNN model trained on the LeafSnap dataset

   • Classifies uploaded leaf images into plant species

   • Retrieves metadata from a local CSV database

2. Question-Answering Assistant:

   • Built with Streamlit + LangChain

   • Accepts natural language queries

   • Leverages Wikipedia, Arxiv, and uploaded PDFs

   • Uses sentence-transformers and FAISS for semantic search

????? Model Details

• CNN trained for 20 epochs

• Dataset: LeafSnap (184 plant species, images resized to 224×224 px)

• Final Accuracy: 88%

• Final Validation Loss: ~0.9284

• Model Size: ~13 million parameters

?? Technologies Used

• TensorFlow – Model training/inference

• Pillow – Image processing

• Pandas – Metadata handling

• LangChain + Streamlit – RAG-based query system

• FAISS – Vector-based search

• Wikipedia/Arxiv APIs – External knowledge retrieval

???? Evaluation

• The CNN model converged well during training

• Frontend is user-friendly and enables:

  • Leaf image upload and plant identification

  • Natural language Q&A on plant science topics

Conclusion

This project successfully demonstrates a unified plant identification and query system using CNNs and transformer-based retrieval. Future work includes developing a mobile application and expanding the training dataset to include more plant features such as flowers or stems.

References

[1] N. Kumar et al., \'Leafsnap: A Computer Vision System for Automatic Plant Species Identification\', ECCV 2012. [2] IJCRT2304306, \'Plant Identification System Using Machine Learning\', IJCRT Vol. 11, Issue 4, April 2023. [3] https://www.kaggle.com/datasets/xhlulu/leafsnap-dataset [4] LangChain Documentation: https://docs.langchain.com/

Copyright

Copyright © 2025 Abhishek Ganjigatti, Adarsh Gayadolli, Chaitali Joshi, Manoj Shekhar. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.