AR-Powered Retail Assistant Using YOLOv8 and LLM Integration

Abstract

In most physical retail settings, customers rely mainly on product labels or search on their phones to understand product details, which can interrupt the overall shopping flow. This project explores a simple approach to improving that experience by combining computer vision, augmented reality, and a basic conversational system. A lightweight YOLOv8 model is used to recognize selected retail products through a mobile camera. When a product is identified, key information such as price, features, and ingredients is shown directly on the screen as an overlay. The system also includes a small question–answer feature where users can ask about the detected product and receive responses from a locally running language model. Everything is designed to run in a browser, so there is no need for installation. To make the system feel complete, a basic cart and payment simulation is also included. During testing, the model was able to detect the selected products with good consistency and respond within a short time under normal conditions. While the system is limited to a small set of products, it shows how combining these technologies can make everyday shopping a bit more interactive and easier to navigate.

Introduction

The text describes a simple AI-based smart retail system designed to improve the in-store shopping experience by bridging the gap between online and offline shopping. While online shopping provides detailed product information and comparisons, physical stores lack easy access to such data, often relying on labels, QR codes, or manual searching. The proposed system addresses this by using a browser-based application that combines real-time product detection, information display, a question–answer feature, and a basic shopping cart.

The system uses a camera to capture product images and applies a YOLOv8-based object detection model to identify retail items such as shampoo, biscuits, oats, and noodles. Once detected, product details stored in a JSON database are displayed as an overlay on the screen. A lightweight LLM (LLaMA 3.1 via Ollama) is used to answer product-related queries, while a simple cart feature simulates shopping actions.

The methodology includes dataset creation, model training, real-time detection workflow, backend processing, and frontend visualization. The system runs entirely in a web browser without requiring installation, making it lightweight and accessible.

Experimental results show good performance, with high detection accuracy (mAP50 ~0.96, precision ~0.94, recall ~0.87). YOLOv8 outperformed SSD and EfficientDet in reliability, though it was slightly slower. The system works well under normal conditions but shows reduced accuracy under poor lighting or blurred images.

Conclusion

In this project, a simple system was developed to improve the in-store shopping experience using object detection and a basic augmented display. The system is able to detect selected retail products using a YOLOv8 model and show relevant details directly on the screen. and also a small feature is added where users can ask questions about the detected product and get a response. The system was tested with a small dataset, and the results were quite good for the selected products. The detection worked well in normal conditions, and the full flow from capturing the image to showing the information was smooth. A basic cart feature was also added to show how the system can be extended later. There are still a few limitations. For example, in low lighting or when the image is not clear, the detection sometimes doesn’t work properly. Also, right now the system supports only a limited number of products. Overall, this work shows that combining object detection with simple user interaction can make shopping a bit easier and more interactive. It is not a full commercial system, but it still gives a good idea of how these technologies can be used together in real-world use.

References

[1] Javornik D., Augmented reality in retail and its impact on customer experience, 2016 https://www.sciencedirect.com/science/article/pii/S0969698916300473 [2] Poushneh A., Vasquez-Parraga C., Impact of augmented reality on shopping experience, 2017 https://www.sciencedirect.com/ science/ article/pii/S0969698916302782 [3] Hilken J. et al., Use of augmented reality in retail environments, 2017 https://www.sciencedirect.com/science/article/pii/S0022435916300786 [4] Pantano E. et al., Smart technologies in retail, 2017 https://link.springer.com/book/10.1007/978-3-319-44263-0 [5] Redmon J. et al., YOLO: real-time object detection system, 2016 https://arxiv.org/abs/1506.02640 [6] Ollama, running LLM models locally https://ollama.com

Copyright

Copyright © 2026 Roshan Raj N, Mahima Jane E. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.