Privacy Preserving Local Retrieval-ALLM

Abstract

With the growing use of cloud-based artificial intelligence (AI) systems for document analysis, there are mounting concerns regarding user privacy. Certain documents may contain private, academic, organizational, or legally protected information that would need to be redacted or withheld before extracting meaningful information from them. This paper proposes Privacy-Preserving Local Retrieval-ALLM, a local document question-answering system capable of performing information retrieval and generating responses based on the contents of the documents. The system described in the paper uses the following process. First, it ingests user documents, performing optical character recognition if necessary, and segments them into text parts and embeddings. Next, the system stores the embeddings in a local vector database for similarity search. The system then employs a local Qwen 2.5 3B Instruct model to generate responses from the retrieved information. Finally, the system provides an optional permission-based web search if the retrieved document does not have the necessary information. In such cases, the solution does not need to transmit the contents of the documents to a third party but still has access to the necessary information. In conclusion, the prototype presented in the paper is an example of how local document analysis, semantic search, and local large language model (LLM) can be combined into a privacy-preserving question-answering system.

Introduction

The paper proposes a Privacy-Preserving Local Retrieval-Augmented Large Language Model (RAG-LLM) system that enables users to analyze documents and ask questions about their contents without sending sensitive information to external AI services. It combines document processing, semantic search, a local vector database, and a locally running language model to generate context-based answers while prioritizing data privacy.

1. Introduction and Objectives

Cloud-based AI tools can summarize documents, extract information, and answer questions, but they may pose privacy risks when sensitive data is uploaded to external servers. The proposed system addresses this issue by performing document processing, information retrieval, and answer generation locally on the user's computer.

Its main objectives are to:

• Protect sensitive personal, academic, financial, and organizational information.

• Extract text from both digital and scanned documents using Optical Character Recognition (OCR).

• Retrieve relevant information through semantic search.

• Generate answers using a locally deployed language model.

• Provide optional, permission-based web access when local documents do not contain sufficient information.

2. Related Work

The system integrates several established technologies from Natural Language Processing (NLP) and information retrieval:

• Retrieval-Augmented Generation (RAG): Combines retrieved information with a language model to produce contextually relevant answers.

• Semantic embeddings: Models such as Sentence-BERT represent text as numerical vectors, enabling similarity-based retrieval.

• Optical Character Recognition (OCR): Tools such as Tesseract extract text from scanned documents and images.

• Lightweight language models: Models such as Qwen 2.5 make local AI inference more accessible on personal computers.

• Existing frameworks: Tools such as PrivateGPT and LangChain provide related capabilities for private document processing and language-model applications.

The proposed system combines these technologies into a unified, local-first document question-answering architecture with optional web retrieval.

3. Proposed Methodology

The system follows a series of processing stages:

1. Document Upload

Accept digital PDFs, text files, and scanned documents.

2. Text Extraction and OCR

Extract readable text from documents and scanned images.

3. Text Chunking and Embeddings

Divide text into smaller segments and convert them into semantic vectors using MiniLM.

4. Local Vector Storage

Store embeddings in ChromaDB for efficient similarity search.

5. Semantic Retrieval

Match the user's question with the most relevant document segments.

6. Local Answer Generation

Use Qwen 2.5 3B Instruct to generate an answer based on the retrieved context.

7. Optional Web Retrieval

With user permission, retrieve additional information if the documents do not provide sufficient context.

This pipeline allows the system to answer questions using relevant document passages rather than processing the entire document for every query.

4. Implementation

The prototype implements the main components of the proposed architecture:

• MiniLM: Generates semantic embeddings for document segments and user queries.

• ChromaDB: Stores embeddings locally and supports similarity-based retrieval.

• Qwen 2.5 3B Instruct: Generates natural-language answers using retrieved document content.

• OCR: Supports text extraction from scanned or image-based documents.

• Permission-based web retrieval: Provides an optional supplementary information source when local document content is insufficient.

The components work together to create a local question-answering pipeline that minimizes reliance on external AI services.

5. Results and Discussion

The prototype demonstrated the following capabilities:

• Document question-answering: Users could upload a PDF and ask questions about its contents.

• Semantic retrieval: The system retrieved relevant text segments instead of passing the entire document to the language model.

• Local answer generation: Qwen 2.5 3B Instruct generated answers from retrieved document context without requiring an external AI API for that process.

• Web-based fallback: The system demonstrated the ability to obtain supplementary information for questions that could not be answered from the uploaded document alone.

These results suggest that integrating semantic retrieval with local language-model inference is a practical approach to privacy-conscious document analysis.

However, the evaluation was primarily functional. The study did not report quantitative measurements of retrieval accuracy, answer quality, processing speed, memory consumption, or comparisons with cloud-based systems. These areas require further testing.

Conclusion

How does this Prototype work? The Qwen 2.5 3B Instruct model above, as illustrated in this demonstration, shows what it\'s able to do based on the context inside the documents. This answers in a document-based question answering, which uses a local processing pipeline: text extraction with optical character recognition (OCR), document text segmentation, semantic embedding generation, local vector storage, local vector similarity search, and local LLM inference. These stages ( … ) collectively comprise the operational pipeline for this prototype. Additionally, the proof of concept uses a permissioned web retrieval engine to enhance the language model with data if the data requested is not available within the context of the document you provided. This adds an additional source of information but this architecture makes local document information retrieval the primary operation. What the existing implementation provides is the core functionality of this method. Evaluations have been only based on function (not quantitative measures, including retrieval accuracy, answer accuracy, latency, resource consumption, etc.); additional efforts in the future are probably the systematic benchmarking, local inference optimization, testing on diverse types of documents and datasets, as well as the comparison of retrieval and response-generation efficiencies.

References

[1] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel, S. Riedel, and D. Kiela, “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” in Advances in Neural Information Processing Systems, vol. 33, pp. 9459–9474, 2020. [2] N. Reimers and I. Gurevych, “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks,” in Proc. 2019 Conf. Empirical Methods in Natural Language Processing and 9th Int. Joint Conf. Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, 2019, pp. 3982–3992. [3] R. Smith, “An Overview of the Tesseract OCR Engine,” in Proc. Ninth Int. Conf. Document Analysis and Recognition (ICDAR), vol. 2, 2007, pp. 629–633. [4] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, “LLaMA: Open and Efficient Foundation Language Models,” arXiv preprint arXiv:2302.13971, 2023. [5] Qwen, A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Tang, T. Xia, X. Ren, X. Zhang, Y. Fan, Y. Su, Y. Zhang, Y. Wan, Y. Liu, Y. Cui, Z. Qiu, Z. Zeng, Z. Liu, and Z. Wang, “Qwen2.5 Technical Report,” arXiv preprint arXiv:2412.15115, 2024. [6] I. Martínez Toro, D. Gallego Vico, and P. Orgaz, “PrivateGPT,” GitHub repository, 2023. [7] H. Chase, “LangChain: Building Applications with LLMs through Composability,” GitHub repository, 2023. [8] C. Bhatt, A. Bisht, R. Chauhan, A. Vishwakarma, M. Kumar, and S. Sharma, “Web Scraping Techniques and Its Applications: A Review,” in Proceedings of the 2023 3rd International Conference on Innovative Sustainable Computational Technologies (CISCT), 2023, pp. 1–8, doi: 10.1109/CISCT57197.2023.10351298.

Copyright

Copyright © 2026 Mohan Mene, Pranav Ingle, Dipanshu Langote, Kunal Bhagat , Om Meherkure, Aditya Singh. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.