How do you make AI answer from your company documents without sending the data outside?

The way to do it is a private RAG: the AI does not memorize your documents, it retrieves the right passages from a knowledge base that lives on your server and answers from them, citing where it got them. With the language model, the search and the documents all running inside the company, no file needs to go to a cloud service.

Have you ever asked a chatbot something about your company and received an answer that was polished, confident and wrong? That happens because the model does not know your return policy, your standard contract or your product manual. This article explains what RAG is, how it works in-house and what we planned in a real technical assessment of local AI for a company.

RAG stands for retrieval-augmented generation. The name comes from a 2020 paper by Patrick Lewis and colleagues, which combined two kinds of memory: the one inside the model, learned during training, and an external memory, an index of documents consulted at the moment of the question.

The paper itself points out the problem this solves: large models store knowledge in their parameters, but their ability to access and precisely use that knowledge is still limited, and showing where answers come from and updating what they know remained open problems. The authors report that, with retrieval, the model generated more specific, diverse and factual text than the version without retrieval.

For a business owner, the translation is simple. Instead of hoping the AI knows everything, you give it a library and require it to look things up before answering.

In 1981, William Gibson published the short story Johnny Mnemonic in Omni magazine, later included in the collection Burning Chrome. Johnny is a data trafficker who had surgery to get a storage system implanted in his head. He carries information too sensitive to travel over the network, locked by a password only the recipient knows, and not even he can read what he carries.

A private RAG is the opposite of Johnny. The model does not carry your documents inside it. They stay in a vault that belongs to you, and the model only sees the few passages the search hands over for that question, from that person, with that person's permissions. If the policy changes tomorrow, you replace the document, not the model.

There are four parts, and all of them can run on a company server.

The documents. Contracts, manuals, policies, catalogs and approved answers are split into short passages.

The embedding model. It turns each passage into a vector, a list of numbers that represents the meaning of the text, so that the search finds "return deadline" even when the document says "right of withdrawal". One open example is Qwen3-Embedding-0.6B, licensed under Apache 2.0, with support for more than 100 languages and a 32 thousand token context, according to the model page.

The vector database. It stores the vectors and runs the search. Qdrant is an open source vector database licensed under Apache 2.0. For those already on Postgres, pgvector adds similarity search to the database itself, and the vectors live alongside the rest of the data.

The language model. It receives the question and the passages found, writes the answer and points to the source. It can be an open model running locally, as we showed in the article on Ollama, vLLM or llama.cpp.

Often a reranker is added as well, a second model that reorders the passages found and keeps only the most relevant ones, such as Qwen3-Reranker-0.6B.

Permissions per department, for real. If finance and HR use the same AI, the rule has to live in the knowledge base, not only in the model's instructions. In our assessment, the written rule was separate vector collections per group, never a single collection filtered only by the prompt.

Documents can attack too. OWASP calls it indirect prompt injection when the model receives content from external sources, such as websites or files, and that content changes its behavior in unexpected ways. In a RAG, a PDF with a hidden instruction can become an order. That is why the plan included tests for injection through documents and for leaks between departments before opening it to everyone.

Sensitive data stays out. Documents with health data, such as medical certificates in HR, do not go into the knowledge base. The recommendation was to put only policy documents in HR, with no personal data.

Numbers come from the tool, not the text. Prices, stock and order status change all the time. That data should come from a read-only query to the company's system, not from an old PDF the search happened to find.

This is the most common question, and the answer is usually to start with RAG. In our technical recommendation, the conclusion was that fine-tuning should not be the first step: what changes often, such as prices, stock and policies, goes into RAG and tools, because it updates immediately. Fine-tuning works better for tone of voice and answer format, and only comes in later, if evaluation shows a gap.

Teaching facts through training has another practical problem: every change requires training again, evaluating and versioning. With RAG, you just replace the document.

In October 2026, we carried out a technical assessment of local AI for a company with a server running a single RTX 5090 and internal assistants for four departments. The design had one assistant per department, all on the same base model, each with its own instructions, its own knowledge base restricted to its group and its own tools.

For search, the choice was Qwen3-Embedding-0.6B with the reranker from the same family and Qdrant as the vector database, with pgvector as the alternative for those who prefer a single Postgres. To keep answers fast with many people using it at once, the recommendation was to limit RAG to the top five passages of about 500 tokens each. The plan also included a set of real questions per department, with a reference answer and source, to measure whether the AI gets it right before opening it to the whole company.

Does RAG eliminate hallucinations? No. It lowers the risk because the answer now rests on real passages, but the AI can still misread them. That is why the answer should show its source, and critical data should be checked by a rule or a tool.

Do I need a GPU to have a private RAG? The search and the embedding model are light. What needs hardware is the language model, and its size depends on how many people will use it at the same time.

Can I use the PDFs and spreadsheets I already have? Yes. The bigger job is usually organizing them: removing old versions, separating what each department can see and leaving out anything with sensitive data.

DOUVRAS Labs builds AI assistants with a knowledge base of declared origin, tests, guardrails, a hand-off to a human when the AI does not know and clear shutdown criteria. That is what we offer in AI agents and assistants, in local AI for law and accounting firms, where processing stays inside the firm, and in automation and integrations to connect the AI to the systems you already use.

And in your company, when someone asks what the current policy is, does the answer come from the right document or from the memory of whoever is closest?

Douglas Henrique, DOUVRAS Labs.

Lewis, P. et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS, 2020. arxiv.org

IBM Technology. What is Retrieval-Augmented Generation (RAG)? YouTube. youtube.com

Qwen. Qwen3-Embedding-0.6B and Qwen3-Reranker-0.6B. Hugging Face. huggingface.co

Qdrant. Vector Database and Vector Search Engine. github.com

pgvector. Open-source vector similarity search for Postgres. github.com

OWASP GenAI Security Project. LLM01: Prompt Injection. genai.owasp.org

Gibson, William. Johnny Mnemonic. Omni, May 1981. Included in Burning Chrome, 1986. wikipedia.org