AI Integration: What Is RAG for Business Documents — What It Can and Cannot Do

RAG can retrieve passages from company documents and draft answers linked to sources, but it is not model training, a truth machine or a substitute for access control.

On Monday morning, the HR manager asks an internal assistant how long candidate data should be retained and receives a confident answer with a link to company policy; the only problem is that the version it found expired eight months ago. Such a system — the practical answer to what is RAG for business documents — may technically have done everything asked of it: found a semantically similar passage, placed it in the model context and written a fluent answer; it has not, however, checked whether the file is the latest approved version if the version status is not reliably recorded in the index.

RAG stands for retrieval-augmented generation, in which retrieved external context is added when an answer is generated; it does not train the model weights on company documents and is not a separate “intelligence layer” that automatically knows which document is true; it is more like a library with a very fast librarian and a talented editor, where the librarian may bring back the wrong volume but the editor will still write a convincing paragraph. The original RAG paper explicitly distinguishes the model’s parametric memory from an externally retrieved, non-parametric source.

The practical benefit is substantial when the limits are stated honestly: the system can find relevant passages in a managed document collection, assemble them into context suited to the question and prepare a draft with a verifiable reference to the source. The boundary of responsibility must also be visible to the user: the interface must not suggest that the presence of a source amounts to legal approval, and reporting errors and ambiguities must be as easy as asking a question. It cannot repair poor documentation, guarantee factual correctness, implement access rights by itself or replace a deterministic workflow and human approval where an error creates legal, financial, security or human-rights risk; this boundary determines both the architecture and what it is meaningful to measure in a pilot.

Why an old policy can be retrieved as the current one

An incident involving an old policy begins not in the language model but in document management: the shared drive contains “Personal_data_final.docx”, “Personal_data_final2.docx” and an approved PDF, yet none of the files has a consistent effective date, status or identifier for the version it replaced. The index sees three candidates with similar content, and semantic search may rank the old document highly because its wording matches the question more closely. The model cannot see a decision made at an organisational meeting if that decision is not in the data, and the filename “final” is not a governance mechanism.

The consequences are deceptive because the answer can look better than an ordinary search result: it is concise, grammatically correct and refers to a real document, which creates the impression that the checking has already been done. A source link proves only that a particular file was displayed or attached to the answer; it does not prove that every claim follows from the cited passage, that the passage has not been taken from a section describing an exception or that the document is entitled to be authoritative; the NIST GenAI Profile does not treat a trustworthy appearance as an adequate risk control and emphasises governance throughout the system lifecycle.

The remedy is publication status, a version chain and priority rules, not a longer prompt: every document in the index needs an owner, start and end dates, status, the document it supersedes, department, confidentiality class and review deadline; the retrieval filter should exclude drafts and expired versions by default; if sources conflict, the system should show the conflict and refrain from pretending that there is one certain answer, while the responsible owner should receive a task to put the document set in order. RAG can expose chaos, but it cannot turn chaos into policy.

In this situation, the answer interface should show not only the document title but also its revision, validity status, passage and a conflict warning; the log should retain which candidates were found and why one was selected, so the error can be reproduced after the index changes. If the system later starts giving a different answer, the team must be able to determine whether the document, chunking, search configuration or model changed; without that traceability, a quality incident becomes speculation about “AI behaviour” instead of a system defect that can be fixed.

What is RAG for business documents and how does it work?

A RAG pipeline begins with ingestion, not a chat window: files are taken from defined repositories, their text, tables and available structure are extracted, while scanned documents require optical character recognition, or OCR; the content is then divided into meaningful chunks, each retaining a link to the document, page, section and governance metadata. Microsoft’s guidance describes chunking as a choice that affects the usefulness of search: a chunk that is too small loses the thought, one that is too large brings back a great deal of noise, while blind division at a fixed character count can cut through a table or an exception condition.

During indexing, the chunks are given a representation suitable for search, usually combining keyword search with semantic comparison using numerical vectors; when a user asks a question, the system may turn it into several search queries, apply department, date and access filters, retrieve candidates and rerank them; only then do the selected passages enter the model’s context window together with the instruction to answer from the available evidence, cite sources and say when the evidence is insufficient. Nothing is “learned forever” at this point; the context applies to that specific request.

The final stage is generation, in which a pretrained language model turns the passages into an intelligible answer, so it may also paraphrase poorly, combine incompatible sources or add a plausible detail from its general knowledge; the result must preserve the link between passage and claim, not merely add a decorative list of sources at the end; if the question calls for an action — changing a price in the CRM, for example — the model should not be allowed to execute it freely: application code, permissions and an approval step check the structured tool request. Retrieval helps find justification; it is not permission to act.

In practice, hybrid retrieval works well: an exact product code or policy number is searched as a keyword, while the meaning of the question is searched semantically; reranking then selects the passages that best answer the whole question. This sequence must be tested with real abbreviations, mistyped codes, inflected forms and multilingual documents because a demonstration question is usually too clean. If the required passage does not appear among the candidates, the generative model cannot recover it through eloquence, so the search error must be fixed before the prompt is reworded.

Which document use cases suit RAG

The best candidates are questions whose answers already live across many managed documents but take a person too long to find: internal procedures, product manuals, technical instructions, quality documentation, explanations of contract templates and a customer-support knowledge base. Here RAG is not meant to invent a new decision, but to find the relevant section, combine a few compatible passages and prepare a draft. A good question is “which instruction describes this fault and what checking steps does it specify?”, not “how should the company act in any emergency?”. A document-research layer before human work can also be useful: a project manager can find delivery terms in contracts, a procurement specialist can find mentions of requirements, and a service technician can find earlier solutions for similar equipment. In these cases, the answer should open the exact source location so the user can check the context, and the system should retain a log of the query, retrieved passages and version used. That makes RAG a navigation and drafting tool, not an anonymous source of judgements whose decision path cannot later be reconstructed.

Poor candidates are tasks with no stable documentary basis, those requiring exact arithmetic or enforcement of rules, or those in which one mistake automatically causes an irreversible action. Payroll calculations, access grants, payment execution and the control of legal deadlines should be governed by code and verifiable business rules; RAG can find an explanation of the procedure but cannot replace the calculation engine or chain of authority. If the real aim is to connect systems and move data predictably, consider business process automation rather than making a generative answer the central switch in the process.

Suitability also depends on having an accountable owner: each document collection needs a person who approves sources, resolves conflicts and decides when something should leave the index, while each use case needs a team that reviews errors and changes the test set; if nobody takes on this work, within a few months the pilot becomes a mirror of old documents even though the model itself has not changed; a technically simple but governed support guide is therefore a better first project than connecting the entire company drive in one evening.

What RAG can do with business documents in day-to-day work

RAG can reduce the time an employee spends guessing the right folder and keywords because semantic search can find a passage even when the words in the question do not match the document’s terminology. It can combine several compatible sources in an answer, explain a complex instruction in plainer language, draft an email or report and show the pages from which each material claim came. OpenAI file search and Microsoft search architecture are specific examples of tools, but the product choice does not remove the need to define your own document statuses, filters and quality checks. The system can also uncover documentation problems that ordinary folder browsing conceals: one question retrieves two contradictory instructions, frequent questions have no source, or one department dominates the results because its files are better structured. These cases are valuable only if they are not hidden behind a single polished answer; a missing source and a conflict should become measurable events visible to the document owner. The RAG quality log then becomes a worklist for knowledge management as well as a chart of model performance.

Another genuine possibility is adaptation by role and context: a technician receives detailed instructions with codes, while a customer adviser receives a shorter explanation, provided both are permitted to see the same sources. The presentation differs, not the truth, and every role must retain the same source status and prohibition on inventing what is missing. Our work on AI solutions for business starts by defining the use case and its risk boundary, not with a model demonstration, because a good prototype proves a specific benefit on your documents while showing the questions to which the system must say “I don’t know”.

In daily work, the greatest benefit comes when a person can see what the system has done on their behalf and what still needs checking. The draft answer can highlight claims with incomplete support, suggest related documents and let the user report an incorrect version in one action; that feedback is more valuable than a simple thumbs-up icon. The correction should be tied to the question, passage and error type so the team can distinguish a missing source from awkward language or an incorrect business rule and choose the corresponding remedy.

What RAG cannot do, however convincing the answer

RAG cannot guarantee truth because an error may arise before, during or after generation: the source may contain an incorrect fact, retrieval may select an irrelevant passage, the context may lose an exception, or the model may combine correct paragraphs incorrectly. A citation reduces the risk of blind trust only when the user can open the precise location and check that the claim really follows from it, but a link to a real PDF is not a quality mark, just as a bibliography in a flawed report does not make its conclusion correct.

It cannot implement access control by itself: if the search layer does not filter documents by a verified user identity and document permissions before retrieval, the model may receive a passage that the user is not allowed to see, and adding “do not disclose secret information” to the prompt afterwards does not repair that architectural error. Microsoft’s guidance on document-level access provides for permission data and security filters in the search path itself; a hidden button in the user interface is not protection if the query can be invoked another way.

RAG does not make untrusted content safe either: a document, web page or email may contain an instruction that tries to rewrite the system’s behaviour — prompt injection — and OWASP identifies this as a distinct risk that a simple prohibition in the system prompt does not fully resolve; external content must therefore be treated as data, not commands, tool calls must be narrowly permitted and validated, and a high-risk action must remain subject to a deterministic rule and human approval. The model may propose; the system grants authority.

The list of limits must also include availability and operational continuity: if the search index is unavailable, a safe system does not pretend that it still has the company sources but clearly switches to an error state or restricted mode; otherwise, the user cannot distinguish an answer grounded in sources from free improvisation by the model. Cost and request limits, an emergency stop and restoration of the previous configuration must also be planned; a RAG product is a chain of several services, and any silent failure can change the meaning of an answer even while the chat window continues to work.

Document readiness: OCR, metadata and version control

The size of a document folder is not a measure of readiness: a scanned contract with a skewed page, a table without a readable header, a PDF with text in the wrong order or a low-contrast photograph may look intelligible to a person but lose a digit, the relationship between columns or a paragraph boundary in OCR output; Microsoft’s description of OCR limitations explicitly relates the result to scan quality, resolution, contrast, lighting, rotation and text characteristics. Representative documents must therefore be checked after extraction by comparing the text, tables, page references and material fields with the original, rather than trusting that a file was “processed successfully”.

Metadata gives a passage its organisational context: document type, business unit, product, language, owner, approver, confidentiality, effective date and version status allow a query to be narrowed before semantic similarity is assessed. Without them, the search system compares sentences but does not know that a warehouse instruction applies only to Lithuania or that a contract appendix has been superseded. The most important fields should come from a trusted system or be checked by a person; a generated guess about a document’s status must not become the filter that determines the next answer.

Updating is also part of the product, not a one-off import task: the team must know how quickly an approved change reaches the index, how a withdrawn passage is deleted, what happens when a file address changes and whether the system continues showing the old version after a failure. Microsoft’s indexing guidance distinguishes incremental updates from reindexing, so every source needs a documented method for synchronisation and error control; before a RAG project, it is worth putting one authoritative document flow in order; otherwise, fast retrieval merely accelerates the consequences of unclear governance.

A document-readiness sample is useful before the first indexing run: select different file types, ages, languages, tables, scans and access classes, then check the extracted text, chunk boundaries, metadata and source link for each. The error rate should not be turned into one average number because the cost of a missing comma in an instruction and a missing amount in a contract are different. The sample provides a basis for deciding which formats to accept automatically, which need human review and which should not yet be indexed; this work often improves quality more than choosing a different language model.

Access rights, privacy and deployment choices

A secure architecture begins with identity: who is asking, which organisation and department they belong to, which document classes they may see and whether those rights are checked on every retrieval request. The permission filter must operate before passages reach the model context, while logs should avoid unnecessary copies of full questions, answers and sensitive passages; edge cases must also be tested — an employee changes role, a document becomes restricted, access is revoked, or one client tries to find another client’s content; “the chat requires a login” is not an adequate acceptance criterion.

There is no honest universal answer to “will my data be used to train the model?” without naming the provider, product, account and settings. OpenAI’s business and API materials state that data from the relevant business products is not used to train models by default, while the API data-controls documentation separately describes retention, abuse monitoring and endpoint exceptions; Anthropic likewise distinguishes processing for commercial products, deliberate consent to improvement and retention conditions. The contract and technical design must therefore verify the specific service rather than rely on the phrase “business API”.

Deployment in an EU region or on your own infrastructure may help meet particular data-location, control or integration requirements, but does not by itself demonstrate GDPR compliance or security. The processing purpose and legal basis, data minimisation, retention periods, subprocessors, deletion, incident process and access audit must still be defined; GDPR principles apply to the whole chain, not only the country of the model server. Sometimes the right decision is not to include certain documents in RAG at all or to remove fields that the answer does not require before indexing.

The threat model must test not only a curious employee but also incorrect group synchronisation, a shared link, an administrator role, a retained cache and a document containing a malicious instruction; test users should cover every role and prohibited role combination, trying direct questions, synonyms and indirect requests for a summary; the result must reveal neither a passage, a document title nor a confidential detail that can be inferred from the answer; the test should be repeated after permissions change because yesterday’s safe filter may remain cached. These checks are acceptance criteria, not decoration for a later security audit.

How to measure retrieval, grounding and correctness

A single “accuracy” score for a RAG system is like one average rating for a hospital: the number may look good while a critical class of errors remains hidden. First measure retrieval separately — whether the required passage appeared within a defined number of top results and whether irrelevant passages displaced it. Then measure the context’s relevance to the question, the answer’s grounding in the supplied passages, factual correctness against an approved reference, whether each source reference supports the specific claim, and the system’s ability to abstain when there is no source or sources conflict.

Microsoft’s documentation on RAG evaluators separates these dimensions, and the ARES research paper similarly distinguishes context relevance, answer faithfulness and answer relevance. In a practical test set, every real question therefore needs not only the “correct answer”, but also a required source, acceptable wording, prohibited claims, a role, document version and expected behaviour when evidence is insufficient. Some examples should come from frequent questions, others from costly exceptions and deliberate traps.

The acceptance threshold should be set for every dimension and risk class before the results are seen; otherwise, after the demonstration the team will choose whichever metric looks best. Pilot measurements should also preserve the distribution of errors by document type, department, language and question type because the overall average may conceal that manuals work well while contract tables work poorly. An automated model evaluator helps scale the testing, but a sample should be reviewed by a person and critical answers compared with an authoritative source, not the confidence of another model.

After launch, the same dimensions should be measured on a controlled production sample and with privacy-preserving logs, while a change to the document corpus, chunking algorithm, embedding model, reranking or generative model may improve one group of questions and degrade another, so every version needs a regression test and comparable baseline with unchanged evaluation rules across the entire test set. An alert should be triggered not only by a fall in the overall score but also by the appearance of a critical error, such as an unauthorised passage or an invented answer where abstention was expected.

RAG, search, long context, fine-tuning and agents

Ordinary full-text search is better when the user knows the exact name, code or phrase and needs a document rather than a composed answer; it is cheaper, more predictable and easier to audit. Semantic search helps with synonyms and vague questions, but generation should be added only where a summary creates real value; RAG is not compulsory for every enterprise search tool: sometimes the right product is a good search page with filters, a passage preview and version status because the user can draw the conclusion from the complete document.

Putting an entire document into a long context window can be simple for a small, stable body of material, but in a large corpus it increases cost, noise and the risk that a material paragraph is lost among irrelevant content. Fine-tuning can reinforce a format, style or particular task behaviour, but it is not a convenient way to store frequently changing prices, policies and instructions because updating and citing the source becomes less transparent. RAG allows the document collection to change independently of training the model weights, but that flexibility comes with the cost of managing the index, versions and retrieval quality.

Automation executes predefined steps, while an AI agent can choose a tool and the next step, so that freedom requires tighter boundaries around authority, validation and stopping; RAG can give an agent information but not rights: if the model finds the leave policy, it still must not approve an absence or change the payroll system itself; a structured function call is only a proposal to the application, which checks the schema, identity, permitted action, amount or other limits, and the necessary human approval. Technology comparisons begin with process risk, not a desire to use the newest label.

The choice can be expressed as a simple test: if you need to find and open a file, start with search; if you need to summarise a few changing sources with references, consider RAG; if you need to maintain a stable format or classification behaviour, fine-tuning may be suitable; if you need to execute a predictable sequence of actions, build automation; add an agent only when the next step cannot be programmed safely and the benefit outweighs the additional risk. These approaches can be combined, but each layer must have its own task, measurement and stopping boundary, or the cause of an error disappears behind the word “AI”.

How to build a limited pilot with real questions

A pilot begins with one document collection, one user group and one decision boundary, such as technical-support manuals where the system only finds sources and drafts an answer. Before development, the team collects real questions from search logs, emails and employee interviews, adds the correct sources and deliberately includes unanswerable, outdated, conflicting and unauthorised cases. For every case, it defines what counts as acceptable: the required passage is found, the claim is grounded in the source, the citation leads to the correct location, the answer is factually correct and the system safely refrains from inventing what is missing.

Thresholds should be fixed before the demonstration and separated by risk: a correctable draft may be acceptable for a frequent informational question, but a personal-data, contract, security or payment question requires stricter checking and human approval. The pilot should also measure response time, cost per request, the operation of access filters, the delay in updating the index, and how often an employee opens the source or corrects the answer. If the system improves only the demonstration examples but does not hold up on a previously hidden test set, no product result has been proved; only the team’s ability to prepare a demonstration has been proved.

Our AI solution development starts from €3,500 and usually takes 3–8 weeks, while a working pilot on your own data can be delivered in 2–3 weeks; these figures describe the service’s starting price and general schedule, not a fixed quotation for an unknown scope. The end of the pilot should bring more than a chat window: it should include a versioned document corpus, test questions, separate quality measurements, an error log, access checks and a decision on what the solution must not do. Later integration requirements are worth recording as clearly as in any other digital project, following the principle described in our article on the mistakes business owners make when commissioning a website: acceptance criteria and owners should be defined before full implementation, not after the first impressive screen.

The pilot should continue only if it meets the predefined thresholds on an unseen part of the test set, handles unauthorised and unanswerable questions safely, and produces a measurable benefit in human work. If retrieval systematically fails to find the correct source, fix the documents, metadata and index first; if the source is correct but generation distorts it, change the context, prompt or model; if the error occurs only in high-risk decisions, leave those decisions to a deterministic system and a person. A stopped pilot is not a failure — it is inexpensive evidence that, for this particular process, the limits of RAG matter more than its demonstration effect.