What is RAG and when a classic search is enough

RAG (retrieval-augmented generation) is a technique in which, before a language model answers, the system searches the company's documents for the fragments relevant to the question and passes them to the model as context. The model writes the answer from those fragments and cites where it came from. It is useful when questions are varied and the answer is spread across many documents; when questions are few and repetitive, an FAQ page or a classic search engine solves the same problem at lower cost and lower risk.

A language model knows nothing about your company: not your price list, not your collective agreement, not your product manual. RAG is the way to give it that knowledge without retraining it. In my projects it is the component clients ask for most and, at the same time, the one I most often replace with something simpler after measuring.

How it works, step by step

1. Ingestion. The source documents are collected: PDFs, web pages, spreadsheets, emails, records from a document management system. They are cleaned (repeated headers, footers, broken tables) and, for each one, its origin, date and who is allowed to see it are stored.
2. Chunking. Each document is split into fragments of a few hundred words, ideally respecting sections and paragraphs. A fragment that is too large mixes topics; one that is too small loses context.
3. Indexing. Each fragment is converted into a numeric vector (an embedding) that represents its meaning, and stored in an index. Many systems combine this semantic index with a classic keyword index, because each one succeeds where the other fails: the semantic index understands synonyms; the keyword index gets product codes, proper names and acronyms right.
4. Retrieval. When a question arrives, it is also converted into a vector and the closest fragments are retrieved, filtered by the user's permissions. More are retrieved than needed (for example, twenty).
5. Reranking. A second model, more precise and slower, reorders those twenty and keeps the best five. This step improves quality a great deal and is usually left out of demos.
6. Generation. An instruction is built for the model: the question, the chosen fragments and clear rules ("answer only with the information provided; if it is not there, say so; cite the source of each statement"). The model writes the answer.
7. Citation and logging. The answer is shown with links to the documents used, and the question, the retrieved fragments and the answer are stored so they can be evaluated and corrected later.

Two improvements that make a difference in practice: adding to each fragment an automatically generated sentence of context ("this fragment belongs to the warranty section of the 2025 maintenance contract"), which Anthropic describes as contextual retrieval, and rewriting the user's question before searching when it arrives incomplete or depends on the previous conversation.

When a classic search or an FAQ is enough

Before building RAG, look at the data from real queries for one or two weeks (the baseline measurement template works). The decision usually comes out of this table:

A concrete example

An installation company has 300 technical manuals from manufacturers, its own internal procedures and the history of fault reports. Technicians call the office to ask "what torque does this model take?" or "have we seen this controller error before?". RAG fits here: large corpus, varied questions, the answer spread between the manual and previous reports, and the technician can verify the citation in seconds. The system is evaluated with a hundred real questions from recent months and their correct answers, and the first version is used only by the technical office, not by the technician in the field.

In the same company, customer questions about opening hours, coverage areas and payment methods come to twenty. Those go on an FAQ page, not into RAG.

What it costs

• Preparing the documents. This is the biggest hidden cost. Duplicate documents, old undated versions, scanned PDFs without text and broken tables make the system answer badly. Cleaning and organising the corpus usually takes longer than building the system.
• Indexing. Generating embeddings is paid per volume of text once, and again every time a document changes. For an SME's corpus it is a low cost.
• Querying. Each question triggers calls to the embedding model, the reranker and the language model. The cost per query depends on fragment size and on the model. It has to be measured with real queries, not estimated.
• Infrastructure. Vector database, storage and server. Small compared with the above.
• Maintenance. Reindexing when documents change, reviewing unanswered questions, repeating the evaluation when the model changes. It is a recurring cost that has to be budgeted from the start.

For concrete figures for your case: [COMPLETAR: document volume, queries per month and chosen models; with those the cost per query can be calculated].

Risks and how to contain them

How to evaluate whether it works

A demo with five questions chosen by the vendor is not an evaluation. This is:

1. A set of real questions. Between 50 and 200 questions that people have really asked, with the correct answer and the document where it is found. Written by someone who knows the business, not by whoever builds the system.
2. Retrieval. Is the correct fragment among the five retrieved? If it is not, the model cannot get it right. This metric isolates the search engine from the writer.
3. Faithfulness. Is everything the answer says backed by the fragments? Check a sample by hand, or use a second model as a judge and then verify part of it by hand.
4. Correctness. Does it match the expected answer? Faithfulness without correctness means the search engine brought back the wrong document.
5. Abstention. Include in the set questions with no answer in the corpus. The system must say it does not know. Measure how often it does.
6. Repeat. The evaluation is run again every time the model, the chunking or the instructions change. Without that, an "improvement" can break what already worked.

If you cannot say what percentage of the hundred test questions it answers correctly, you do not know whether it works. You are guessing.

When to move beyond RAG

RAG answers questions. If what you need is for the system to do something with the answer (create a ticket, prepare a quote, update a record), you are already talking about an agent, with its own risks and controls. And if most queries require live data, the central piece is not the model but the integration with your systems.

If you want to know whether your case is RAG, an FAQ or a search engine, the AI implementation service starts with a week of analysing real queries and ends with a recommendation, even if the recommendation is to build nothing with AI.