# RAG vs. Fine-Tuning: Diagnose the AI Failure First

> Compare RAG, fine-tuning and prompting by the failure you see: knowledge, behavior or workflow. Same test cases, access rules and upkeep.

Source: https://grandrounds.tech/insights/rag-vs-fine-tuning · Published 2026-11-16 · CTO Grand Rounds

*Figure: Source documents, adjustment controls and a broken handoff represent knowledge, behavior and workflow hypotheses under Diagnose before you tune. None is marked as the proven cause or repair.*

**Start with what failed. If the right permitted evidence never reaches the model, investigate context, retrieval and source versions; if the evidence is present but the output keeps mishandling the task, try clearer instructions and examples before fine-tuning.** A data leak or a repeated external action needs application and workflow controls, because RAG and tuning alone repair neither.

That is a working diagnostic heuristic, not a tested decision procedure. Several causes can coexist: compare changes on the same cases, and keep an ask-or-leave-unresolved path for a missing fact.

Diagnose the failure before choosing retrieval-augmented generation, fine-tuning or another prompt revision. If a paid pilot fails because the application supplies an outdated source, changing the model may improve the wording and preserve the error. If a job runs twice, answer-generation techniques cover only part of the repair.

For a product company with code, an implementation team and a named pilot customer, compare the smallest plausible changes against the same task. The founder or CEO owns the commitment and budget, product defines the useful outcome, engineering evaluates the system, and domain and security/data owners accept their boundaries.

## What RAG, fine-tuning and prompting each change

**Prompt engineering (prompting)** changes the instructions, examples and context supplied for an invocation. It is a candidate for clarifying the task or desired output without training a new model.

**RAG** retrieves relevant source material and supplies it as context for generation. **Fine-tuning** trains a pretrained model further, changing learned parameters instead of adding a document to the current request. [IBM's explanation](https://www.ibm.com/think/topics/rag-vs-fine-tuning) describes that retrieval/context distinction and the updating of model weights during fine-tuning. It cannot tell you which approach will meet your pilot's requirements.

The approaches combine. [AWS's comparison](https://docs.aws.amazon.com/prescriptive-guidance/latest/retrieval-augmented-generation-options/rag-vs-fine-tuning.html) describes combining retrieval with a fine-tuned answer model. That is a possible architecture, not a reason to buy both unless evidence shows the need. A RAG system also needs a prompt, and every route needs an operating workflow.

AWS recommends starting with RAG for question answering that references your own documents and considering fine-tuning for additional tasks such as summarization. This is vendor guidance rather than a result measured on your cases. The phrase "RAG vs. LLM" confuses the layers: RAG supplies retrieved context to a language model and leaves the model in place.

## Diagnose the failure before funding the change

| Observed problem | Evidence to inspect | Candidate change |
| --- | --- | --- |
| Answers invent or omit facts from your documents, often called hallucination | Does a permitted, applicable source exist, and did it actually reach context? | Supply context, repair retrieval or request missing evidence; inspect generation too if the right evidence was present |
| Correct evidence is present but the response mishandles the task | Actual context, instructions, output and reviewer-defined error | Clearer prompting/examples; consider tuning for a repeated, evidenced behavior gap |
| The answer uses the wrong period or version | Source applicability, ingestion/index state and retrieved fragments | Source/version handling and retrieval repair; reassess generation afterward |
| A user can access another customer's material | Identity, permissions and what reached context or stored output | Application/data-access repair and qualified security review |
| A job duplicates an action or usage event | Request/action identity, attempts and external result | Workflow and recovery repair |
| Intake does not establish the requested fact | Missing information and permitted clarification path | Ask, narrow or escalate in place of inventing an answer |

A symptom is not yet a cause. "Wrong answer" can include parsing, retrieval, reasoning, source truth and workflow errors, so keep them separate. Start with current search, rules and human review too, since some tasks need no generation.

## A superseded policy, an omitted field and a missing date

*Figure: Three invented cases: superseded P1 reaches context, applicable P2 is present but the candidate omits the requested source/version field, or the submission date is missing. Each needs a different investigation. — Invented and untested: the 30- and 60-day periods are fictional, not retention advice.*

A fictional B2B product prepares draft answers from each customer's support policies and never sends them automatically. The following packet, behavior and proposed comparisons are invented and untested, and none of it is client evidence.

The applicable policy P2 says an administrative form must be retained for **30 days**; superseded P1 says **60 days**. The application supplies only P1 and the candidate draft repeats 60. This is a retrieval/version hypothesis, and it leaves open whether the model can follow instructions. Compare a corrected source path before assuming fine-tuning is needed.

In another constructed case, the correct permitted P2 passage is present, but the candidate repeatedly omits the requested source/version field. Clear instructions and an example output are a smaller candidate change. If a consistent behavior gap remains across representative cases, a tuning experiment may deserve consideration, provided the training rights, examples, supported model and lifecycle are established.

In a third case, the form's submission date is missing. A larger retrieval packet or a tuned model cannot establish that date. The workflow needs an approved clarification or unresolved-answer path, because a plausible date would make the answer look complete by changing the truth standard.

If ordinary search already gives the reviewer the needed policy and generation adds checking work, retaining search is a valid option.

## Compare on one evaluation set, not three demonstrations

*Figure: Development cases and an untouched acceptance set are separated by a dashed boundary. Do not train on acceptance cases; the model is not its own reference. — Keep development examples separate from an untouched acceptance set.*

Freeze the task, authorized user/tenant, source versions, expected behavior and review standard. Include representative work and difficult cases: conflicting versions, missing context, unanswerable requests, inaccessible sources and relevant operational failures. A domain reviewer defines the expected support, since a model answer cannot be its own reference.

Use development examples to adjust prompts, retrieval or training, and hold back an untouched acceptance set for the final comparison. Acceptance cases placed in the tuning data would make the final result look independent when it is not.

For the reviewer record, see [AI Evidence and RAG Evaluation: What Should a CTO Accept?](https://grandrounds.tech/insights/ai-evidence-rag-evaluation).

For each route retain:

- Exact model, prompt, retrieval/index and workflow versions, plus attempt identity.
- What evidence actually reached generation and whether it was permitted/applicable.
- Output, claim support, omitted qualifications, citations and unresolved questions.
- Review/correction effort, repeated attempts, end-to-end latency and complete task cost.
- Operational observations and untested conditions, separately from answer quality.

Change one layer at a time where practical. If a bundle changes, record the bundle and attribute its effect to the bundle as a whole. Keep permission and recovery questions even when the experiment targets output behavior.

## Compare maintenance and access, not just initial quality

*Figure: A source citation leaves factual-support questions and permitted-use questions open: claim support, applicable version, qualifications, authorized user and tenant, permitted context, saved outputs and revocation. — A citation does not prove factual support or permission to use the source.*

| Approach | Fits when / weak when | Maintenance to include |
| --- | --- | --- |
| Prompting | Try first when evidence is present but instructions or format are unclear; it cannot supply an absent fact | Instruction/example versions, context assembly and regression checks when the model changes |
| RAG | Investigate for changing documents and source-referenced answers; AWS cautions about whole-document summarization, so test the actual task | Rights, ingestion/deletion, applicable versions, retrieval relevance, permissions and affected saved outputs |
| Fine-tuning | Investigate a repeated behavior gap after simpler changes; it cannot replace current source context or user-specific permissions | Training rights and example quality, held-out evaluation, model support and retraining/regression work |
| Combined retrieval and tuned generation | Consider when separate evidence shows a need for both | Both maintenance burdens, plus evaluation of their interaction |

These fit conditions are hypotheses rather than measured rankings. Workflow repair is a different kind of fix from answer-generation techniques: identity, authorized handoffs, retry behavior, fallback and receiving capacity remain responsibilities in every route.

A citation can appear without factual support behind it. Tuning leaves current per-user source permissions and deletion/revocation design to the application, and a prompt asking the model to keep data separate is weak evidence of tenant isolation.

When sources change, compare what each route needs to update and which prior outputs require review. When behavior requirements change, compare prompt/example changes with the cost and maintenance of retraining.

Check current platform support before planning any real tuning proposal. On 9 November 2026 the [OpenAI pricing page](https://developers.openai.com/api/docs/pricing) stated: "OpenAI is winding down the fine-tuning platform." It also stated: "All fine-tuned models will remain available for inference until their base models are deprecated." The same notice stated: "The platform is no longer accessible to new users, but existing users of the fine-tuning platform will be able to create training jobs for the coming months." Vendor availability changes, so verify the current position with the vendor you would use.

## Decide the next experiment

The result may be to retain the current approach, repair retrieval, improve prompting, investigate tuning, fix the workflow or combine bounded changes. Write down the observation, suspected cause, cases and held-out set, full cost and review effort, owner and what remains untested. The responsible owners choose acceptance conditions, and no weighted score or universal accuracy threshold approves a release.

Use the record to select the next justified experiment instead of a wholesale rewrite. Any execution needs separate authorization and budget. If the uncertainty spans product quality and technical ownership, [talk with the Grand Rounds curator](https://grandrounds.tech/talk-with-the-curator) about whether a session fits your case. Grand Rounds does not build the changes.

## Boundaries

Grand Rounds is an initiative of Dinarys, and a conversation may lead to cooperation with Dinarys.
