Start with what failed. If the right permitted evidence never reaches the model, investigate context, retrieval and source versions; if the evidence is present but the output keeps mishandling the task, try clearer instructions and examples before fine-tuning. A data leak or a repeated external action needs application and workflow controls, because RAG and tuning alone repair neither.
That is a working diagnostic heuristic, not a tested decision procedure. Several causes can coexist: compare changes on the same cases, and keep an ask-or-leave-unresolved path for a missing fact.
Diagnose the failure before choosing retrieval-augmented generation, fine-tuning or another prompt revision. If a paid pilot fails because the application supplies an outdated source, changing the model may improve the wording and preserve the error. If a job runs twice, answer-generation techniques cover only part of the repair.
For a product company with code, an implementation team and a named pilot customer, compare the smallest plausible changes against the same task. The founder or CEO owns the commitment and budget, product defines the useful outcome, engineering evaluates the system, and domain and security/data owners accept their boundaries.
What RAG, fine-tuning and prompting each change
Prompt engineering (prompting) changes the instructions, examples and context supplied for an invocation. It is a candidate for clarifying the task or desired output without training a new model.
RAG retrieves relevant source material and supplies it as context for generation. Fine-tuning trains a pretrained model further, changing learned parameters instead of adding a document to the current request. IBM's explanation describes that retrieval/context distinction and the updating of model weights during fine-tuning. It cannot tell you which approach will meet your pilot's requirements.
The approaches combine. AWS's comparison describes combining retrieval with a fine-tuned answer model. That is a possible architecture, not a reason to buy both unless evidence shows the need. A RAG system also needs a prompt, and every route needs an operating workflow.
AWS recommends starting with RAG for question answering that references your own documents and considering fine-tuning for additional tasks such as summarization. This is vendor guidance rather than a result measured on your cases. The phrase "RAG vs. LLM" confuses the layers: RAG supplies retrieved context to a language model and leaves the model in place.
Diagnose the failure before funding the change
| Observed problem | Evidence to inspect | Candidate change |
|---|---|---|
| Answers invent or omit facts from your documents, often called hallucination | Does a permitted, applicable source exist, and did it actually reach context? | Supply context, repair retrieval or request missing evidence; inspect generation too if the right evidence was present |
| Correct evidence is present but the response mishandles the task | Actual context, instructions, output and reviewer-defined error | Clearer prompting/examples; consider tuning for a repeated, evidenced behavior gap |
| The answer uses the wrong period or version | Source applicability, ingestion/index state and retrieved fragments | Source/version handling and retrieval repair; reassess generation afterward |
| A user can access another customer's material | Identity, permissions and what reached context or stored output | Application/data-access repair and qualified security review |
| A job duplicates an action or usage event | Request/action identity, attempts and external result | Workflow and recovery repair |
| Intake does not establish the requested fact | Missing information and permitted clarification path | Ask, narrow or escalate in place of inventing an answer |
A symptom is not yet a cause. "Wrong answer" can include parsing, retrieval, reasoning, source truth and workflow errors, so keep them separate. Start with current search, rules and human review too, since some tasks need no generation.
A superseded policy, an omitted field and a missing date
A fictional B2B product prepares draft answers from each customer's support policies and never sends them automatically. The following packet, behavior and proposed comparisons are invented and untested, and none of it is client evidence.
The applicable policy P2 says an administrative form must be retained for 30 days; superseded P1 says 60 days. The application supplies only P1 and the candidate draft repeats 60. This is a retrieval/version hypothesis, and it leaves open whether the model can follow instructions. Compare a corrected source path before assuming fine-tuning is needed.
In another constructed case, the correct permitted P2 passage is present, but the candidate repeatedly omits the requested source/version field. Clear instructions and an example output are a smaller candidate change. If a consistent behavior gap remains across representative cases, a tuning experiment may deserve consideration, provided the training rights, examples, supported model and lifecycle are established.
In a third case, the form's submission date is missing. A larger retrieval packet or a tuned model cannot establish that date. The workflow needs an approved clarification or unresolved-answer path, because a plausible date would make the answer look complete by changing the truth standard.
If ordinary search already gives the reviewer the needed policy and generation adds checking work, retaining search is a valid option.
Compare on one evaluation set, not three demonstrations
Freeze the task, authorized user/tenant, source versions, expected behavior and review standard. Include representative work and difficult cases: conflicting versions, missing context, unanswerable requests, inaccessible sources and relevant operational failures. A domain reviewer defines the expected support, since a model answer cannot be its own reference.
Use development examples to adjust prompts, retrieval or training, and hold back an untouched acceptance set for the final comparison. Acceptance cases placed in the tuning data would make the final result look independent when it is not.
For the reviewer record, see AI Evidence and RAG Evaluation: What Should a CTO Accept?.
For each route retain:
- Exact model, prompt, retrieval/index and workflow versions, plus attempt identity.
- What evidence actually reached generation and whether it was permitted/applicable.
- Output, claim support, omitted qualifications, citations and unresolved questions.
- Review/correction effort, repeated attempts, end-to-end latency and complete task cost.
- Operational observations and untested conditions, separately from answer quality.
Change one layer at a time where practical. If a bundle changes, record the bundle and attribute its effect to the bundle as a whole. Keep permission and recovery questions even when the experiment targets output behavior.
Compare maintenance and access, not just initial quality
| Approach | Fits when / weak when | Maintenance to include |
|---|---|---|
| Prompting | Try first when evidence is present but instructions or format are unclear; it cannot supply an absent fact | Instruction/example versions, context assembly and regression checks when the model changes |
| RAG | Investigate for changing documents and source-referenced answers; AWS cautions about whole-document summarization, so test the actual task | Rights, ingestion/deletion, applicable versions, retrieval relevance, permissions and affected saved outputs |
| Fine-tuning | Investigate a repeated behavior gap after simpler changes; it cannot replace current source context or user-specific permissions | Training rights and example quality, held-out evaluation, model support and retraining/regression work |
| Combined retrieval and tuned generation | Consider when separate evidence shows a need for both | Both maintenance burdens, plus evaluation of their interaction |
These fit conditions are hypotheses rather than measured rankings. Workflow repair is a different kind of fix from answer-generation techniques: identity, authorized handoffs, retry behavior, fallback and receiving capacity remain responsibilities in every route.
A citation can appear without factual support behind it. Tuning leaves current per-user source permissions and deletion/revocation design to the application, and a prompt asking the model to keep data separate is weak evidence of tenant isolation.
When sources change, compare what each route needs to update and which prior outputs require review. When behavior requirements change, compare prompt/example changes with the cost and maintenance of retraining.
Check current platform support before planning any real tuning proposal. On 9 November 2026 the OpenAI pricing page stated: "OpenAI is winding down the fine-tuning platform." It also stated: "All fine-tuned models will remain available for inference until their base models are deprecated." The same notice stated: "The platform is no longer accessible to new users, but existing users of the fine-tuning platform will be able to create training jobs for the coming months." Vendor availability changes, so verify the current position with the vendor you would use.
Decide the next experiment
The result may be to retain the current approach, repair retrieval, improve prompting, investigate tuning, fix the workflow or combine bounded changes. Write down the observation, suspected cause, cases and held-out set, full cost and review effort, owner and what remains untested. The responsible owners choose acceptance conditions, and no weighted score or universal accuracy threshold approves a release.
Use the record to select the next justified experiment instead of a wholesale rewrite. Any execution needs separate authorization and budget. If the uncertainty spans product quality and technical ownership, talk with the Grand Rounds curator about whether a session fits your case. Grand Rounds does not build the changes.
Boundaries
Grand Rounds is an initiative of Dinarys, and a conversation may lead to cooperation with Dinarys.