# AI Evidence and RAG Evaluation: What Should a CTO Accept?

> Evaluate RAG retrieval, claim support, citations, permissions, versions, missing evidence and review effort with a case-level acceptance record and example.

Source: https://grandrounds.tech/insights/ai-evidence-rag-evaluation · Published 2026-11-13 · CTO Grand Rounds

Accept a bounded use only against a case-level record of claim support, source access, missing-evidence behavior and review effort, with each responsible owner's decision. Under the proposed rule here, an unresolved required permission check or unsupported material claim blocks that use; strong retrieval scores cannot compensate. A missing locator may call for repair or a narrower use, not abandonment of every case.

*Figure: A magnifying lens reveals a source fragment and qualifying note beneath an answer, inviting inspection beyond a citation link.*

**The answer sounds correct. Can a reviewer verify every material claim?** In a financial-document operation, that is a more useful acceptance question than whether a retrieval-augmented generation (RAG) demo produces fluent prose or a high average score.

The business task is to prepare reviewable case evidence without recreating the analysis during review. A CTO should ask for a versioned acceptance record that separates retrieval, factual support, citations, permissions, missing evidence, and human effort. Technical acceptance must remain distinct from the domain review and authority to use the conclusion.

## Establish the decision and its owners

Name the domain reviewer who judges substantive support, the owners of data/security boundaries and the technical reviewer. Record actual participation and acceptance for each part; a proposed organization chart does not establish them.

Define one task, permitted source packet, user/role, output, and review endpoint. Keep autonomous financial conclusions and external actions outside a drafting evaluation unless separately authorized and scoped. Preserve ordinary search, rules, ready-made document tools, and manual review as alternatives. RAG development is not the predetermined outcome.

## Evaluate seven dimensions separately

RAG evaluation metrics commonly include context precision and recall, faithfulness or groundedness, answer relevance, and citation precision and recall. Names vary by tool. Dimensions 1–3 translate that vocabulary into inspectable questions; dimensions 4–7 address permission, applicability, missingness and effort that a headline score cannot settle. No universal numeric threshold is proposed here.

*Figure: Seven separate findings cover relevance, support, citations, permissions, applicable versions, missingness and full review effort. They are not a weighted approval score.*

### 1. Retrieval relevance: did the right evidence reach the answer stage?

Record the question, retrieved fragments, source identities, and their applicability. Ask both whether retrieved material is useful and whether necessary permitted evidence was missed. A passage can share the right keywords while addressing a different entity, period, or question.

Retrieval relevance has two parts. **Precision** asks how much retrieved material applies to this entity, period and question. **Recall** asks whether the permitted evidence needed for the answer was retrieved at all.

Separate “not retrieved” from “not in the permitted packet.” The first may call for a retrieval change; the second may require a missing-evidence response. A more capable model cannot establish a fact that the available evidence does not establish.

### 2. Factual support and faithfulness: does each claim follow?

Break the output into material claims. For each, preserve the exact supporting fragment and check qualifications, contradictory evidence, and the distinction between a source fact, calculation, and interpretation.

For numerical claims, retain row/column headers, period, unit, sign, denominator, and notes. Reproduce any arithmetic. For this article, **faithfulness** means staying within the supplied evidence; that is not the same as proving the source itself is correct or sufficient for the business decision. The qualified domain reviewer handles the latter judgment.

### 3. Citations: can the reviewer reach the support efficiently?

A citation needs a usable document/version and exact location—not just a homepage or an entire file. Check whether each cited fragment supports its attached claim and whether every claim requiring evidence has an adequate citation. These are practical citation-precision and coverage questions; a count of links answers neither.

Record broken links, missing locators, and relevant notes the reviewer must search for manually. Even correct text may not reduce review burden if its support is hard to inspect.

### 4. Permissions: was the evidence permitted for this user and use?

*Figure: A usable citation does not establish permission for this user and use, or whether the source version is applicable to the required case period. Inspect the path before retrieval and model context; most recently uploaded is not necessarily applicable.*

Inspect the path into retrieval and model context, not only the final answer. Ask how roles, case/tenant boundaries, revoked access, saved outputs, exports, and logs are covered. Use authorized evaluations with synthetic or otherwise permitted data, not live confidential documents as a test payload.

Refresh behavior matters. [Microsoft's Azure AI Search documentation](https://learn.microsoft.com/en-us/azure/search/search-document-level-access-overview#enforce-document-level-permissions-at-query-time-preview), updated in September 2026 and checked on November 2, describes token-based enforcement as a preview capability. It compares the caller's claims with permission metadata already stored in the index; source changes affect results after the relevant synchronization. That dependency illustrates why “access control supported” is not evidence of immediate revocation in your workflow.

### 5. Version and applicability: is this the source the task requires?

Keep the source version, effective date where relevant, and who determined applicability. “Most recently uploaded” does not necessarily mean applicable to the case period. Preserve the retrieval/index and application configurations needed to interpret the result.

Plan a controlled changed-source variant. Record what was expected, what was actually observed, and how affected prior outputs are handled. A successful refresh does not itself decide the domain meaning of a revised policy.

### 6. Missingness: does the answer preserve what cannot be established?

Include incomplete packets, contradictions, unanswerable questions, and inaccessible sources. Specify whether the expected outcome is a limited answer, clarification, denial, or escalation. Distinguish absence of evidence from evidence of absence.

Do not reward a complete-looking answer for filling a gap. Equally, a blanket refusal can be unhelpful where a supported partial answer is allowed. The reviewer needs the boundary: what is established, what is not, and the next evidence required.

### 7. Review time: is the output usable without hidden reconstruction?

Measure active effort to find support, check numbers, correct text, resolve exceptions, and review the revision. Keep machine waiting and end-to-end elapsed time separate. Count rejected work and repeated attempts, not just the fastest accepted draft.

Compare the current method and candidate on comparable tasks under the same review standard. Report case mix and difficult/no-answer cases separately. A short drafting interval does not establish greater casework capacity.

## Invented example: relevant source, unsupported completion claim

*Figure: Register R/v2 records eight received files and three awaiting verification; an exceptions annex is mentioned but not supplied. The invented candidate claims all verified and no exceptions, neither of which the packet supports.*

This example is wholly invented, unperformed, and not a financial recommendation or client result. A proposed internal case-preparation task asks: **“What does the permitted packet establish about document-review status?”**

Synthetic source **Register R/v2, row “Files,” column “Received”** says **8**. A note says **3 of those files await verification**. The packet mentions a separate exceptions annex but does not include it. An invented candidate says, “All eight files have been verified and there are no exceptions,” citing R/v2.

| Dimension | What the synthetic inputs illustrate—not a tool observation |
| --- | --- |
| Support | “Received” does not mean “verified.” The note conflicts with the candidate's completion claim. |
| Citation | A valid pointer to R/v2 cannot support the claim merely because the file is relevant. Include the header and note in the check. |
| Missingness | The absent annex does not establish “no exceptions.” It also does not prove that exceptions exist. |

Relevance, permission, version and effort were not observed in this example, so none is scored.

A proposed evidence-limited draft would say that the register records eight received files, with three awaiting verification, and that the supplied packet does not establish the exceptions position. That wording still needs qualified review for the intended use. It does not clear the underlying case.

Use the same packet and question to compare manual search, the existing document tool, and a RAG candidate. No option inherits a result from this illustration. A later variant could change permissions or source content, with the expected behavior agreed by the responsible owners before evaluation.

## Build the evaluation set from your own queue

Draw cases from real question types using permitted materials. Include answerable questions, questions the packet cannot answer, conflicting sources, a changed source and a role that should be denied access. Keep the mix and difficult cases visible in results. This is an evaluation set defined by your owners, not training data or a public benchmark.

## Use the RAG acceptance scorecard—as an evidence record

Do not add the dimensions into a weighted approval score. Copy this record for each case, alternative, configuration, and attempt:

> **Scope:** task, intended use, user/role/case boundary, permitted packet, exclusions. **Identity:** case/question, alternative, application/model/prompt/retrieval versions, attempt, output version, performed or planned status. **Expected evidence:** required claims, supporting fragments and locators, applicable versions, missingness/no-answer behavior; who defined them. **Seven findings:** relevance; support; citation coverage and correctness; permissions; version/applicability; missingness; full review effort. Record evidence and unresolved items separately for each. **Review:** reviewer identity, remit, actual participation, disagreements, corrections/re-review, acceptance or rejection and rationale. Technical review is not domain approval. **Decision:** retain, repair/retest, narrow, stop, or bounded use; accepting owners, excluded cases, remaining conditions, receiving owner, and changes that reopen the decision.

Start unestablished fields as **Unknown**. Keep case-level results as well as summaries. An unresolved required permission check cannot be offset by strong relevance scores. There is no universal numeric threshold: owners need task-specific conditions tied to consequences and coverage.

[OpenAI's evaluation guidance](https://developers.openai.com/api/docs/guides/evaluation-best-practices) recommends task-specific evaluations and calibration of automated scoring with human feedback. Apply those principles without copying illustrative numerical targets. This is a methodology reference, not a recommendation to adopt its retiring Evals platform. A model grader can assist triage; it is not the domain acceptance owner or independent proof of source truth.

Choose one recurring case and complete the scorecard with its technical, domain and data/security owners. Identify the first material unsupported claim or unverified boundary, then compare the smallest relevant repair against the current method. If the change to test is still unclear, see RAG vs. Fine-Tuning: Diagnose the AI Failure First. If the gap crosses permissions, review and several tools, [ask the Grand Rounds curator about the relevant next conversation](https://grandrounds.tech/talk-with-the-curator), not a presumed RAG build or an available CTO.

## Boundaries

The acceptance record is an editorial method, not a validated scoring algorithm, financial approval or authority to act on a case. Technical acceptance and the domain owner's decision remain separate.

Grand Rounds is an initiative of Dinarys, and a conversation may lead to cooperation with Dinarys.
