AI Pilot Acceptance Criteria: Define Success Before You Start

Short answer. Agree five things before comparing candidates: the workflow and users, permitted test cases, blocking failures, cost/time/review limits and the decision rule. Agreement on criteria is preparation—not acceptance. Accept the pilot only when current evidence shows it meets the blocking conditions and limits, required owners approve their parts and the operating arrangements are accepted. The decision is stop, repair and retest, or go within named bounds. Write the record first; compare model, retrieval and process changes on the same cases.

A proof of concept asks whether an approach can perform a narrow task. A pilot asks whether a defined user group can complete a workflow under agreed conditions. Pilot acceptance concerns those conditions rather than a successful demo alone.

A pilot that a customer calls "not ready" often hides several different failures: a missing source, an access gap, a duplicated billing event. Each needs a different fix. Missing sources need retrieval or input decisions; access needs permission controls; duplicate billing needs operation identity. Acceptance criteria let a product company, its pilot customer and its engineers separate those decisions.

This page gives you a record to copy, an evaluation-set method, a comparison of model, retrieval and process changes, and an invented example to adapt.

Cover graphic: one document box branches into three equal boxes under the headline Same cases. Clear limits, with the note No automatic winner.

Proof of concept vs pilot vs production: what each one decides

A proof of concept tests whether an approach can do a narrow task on selected inputs. A pilot tests whether a defined user group can complete one bounded workflow under agreed conditions. Production is the agreed service operated, monitored, supported and changed under its commitments. Teams draw these lines differently, so write down which meaning your contract and your customer use. For planning acceptance, use these working distinctions:

StageDecision it supportsNot established by this stage
Proof of concept (PoC) Can this approach perform a narrow task on selected inputs? Reliable use by customers within their permissions, costs and operating process.
Pilot Can a defined user group complete one bounded workflow under agreed conditions? Coverage of untested users, data, volume or integrations.
Production Can the agreed service be operated, monitored, supported, and changed under its operating commitments? Automatic acceptance of future changes or new use cases.

A successful PoC can justify the next pilot. Pilot acceptance supports only the explicitly agreed rollout bounds.

Write down the intended user, workflow start and end, permitted data, excluded operations, deployment destination, and acceptance owners. If those are unresolved, a model comparison may answer the wrong question.

Build an evaluation set around real scenarios

A first record can start with one workflow, user role and measured baseline. Scope grows with tenants, integrations, domain-review capacity and the rights to data you can test.

Record fieldFill with your owners
Case and purpose ID, task, representative/difficult/abstention category
Input and rights Permitted source locators, versions, role and tenant
Expected behavior Required output, allowed variation and blocked behavior
Review Named domain reviewer, rubric and disagreement resolution
Run Configuration, result and repeated-run history
Disposition Accepted/rejected/unresolved, reason, owner and next check

Build a versioned set covering ordinary tasks, difficult inputs, missing evidence, denied access and repeated operations. Choose expected behavior and reviewers first. Preserve every run used in the decision, including regressions and abstentions.

Compare model, retrieval, and process changes on the same cases

Give the baseline and each candidate the same cases, permissions and source versions. Compare model/prompt, retrieval and process changes by failure type. Keep a held-out acceptance subset where feasible and record every bundled change.

One frozen case set branches equally to baseline, model or prompt, retrieval and process comparisons; no arrow implies one candidate depends on another.
OptionWhen it is worth testingConfirm separately
Existing rules, search, or manual process The task is structured, evidence lookup is sufficient, or generation adds review burden. Reviewer judgment for ambiguous tasks.
Retrieval change Required evidence exists but the workflow selects the wrong passage, version, or authorized source. That the model will use the evidence correctly or the permission system is sound.
Model or prompt change The right evidence is present, but the output fails a defined reasoning, extraction, or format requirement. That absent evidence, tenant isolation, or billing failures have been repaired.
Process change Intake is incomplete, approval is missing, or the task needs clarification or a different handoff. That the underlying deployment or generation behavior now passes its checks.

For choosing a change against an observed failure, see RAG vs. fine-tuning. Here, every candidate must run the same agreed cases.

Invented example: extracting renewal terms from supplier contracts

Imagine a software company piloting an assistant that drafts a "renewal and notice terms" summary for a contract manager. The assistant retrieves only contracts the manager is entitled to see; the manager approves every summary. Every document, behavior and decision here is invented, fictional teaching material.

CaseInput conditionExpected behavior
C01: ordinary One current contract with a clearly stated renewal clause. Summarize the clause and give its exact location.
C02: wrong source Two suppliers with similar names; retrieval returns the other supplier's contract. Use the matching contract or ask which supplier is meant.
C03: difficult input The notice period is on an unreadable scanned page. Flag the page for human reading; withhold the term pending evidence.
C04: absent evidence A renewal clause is absent from the contract. Report that evidence gap and route to the contract owner.
C05: forbidden source Another business unit’s contract is outside the requester’s rights. Return the approved denial and protect restricted metadata and content.
C06: repeated operation A summary job completes but its confirmation times out. Recover the completed job status, preserving one summary and one usage entry.

Read the failures by layer: C02 tests source selection, C03 extraction and human review, C04 abstention, C05 permission denial and C06 recovery. Every candidate runs the same full agreed set. Domain and risk owners choose the coverage needed beyond this six-case teaching example.

Acceptance record for this invented pilot. Blocking: C02 returns the wrong supplier's term without clarification; C05 reveals restricted content or metadata; C06 creates duplicate summaries or usage entries. The contract manager and finance first measure today's manual baseline, then agree review-time and cost-per-accepted-summary limits no worse than that baseline. A failed or untested blocker means stop or repair and retest. Meeting these checks permits consideration of this team and contract set only, subject to the remaining operating evidence and required owner approvals—not automatic release.

Keep operating acceptance beside answer quality

In an authorized environment, test the exact deployable version and recovery path, cross-role/tenant permissions, repeat-job identity, whole-task cost and time, and a staffed fallback. Record the owner and evidence for each condition. The five demo-to-paid-pilot boundaries describe how to gather that evidence.

Pilot acceptance record (copy and fill)

SectionWhat to writeOwnerStatus
Scope Users, one workflow start to end, permitted sources, exclusions, environment, release/configuration and evaluation-set versions Product sponsor agreed / unresolved
Blocking conditions Failures that stop release (for example an unsupported answer or a denied path that opens), chosen before comparing candidates Domain, security, risk owners agreed / unresolved
Limits Cost per accepted task, elapsed time, review capacity, with rationale from the baseline Sponsor, finance, engineering agreed / unresolved
Evidence Baseline and candidate on the same cases; repeated runs kept; regressions; limitations Engineering, domain complete / partial
Operation Monitoring signal and owner; fallback trigger, destination and capacity; rollback; support owner Engineering agreed / unresolved
Decision Stop, repair and retest, or go within named users and limits; approvers; date; what forces reassessment All owners open / decided

Decision rule. A blocking condition that failed, was not tested or remains disputed means stop or repair, not “go with a better average.” Define narrow, observable blockers rather than an impossible demand for universal correctness. A change to model, source, permission or workflow can require renewed checks; passing the example set never replaces required specialist or operating acceptance.

Your next step: write the acceptance record for one workflow

Take one workflow, assemble representative, difficult and abstention cases, choose the smallest plausible change to compare against today's process, and fill in the record with your owners before anything is expanded.

To discuss a second view on that record, describe it to the Grand Rounds curator, who listens and can identify relevant CTO experience in the network.

What to ask a CTO who has built evaluation pipelines

Marton Csernai's profile describes AI work for compliance and back-office automation with evaluation pipelines that make outputs checkable. Ask what counted as acceptable, who reviewed it and what was rejected. The profile also mentions citation provenance; that concerns traceability, while answer accuracy and your workflow's acceptance need separate evidence. Fit and availability require confirmation through the curator.

Boundaries

This framework fits a product company with a defined workflow, a pilot customer or sponsor, and owners who can choose limits. It supports one bounded decision: stop, repair and retest, or go within named bounds. Thresholds, sample sizes and risk approvals stay with your owners, and domain, security and legal sign-off stay with the qualified people. Checks run in an authorized environment with permitted or synthetic data. Owners consider blocking conditions and operating evidence alongside any score. CTO leadership and implementation by your team or a selected provider are scoped separately.

Grand Rounds is an initiative of Dinarys, and a conversation may lead to cooperation with Dinarys.

Questions and answers