# Your AI Demo Works but the Paid Pilot Does Not

> Turn paid-pilot symptoms into deployment, permissions, operation-identity, answer-support and cost evidence before choosing a repair.

Source: https://grandrounds.tech/insights/ai-demo-to-paid-pilot · Published 2026-11-10 · CTO Grand Rounds

**Short answer.** A demo can show a useful interaction on controlled inputs while leaving customer conditions untested—for example, another user's permissions, incomplete inputs or recovery without the author nearby. A paid pilot adds named users, permitted inputs, agreed cost/time and an accepting reviewer. The five boundaries below are an editorial starting checklist, not a complete taxonomy; latency, identity integration and other customer conditions may need separate checks:

1. **Deployment:** someone other than the author can deploy and recover the exact version.
2. **Permissions:** access rules hold from source to answer to export, across roles and tenants.
3. **Repeated operations:** retries and timeout recovery must not duplicate the logical action or its agreed billing unit. Define how completion, failure and cancellation are treated.
4. **Answer support:** every answer traces to a source, and unsupported requests get a defined abstention route.
5. **Cost per accepted task:** spend and review time are counted until a reviewer accepts the result.

The fix starts with a register of what is observed, unverified or accepted for one workflow, not with a rewrite or a new model.

*Figure: A cutaway demo window sits above a longer workflow with four open evidence sockets under the hook Trace the whole paid workflow.*

## What the paid pilot adds to the demo

Write a narrow customer promise: one workflow, its users, permitted sources, output and accepting reviewer. Keep the current manual workaround and its effort in the baseline. A paid pilot adds operational and commercial conditions to the interaction shown in the demo.

The diagram below starts from that promise (user, input, result, time, cost, reviewer, fallback) and groups the checks into four boxes. Cost per accepted task is carried by the promise itself and counted across all four.

*Figure: One customer promise branches into four independent checks: deployment by another engineer, end-to-end role and tenant permissions, delivery with one operation identity, and source-supported review or no-answer. Unknown is not failed; name an owner and next check.*

## What to inspect at each of the five boundaries

The symptoms are illustrative prompts, not incident-frequency data. Collect evidence in an authorized environment:

| What the first paid customer reports | Boundary | First evidence to collect |
| --- | --- | --- |
| Only the author can fix or redeploy it | Deployment | A second engineer deploys the exact version from written steps and uses its recovery path. |
| A user saw another company's document or cannot open the cited source | Permissions | Test the same query across permitted and denied roles/tenants, including retrieval, saved results, exports and support views. |
| The job was recorded or charged twice after a timeout | Repeated operations | Trace one operation ID through job, result, ledger and downstream action. [Stripe's idempotency keys](https://docs.stripe.com/api/idempotent_requests) cover supported API requests; your queue and ledger require separate evidence. |
| It answered confidently without a source | Answer support | Record source version and the reviewer's accept, correct or reject reason; test abstention on an incomplete pack. |
| Cost or turnaround exceeds the promise | Cost per accepted task | Count all attempts plus preparation, correction and review; divide cost by accepted tasks and report elapsed time, abstentions and failures separately. |

## A failure, cost and permissions register for your AI pilot

Create one entry per observed failure or material unknown. Link to approved evidence rather than copying customer data into a general project document.

**Reusable entry**

- **Workflow and promise:** task, user, tenant, input boundary, intended result.
- **Reproduction:** environment; application/model/prompt/retrieval versions; permitted case ID; steps; expected and observed behavior; evidence location.
- **Permissions:** actor, role, tenant, source/output rights, and the boundary under review.
- **Failure and impact:** deployment, access, answer support, duplicate action, delay, or other category; customer consequence; known extent versus unknown extent.
- **Cost and time:** calls/retries, provider charges or estimate basis, review/rework minutes, elapsed time, and outcome denominator.
- **Disposition:** contain, repair, restrict, investigate, or retain manual handling; owner, reviewer, next evidence, and retest status.

An assignment is only proposed until the responsible person accepts it. A gap is only closed when the agreed check has been repeated on the changed version.

## Invented example: a report-drafting pilot

Invented example: a paid pilot turns operational documents into a reviewer-approved draft report. Inspect the gaps below as separate decisions: input completeness, source-review access, job identity and full review cost. Start each repair at its evidenced boundary.

*Figure: Invented diagnostic example titled Same task, new conditions: the demo used a complete document pack and an admin account, while the paid workflow adds incomplete packs and customer reviewers. A missing appendix produced a confident unsupported conclusion in the synthetic case; the proposed next check is that the domain owner specifies the no-answer and escalation route. Reviewer source access is a separate question, still unverified.*

| Boundary | Observation or unknown | Business consequence | Next bounded check and owner |
| --- | --- | --- | --- |
| Deployment | Only the author has deployed the current version. | Recovery depends on one person. | Engineering lead arranges an independent deployment in the approved test environment. |
| Access | The reviewer can open an output, but source access awaits a check for that role. | The reviewer may be unable to verify the draft; inappropriate disclosure is also unverified. | Security and engineering owners agree role/tenant checks across sources, outputs, and exports. |
| Answer support | A missing appendix produces a confident conclusion in the synthetic case. | The conclusion exceeds the available evidence. | Domain owner specifies the required abstention message and escalation. |
| Repeated work | The test job finishes after a simulated timeout; duplicate usage accounting is unknown. | The same task might be recorded or charged more than once. | Engineering and billing owners trace one operation ID through job and ledger records. |
| Cost and delay | The demo records model spend but omits reviewer correction time. | The accepted-task baseline remains unresolved. | Product owner records all attempts and review effort for the permitted sample. |

## Decide what blocks this pilot—not every future release

Contain suspected exposure through the authorized owner; keep unresolved cases in the register with the next evidence request. If inputs, acceptance ownership or resources are missing, the sponsor chooses a pause or narrower promise. Agree exclusions with the customer.

As planning advice, unresolved cross-customer data exposure and recovery depending solely on the author warrant a pause or restriction until the responsible owners accept a safe route. Cost and turnaround are blocking when the agreed promise depends on them. A convenience gap may be deferrable only after checking its actual risk, customer commitment and required approvals, with an accepting owner and retest date; this checklist grants no release permission.

## Relevant CTO experience on answer support

For the answer-support boundary, [Marton Csernai’s profile](https://grandrounds.tech/ctos/marton-csernai) mentions a hybrid RAG pipeline for a RegTech SaaS with citation provenance, and evaluation pipelines that make AI outputs checkable. Citation provenance concerns traceability; whether answers are accurate and acceptable in your workflow needs separate evidence, so ask what the reviewers accepted and rejected. Fit and availability go through the curator.

## Next step: find the gaps that block your paid pilot

**What to have ready for the gap review.** Your workflow, authorized code/test environment, permitted representative inputs, acceptance owner, specialist-review needs and deadline. Describe these to the curator; any later scope or terms are agreed with whoever you appoint.

Turn the gaps into [agreed pilot acceptance criteria](https://grandrounds.tech/insights/ai-pilot-acceptance-criteria), then [choose who leads the first paid pilot](https://grandrounds.tech/insights/cto-for-paid-ai-pilot). Bring the promise sentence and register to the [curator](https://grandrounds.tech/talk-with-the-curator).

## Boundaries

Use synthetic or explicitly permitted data in an authorized environment. This is a diagnostic register, not a security audit, legal opinion or acceptance standard. The example is invented. Each qualified owner keeps their own sign-off. CTO leadership and implementation are separate scopes.

Grand Rounds is an initiative of Dinarys, and a conversation may lead to cooperation with Dinarys.

## Questions and answers

### Why does an AI demo work when the paid pilot does not?

The demo usually runs on controlled inputs, one privileged account and the author nearby. A paid pilot adds other users, other roles, incomplete inputs, repeated requests and an accountable reviewer, so each of those needs its own evidence.

### Does a failing pilot mean we have to rebuild the product?

Begin with the evidence. A register separates the gaps: a missing input rule, an access decision, an operation-level fix, or a model change. Replacing the model, framework or application is a later decision taken on that evidence.

### What should we check first?

Write the promise for the one workflow your paid customer is waiting for, then record each of the five boundaries as observed, unverified or accepted, with an owner and a next check.

### Can we test permissions and billing on live customer data?

Use synthetic or explicitly permitted data in an authorized test environment, and agree the tests with the engineering and security owners first.

### How does a CTO from Grand Rounds fit in, and what does the curator do?

Grand Rounds does not build or run the pilot. The curator listens and can identify relevant CTO experience in the network. Your team or selected provider implements any change under its own agreement.
