Your notebook gives the expected result when its author runs it. A colleague, partner or customer analyst follows the same instructions and gets a different output, or cannot finish the run. First check for a hidden difference in what was run: input-file version, code version, environment, configuration or random seed. A successful rerun shows repeatable execution; scientific validity and buyer value require different evidence.
This page is for founders, R&D and product leads with a research prototype, a named customer or partner task and a budget for the next stage.
Map your three unknowns or talk with the Grand Rounds curator if senior technical judgment is the missing piece.
Three unknowns that do not substitute for each other
Distinguish scientific validity, repeatable execution and buyer value. Give each its own evidence and reviewer; identify which one blocks the next customer or partner commitment.
The author-led workaround may still be useful
For a supervised, limited investigation the scientist-led run may suffice. Record the effort and interpretation it requires. A receiving person who must operate independently creates a different handover requirement.
What to check first when the rerun differs
Practical advice, not a validated diagnostic checklist:
| Symptom | First thing to check |
|---|---|
| The next person cannot install or run it | Declared dependencies and versions, runtime version and missing files. |
| It runs but gives different numbers | Input-file hash, code commit, configuration and random seed. |
| It differs only in the last digits | Whether the tolerance was agreed before the rerun; numerical differences between libraries or hardware. |
| It gives the expected numbers only on the author's machine | Undeclared local files, environment variables, cached state or manual steps. |
An unpinned dependency causing an install failure belongs in the first row; an identical environment with a different input hash belongs in the second. That distinction determines the first check, not the final cause.
A published computational-reproducibility example
Samuel and Mietchen's 2023 study examined 27,271 notebooks from 2,660 GitHub repositories associated with 3,467 articles. Of those notebooks, 22,578 were written in Python and 15,817 declared dependencies in standard files. For 10,388 of that latter group, all declared dependencies could be installed; 1,203 of those ran without error and 879 produced results identical to the originals. This is a publication sample, not a company-prototype failure rate. Dependency installation, successful execution and matching output are separate records.
Worked illustration: a toy ranking notebook
Invented, unperformed example. A toy ranking notebook with arbitrary IDs and dimensionless values illustrates execution only. Minimal rerun checklist: pin the code commit; record the environment or lockfile; record the input-file hash; fix or record the random seed; agree the tolerance before the rerun. Suppose the receiver's input hash differs from the author's: first resolve which file was meant. If both people then obtain the same ranking from the same file within the agreed tolerance, that answers the bounded execution question. Scientific validity and buyer value remain open.
| Observation to investigate | Candidate unknown | Confirm separately |
|---|---|---|
| The receiver used a different input-file version. | Execution provenance: did both people perform the same calculation? | That either file represents the intended application. |
| Both people obtain the same ranking from the same toy inputs. | The bounded execution question may be answerable. | That the ranking method is scientifically valid. |
| The partner likes the ranking demonstration. | The task and buyer hypothesis need clarification. | That a buyer will pay or replace an existing method. |
Make a scientific, software and buyer uncertainty map
Keep a separate evidence record and responsible owner for each question:
| Category | Evidence currently held | Unestablished question | Receiving owner |
|---|---|---|---|
| Scientific | State the actual evaluation and its scope, not just a result label. | Does the comparison support the intended claim on relevant inputs? | Scientific lead or appropriate domain reviewer. |
| Software/data | Identify the last agreed run and any receiver attempt. | Can the next person perform that run and explain differences? | Software/data lead and intended receiving user. |
| Buyer | Record an actual task, alternative and conversation or commitment. | Who would buy what, and what evidence is missing? | Founder/product sponsor and the actual buyer. |
Your next step: choose the specific unknown
Take one disputed result and receiving task to the scientific/software owners. Choose whether the next check concerns the claim, the rerun or the buyer before expanding the backlog. Once the unknown is named, compare routes in research prototype to customer pilot; to select outside help, see the scientific-product CTO remit. If a senior technical view is missing, talk with the Grand Rounds curator about where your milestone sits.
What shapes the scope
What shapes the scope of a conversation about this problem
- Whether a named receiving person exists who must run the result.
- How much hidden state the first rerun exposes: inputs, code version, configuration, preprocessing.
- Who owns the scientific claim and can decide what counts as a meaningful comparison.
Time and cost follow from these points and the permitted access, and are discussed with the curator. The curator takes you from the first conversation to setting up how you work with a CTO, and supports you afterwards.
Boundaries
A bounded rerun identifies execution gaps. Scientific validity stays with a qualified scientific reviewer; buyer value needs evidence from the actual buyer. Permission to inspect data is task-specific. The toy ranking example is invented; implementation is separate work.
Grand Rounds is an initiative of Dinarys, and a conversation may lead to cooperation with Dinarys.