A prototype passes its bench test, then gives a different result on a test site. The hardware team suspects conditions, the ML team suspects input, and the app team sees no error in its own component. A firmware change may or may not be part of the discrepancy.
Short answer: a changed build is a hypothesis until a record ties the observation to the complete configuration that produced it. For a MedTech founder or R&D lead, the first useful step is a version-and-observation map for one discrepant event: sensor and calibration, firmware, preprocessing, model and app, each linked to preserved evidence and a named owner.
You can build the first map in an existing issue tracker or spreadsheet. This page shows what to record, how to keep observation, hypothesis and missing evidence apart, and when your current team or vendor is already enough.
The worked example compares two benches and one invented field site, not a real deployment.
This is engineering framing for records you are already authorized to examine. Decisions on patient-facing use, regulatory route and clinical validation stay with your qualified quality, regulatory and clinical owners.
If you want an outside technical read of one discrepancy, talk with the Grand Rounds curator and bring the observation and what you know so far.
Why can the sensor, firmware, model and app give different results?
A device result depends on the complete configuration. Record sensor and calibration, active firmware settings, preprocessing, model and app together with the conditions and evidence for the discrepant event.
“The new firmware broke the model” is a hypothesis
Compare the nearest meaningful observation and identify every changed part. Inspect decoded values and input contracts—units, channel order, sampling and timestamps—before attributing the difference to a firmware build.
Practical advice, not a validated checklist:
- Is the build the same, including active configuration and calibration?
- Is the input contract the same: units, channels, sampling, timestamps and preprocessing?
- Are conditions the same: placement, motion, ambient conditions, connectivity, power state and clock source?
A difference at step 1 does not eliminate steps 2 and 3. If the records agree but the result differs, the chain has not explained it; identify the remaining evidence and its owner rather than declaring a cause.
Why a lab result may not repeat outside the lab
Ask domain/quality owners which conditions matter in the intended setting. Keep observed coverage, proposed tests and missing records distinct. Sensor event time, packet arrival and display time are separate. Record placement and motion, ambient conditions, connectivity, power state, handling and clock source. Treat each difference as a hypothesis needing evidence, not a demonstrated cause.
Build one version-and-observation map
Make one entry per discrepancy with preserved artifact references, configuration, data context, boundary observations and record owners. Keep a hypothesis separate from its supporting evidence and assign the next clarifying check.
Invented, nonclinical example—not a test result. IDs and locators below are illustrative; no such test files are supplied.
| Record | A: lab bench | B: second-room bench | C: first field site (invented) |
|---|---|---|---|
| Observation/display | `O-A`, sequence `42`; “fixture moving” | `O-B`, sequence `42`; “fixture idle” | `O-C`; reported result differs; sequence not recorded |
| Physical setup | `S-01`, `HW-A`, `JIG-1` | Same IDs; orientation not captured | Setup IDs not recorded |
| Calibration | `CAL-C03` in manifest | Not captured | Not recorded |
| Firmware / active config | `FW-A17` / `CFG-K02` | `FW-B24` / `CFG-K02` | Not recorded |
| Preprocessing / parameters | `PRE-P04` / `NORM-N02` | `PRE-P05` / `NORM-N02` | Not recorded |
| Model / runtime | `MODEL-M03` / `RT-R02` | Same versions | Not recorded |
| Model-data context | `DATA-T02` / `DATA-E02`, fixture data | Same references; relevance unverified | Relevance to field conditions unverified |
| App / platform / config | `APP-A09` / `OS-O01` / `UI-U02` | Same versions | Not recorded |
| Conditions | Script `Q2`; continuous link reported | Script `Q2`; intermittent link reported | Fixture script not applicable; intermittent link reported; battery state not captured |
| Placement/handling | Fixed jig reported | Orientation not captured | Not recorded |
| Ambient conditions | Not recorded | Not recorded | Not recorded |
| Clock source | Not recorded | Not recorded | Not recorded |
| Evidence locators | `bench-A/manifest.json`, `raw.csv`, `input.json`, `model-event.json`, `ui.txt` | `bench-B/manifest.json`, `raw.csv`, `ui.txt`; input/event unrecorded | Site report only; configuration and input/event records missing |
| Record owner | Named for each record | Calibration/event records unassigned | Configuration/condition records unassigned |
When the existing team or vendor may be enough
Use the existing team or vendor when they can supply a missing record or explain a supported interface. Outside technical leadership helps when the configuration link across component owners remains unowned.
Bring one observation, its comparator, artifact versions and open questions to the owners. Agree the next check and receiving person before commissioning a broader technical investigation.
Any outside contribution depends on component ownership and evidence your team may share; agree its scope separately.
Boundaries
This page covers engineering evidence for one discrepancy. It does not decide patient-facing use, regulatory route or clinical validity; your quality, regulatory and clinical owners do. Examples here are invented and unperformed.
Grand Rounds is an initiative of Dinarys, and a conversation may lead to cooperation with Dinarys.
After mapping the discrepancy, compare MedTech readiness options before selecting the technical-lead contribution.