Your MedTech Prototype Works in the Lab but Fails in the Field

A prototype passes its bench test, then gives a different result on a test site. The hardware team suspects conditions, the ML team suspects input, and the app team sees no error in its own component. A firmware change may or may not be part of the discrepancy.

Short answer: a changed build is a hypothesis until a record ties the observation to the complete configuration that produced it. For a MedTech founder or R&D lead, the first useful step is a version-and-observation map for one discrepant event: sensor and calibration, firmware, preprocessing, model and app, each linked to preserved evidence and a named owner.

You can build the first map in an existing issue tracker or spreadsheet. This page shows what to record, how to keep observation, hypothesis and missing evidence apart, and when your current team or vendor is already enough.

The worked example compares two benches and one invented field site, not a real deployment.

This is engineering framing for records you are already authorized to examine. Decisions on patient-facing use, regulatory route and clinical validation stay with your qualified quality, regulatory and clinical owners.

If you want an outside technical read of one discrepancy, talk with the Grand Rounds curator and bring the observation and what you know so far.

Why can the sensor, firmware, model and app give different results?

A device result depends on the complete configuration. Record sensor and calibration, active firmware settings, preprocessing, model and app together with the conditions and evidence for the discrepant event.

Observe sensor/calibration, firmware, preprocessing, model and app boundaries. A separate training/evaluation data-provenance record feeds the model context; configuration links are not proof of cause.

“The new firmware broke the model” is a hypothesis

Compare the nearest meaningful observation and identify every changed part. Inspect decoded values and input contracts—units, channel order, sampling and timestamps—before attributing the difference to a firmware build.

Practical advice, not a validated checklist:

  1. Is the build the same, including active configuration and calibration?
  2. Is the input contract the same: units, channels, sampling, timestamps and preprocessing?
  3. Are conditions the same: placement, motion, ambient conditions, connectivity, power state and clock source?

A difference at step 1 does not eliminate steps 2 and 3. If the records agree but the result differs, the chain has not explained it; identify the remaining evidence and its owner rather than declaring a cause.

Why a lab result may not repeat outside the lab

Ask domain/quality owners which conditions matter in the intended setting. Keep observed coverage, proposed tests and missing records distinct. Sensor event time, packet arrival and display time are separate. Record placement and motion, ambient conditions, connectivity, power state, handling and clock source. Treat each difference as a hypothesis needing evidence, not a demonstrated cause.

Build one version-and-observation map

Make one entry per discrepancy with preserved artifact references, configuration, data context, boundary observations and record owners. Keep a hypothesis separate from its supporting evidence and assign the next clarifying check.

Invented, nonclinical example—not a test result. IDs and locators below are illustrative; no such test files are supplied.

RecordA: lab benchB: second-room benchC: first field site (invented)
Observation/display `O-A`, sequence `42`; “fixture moving” `O-B`, sequence `42`; “fixture idle” `O-C`; reported result differs; sequence not recorded
Physical setup `S-01`, `HW-A`, `JIG-1` Same IDs; orientation not captured Setup IDs not recorded
Calibration `CAL-C03` in manifest Not captured Not recorded
Firmware / active config `FW-A17` / `CFG-K02` `FW-B24` / `CFG-K02` Not recorded
Preprocessing / parameters `PRE-P04` / `NORM-N02` `PRE-P05` / `NORM-N02` Not recorded
Model / runtime `MODEL-M03` / `RT-R02` Same versions Not recorded
Model-data context `DATA-T02` / `DATA-E02`, fixture data Same references; relevance unverified Relevance to field conditions unverified
App / platform / config `APP-A09` / `OS-O01` / `UI-U02` Same versions Not recorded
Conditions Script `Q2`; continuous link reported Script `Q2`; intermittent link reported Fixture script not applicable; intermittent link reported; battery state not captured
Placement/handling Fixed jig reported Orientation not captured Not recorded
Ambient conditions Not recorded Not recorded Not recorded
Clock source Not recorded Not recorded Not recorded
Evidence locators `bench-A/manifest.json`, `raw.csv`, `input.json`, `model-event.json`, `ui.txt` `bench-B/manifest.json`, `raw.csv`, `ui.txt`; input/event unrecorded Site report only; configuration and input/event records missing
Record owner Named for each record Calibration/event records unassigned Configuration/condition records unassigned
Invented nonclinical A/B bench summary: firmware and preprocessing changed; B lacks calibration and model-event records. The added field-site C is in the table, not this summary; no cause is established.

When the existing team or vendor may be enough

Use the existing team or vendor when they can supply a missing record or explain a supported interface. Outside technical leadership helps when the configuration link across component owners remains unowned.

Bring one observation, its comparator, artifact versions and open questions to the owners. Agree the next check and receiving person before commissioning a broader technical investigation.

Any outside contribution depends on component ownership and evidence your team may share; agree its scope separately.

Boundaries

This page covers engineering evidence for one discrepancy. It does not decide patient-facing use, regulatory route or clinical validity; your quality, regulatory and clinical owners do. Examples here are invented and unperformed.

Grand Rounds is an initiative of Dinarys, and a conversation may lead to cooperation with Dinarys.

After mapping the discrepancy, compare MedTech readiness options before selecting the technical-lead contribution.