Repair, Replace or Change? The Next Test Iteration After a MedTech Field Failure

If you have already traced a lab-to-field discrepancy, the next question is which part to keep, repair, replace or change, and which evidence would tell the options apart. Still diagnosing it? Start with Your MedTech Prototype Works in the Lab but Fails in the Field. This page compares ways to reach the next test iteration; it does not decide whether a pilot may start.

Short answer: compare three routes against one bounded scenario before funding a rebuild. Keep and repair the current system, including a supplier fix. Take a vendor-supported alternative. Change selected components. For each route, write down the linked changes it needs, the evidence it would produce and what it would leave open. The result is a configuration, failure-hypothesis and evidence matrix your owners can agree as the plan for the next test iteration, the step before a field pilot.

Test-release planning here means agreeing a configuration, engineering questions, proposed tests, evidence gaps and decision owners. Decisions on patient-facing use, clinical validity and regulatory route stay with your qualified quality, regulatory and clinical owners.

For an outside technical perspective, talk with the Grand Rounds curator and bring your matrix and what you know so far.

Compare changes against the current system—not against a blank page

Compare retaining/repairing the current bundle, a supported vendor alternative and selected component changes on one bounded scenario. Include new interface dependencies and the evidence each route would produce.

Three routes compared by the evidence to request and the gap left open: retain or repair, use a vendor alternative, or change selected components.

What each route leaves unproven

Each route answers one question and leaves others open. Here the question is what each proposal would and would not settle.

If a firmware change appears to break the model

Ask the firmware and ML owners which input contracts the model expects: sampling, filtering, units, timestamps and missing-data flags. A firmware or decoder pairing might close the mismatch while the model stays as it is. A preserved wire format leaves the meaning of the signal open, so compare decoded values and model inputs as well as the final output. Treat these as hypotheses to examine about your device.

If the lab result does not cover the care environment

List the conditions your owners consider relevant and mark each as evidenced or only proposed. The joint good machine learning practice principles from the US FDA, Health Canada and the UK MHRA (GOV.UK page updated 5 October 2026) emphasize human-AI team performance (principle 7) and testing during clinically relevant conditions, including measurement inputs and potential confounding factors (principle 8). They address AI-enabled devices; your quality and domain owners decide what the intended use requires beyond bench evidence.

If two vendors each own a component

Ask both to review the same interface example: emitted values, units, time meaning, missing or invalid states and consumer behavior. Request vendor evidence with its configuration and coverage limits. A component test from one supplier covers that component; the application's behavior with delayed output from the whole chain needs its own evidence, so name a system owner who assembles the end-to-end evidence. Keep the current supplier in the comparison if it can close the gap.

If the change has no controlled release

Discuss the full bundle, not "the latest firmware": hardware revision, calibration, firmware, decoder/features, model, app, service configuration and the input and test-data references. A rollback bundle needs its own compatibility check; an older version is one candidate among several. How to judge who should own these records is in How to Select a CTO for a MedTech Field Pilot.

Which route first? Rank the candidate routes by questions. Which route can produce evidence that separates the competing hypotheses soonest, with the access you already have? How many interfaces does it touch, and who owns each one? Does the supplier or your own team already hold configuration-specific evidence for it? Cost and timing stay unknown until the responsible people estimate the next evidence-producing iteration.

A configuration, failure-hypothesis and evidence matrix

The following matrix is an invented, unperformed example. Letters identify candidates, not component versions: B = baseline, V = supplier repair, H = acquisition or calibration change, F = firmware/decoder change, M = preprocessing/model change, A = app/integration change, R = replacement subsystem. Each belongs to one of the three routes. Record linked changes, a failure hypothesis, proposed evidence, review owner and the decision the evidence could inform. On a phone, scroll the table sideways.

Proposed configuration / routeFailure hypothesis to discussEvidence proposed for discussionWhat it would still not establish
Keep and repair — B: unchanged baseline The discrepancy may depend on a condition absent from the reference run. Compare the reference and disturbance event scripts; retain emitted, decoded and displayed observations. That the unchanged baseline is compatible, suitable for care use or acceptable for a pilot.
Keep and repair — V: supplier's firmware/decoder repair; versions unknown Producer and consumer disagree about timing or validity semantics. Obtain the supplier's change description; compare both sides of the affected contract and the downstream display. Whole-system coverage from the supplier's component results alone.
Change components — H: acquisition or calibration change Input variability begins before decoding. Use the physical fixture to compare acquisition behavior under matched source conditions; inspect downstream consequences. Real-world placement effects, care-environment performance or model suitability.
Change components — F: firmware and decoder change Window boundaries, units or missing-data handling differ across the interface. Examine paired emitted/decoded examples, then follow their effects through the unchanged model and app. That correct parsing establishes model validity or appropriate user interpretation.
Change components — M: preprocessing/model change The model pipeline handles a proposed input condition poorly despite an agreed contract. Record the data basis and compare pipeline outputs; plan independent evaluation rather than reusing tuning examples as proof. Performance outside the evaluation coverage or clinical effectiveness.
Change components — A: app/integration change A delayed or unavailable output is displayed as current. Compare output time, arrival time and displayed state during the proposed pause/reconnection scenario. That the underlying sensor or model is correct.
Vendor alternative — R: integrated acquisition subsystem; versions unknown Replacing a subsystem may remove one disputed boundary. Request supported configurations and evidence; map remaining interfaces and examine the complete proposed chain. Compatibility with the current model and app or suitability for intended use from a vendor demo.

When each route wins or loses — editorial guidance to test. Keep and repair is useful when the supplier can show the changed contract and your assembled chain is covered; a component-only test leaves the integration question open. A vendor alternative is useful when it removes a disputed interface and comes with evidence for your configuration; it loses that advantage if it adds interfaces you cannot test. A targeted component change is useful when the hypothesis points to something you control; changing several things at once makes the cause harder to separate.

Worked route choice — invented, not executed. Suppose a timing mismatch is already reproduced on the bench and the supplier's change touches only the decoder: compare V with B first. If raw acquisition differs under matched conditions, investigate H instead. Counterexample: a decoder fix separates nothing when the bench cannot reproduce the mismatch; first extend B with the disturbance scenario. This is a rule of thumb, not a validated readiness method.

Have hardware, firmware, ML, app and qualified domain and quality owners agree the next evidence-producing iteration: configuration, permitted inputs, owner, receiving evidence and review point.

Start with your own owners: agree which system-change hypothesis to examine next and what the evidence would settle. If an outside technical perspective would help, talk with the Grand Rounds curator. The curator guides you from the first conversation to setting up how you work with a CTO, and supports you afterwards.

What shapes the scope of a conversation like this:

  • how many owners (hardware, firmware, ML, app, vendors) hold records, and whether anyone owns the link between them;
  • which artifacts and evidence your team can open to an adviser, and under what data rights;
  • whether you want a second technical opinion on your matrix, support for the owners' discussion, or technical leadership of the comparison with your team or vendor doing the build;
  • which qualified domain and quality owners accept the outcome.

Timing and cost depend on these points and are agreed after the curator understands them.

Boundaries

This page covers engineering evidence for comparing routes. Decisions on patient-facing use, regulatory route and clinical validity belong to your qualified quality, regulatory and clinical owners. The examples are invented and unperformed.

Grand Rounds is an initiative of Dinarys, and a conversation may lead to cooperation with Dinarys.

Questions and answers