On-Device or Cloud AI: A CTO Deployment Decision

A phone labeled On-device and a cloud labeled Cloud flank a central Split work? box, joined by solid lines and crossed by dotted boundaries. Headline: Location is not acceptance.

Does your product still work when connectivity, latency or data-residency constraints change? For a MedTech product company with an integrated prototype heading toward a field pilot, the deployment question is which complete configuration can support the proposed pilot task under stated conditions, with an owner for data, updates and exceptions. Where the model can run is only part of it.

Start with the condition the task must survive. If the task needs a current result during disconnection, it needs a viable local path. If delayed analysis is acceptable, the data path is permitted and device resources fall short, investigate cloud. Consider hybrid when the local and remote work can be separated with explicit timing and version boundaries. If the required model cannot run on the device and the task has to keep working offline, reconsider the model, hardware or task. These are starting hypotheses to test, not a validated selection rule.

Compare edge AI (on-device inference), cloud and hybrid routes against one system scenario, and keep the current supported arrangement in the comparison. This is engineering decision planning, not a clinical validation protocol, permission for field use or a device-development offer from Grand Rounds.

Define the pilot decision before choosing a location

The founder or CEO authorizes the budget. The CTO or VP of R&D coordinates the technical comparison with hardware, firmware, ML and app leads. Product defines the proposed task. Qualified quality, clinical and regulatory owners determine their respective evidence and use requirements, and privacy, data-residency and security reviewers assess their own boundaries.

Begin with a short statement: which input must produce which useful output, for whom, under which intended conditions? Then identify the device/calibration, firmware, preprocessing, model/runtime, app and service configuration. A model filename is only part of the system version.

If a known interface fix resolves the problem, commission that fix. Moving inference leaves an untraced sensor discrepancy or a missing update owner exactly where it was.

Compare three routes on the same task

Blank local/remote sketches: on-device inference still needs support and sync paths inspected; cloud needs a permitted data path; hybrid needs crossing data, timing and compatible versions specified. Map storage, logs, caches and support for all routes.
Blank sketches: fill in the actual data, destinations, timing, access and retention for each route.
RouteWhen to investigate itWhat can rule it out or change the scope
On-device inference A current output is required offline, or a data restriction requires local processing, and the hardware/runtime can support the task Model, storage, power or thermal limits; exposed local data; no accepted fleet-update owner
Cloud inference Delayed results are acceptable and the permitted remote path supports the workload, including a model too large for the device A current result is required during disconnection; unacceptable transfer, queue, retention or service dependencies
Hybrid Distinct parts have different timing or location needs; the team can name which work runs where and what selects the route Indistinguishable stale results, incompatible versions, synchronization failures or an unowned local/remote boundary

These are investigation questions, and the table records no measured advantages. Privacy and data residency apply to every route. On-device inference can still expose local storage, logs, backups, synchronization and support access. Cloud inference depends on the applicable data conditions, agreements, configuration and qualified review. A deployment label answers neither question.

Speed alone settles little. A fast cloud model that cannot tell a late result from a current one after reconnection leaves the display requirement open, and local execution may avoid that network dependency while still lacking evidence on sustained device performance or update recovery. If the real pilot only needs delayed batch analysis and the remote path is accepted, continuous local inference may add maintenance. If it needs a current output during disconnection, a cloud-only proposal carries a dependency to resolve or a task to narrow. These are different requirements, and each calls for its own evidence.

A concrete, invented comparison fixture

Imagine a fictional wearable-product team preparing a future pilot. Its engineering bench uses a mechanical signal fixture to generate a nonclinical pattern label in an app. No people, patient data, care decisions or clinical output are involved. Everything below is proposed and untested.

Identify the reference sensor and calibration, firmware, preprocessing, model and app versions. Compare local, remote and hybrid inference against the same fixture script and output contract. Record each route's runtime and service versions too, since a version inventory does not establish compatibility.

Use a shared, owner-agreed scenario register:

Proposed conditionEvidence question for all routes
Continuous connection, reference fixture pattern Can the team link input, model event and display under the identified bundle?
Simulated 30-second transport interruption What is displayed while no fresh remote result is available? Are delayed results distinguishable from current ones?
Reconnection with queued old inputs Which inputs are processed, discarded or retained under the agreed policy, and how are their times shown?
Missing input segment or altered bench noise How does unavailable/uncertain input propagate to the displayed state?
Sustained scripted workload on target hardware What happens to latency, resource use and relevant power/thermal observations?
Proposed model update interrupted partway Which version actually runs, and how does the team identify or recover the resulting state?

The 30-second pause is an invented bench disturbance and carries no meaning as an acceptable outage or safety threshold. Owners select representative conditions and criteria for the actual product. Replaying a recorded input can compare downstream processing, but it cannot validate a replacement sensor or establish real-world representativeness.

Measure end-to-end latency, not just model execution

A vertical list of potential latency events runs from input to display with a Start to finish bracket. Transport and remote waiting are route-dependent; define clocks and retain unavailable paths.
An event map, not measured durations. Define clocks and include only the stages present in the route.

Define start and finish events. Capture relevant input time, transport, waiting, preprocessing, inference, result arrival and display. Mark absent stages explicitly: local-only inference has no remote transport or remote-service queue, although local waiting can still occur. Record clock definitions and alignment limits, because timestamps from separate systems may be misaligned.

Report routine and difficult paths, including retries, reconnection and unavailable outputs, and keep missing measurements visible. Show the slowest and failed paths beside the typical ones, since the pilot depends on the path that matters most to it. An observed bench delay is not a clinical safety boundary.

Give data and updates an owner

Blank record ties device and calibration, firmware and preprocessing, model and runtime, app and service to named update responsibilities: approval, distribution, installed-version confirmation, changed-behavior evaluation and partial deployment.
Blank template: name an owner for each update step. Previous does not mean suitable.

For each route, map raw input, derived features, outputs, logs, caches and diagnostic exports. Record purpose, authorized access, destination (including the country or region where relevant), retention conditions, deletion handling and unresolved permissions. Include the support path as well as inference traffic.

For updates, name who approves the technical bundle, distributes it, confirms installed versions, evaluates changed behavior and handles partial deployment. Record compatibility across device, model, app and service versions. A rollback proposal needs its own assessment, because the previous version may be unsuitable.

Compare lifecycle costs as well: device resources, cloud/service charges, transfer and storage, fleet support, evidence production and maintenance. Use actual estimates with stated assumptions in place of a generic claim that local or cloud is cheaper.

Use the edge-versus-cloud decision checklist

Complete one record per candidate route:

  1. Task and bundle: proposed pilot use, input/output contract, identified versions and current evidence.
  2. Conditions: connectivity, workload, missing/noisy inputs, relevant environment and limits of representativeness.
  3. Behavior: end-to-end latency, unavailable/stale state, recovery and remaining untested paths.
  4. Data: permitted locations, access, logs/exports, retention and reviewer decisions still needed.
  5. Updates: accepted owner, compatibility evidence, partial-update handling and maintenance responsibility.
  6. Decision: retain, investigate a narrow gap or propose a bounded change; alternatives and reasons; qualified approvals outside this technical decision.

Use the checklist with your R&D and domain owners to agree the next evidence-producing iteration. It is not meant to produce a readiness score. If the gap is cross-system technical ownership, ask the Grand Rounds curator whether it fits a session and bring the one-task comparison record. This is not a device-development, validation or field-use offer, and it promises no CTO's availability.

Boundaries

Grand Rounds is an initiative of Dinarys, and a conversation may lead to cooperation with Dinarys.