Edge AI vs cloud AI: how to decide

Most teams ask "should this be edge AI or cloud AI?" as if it were a single choice. In practice, connected-operations projects almost always split the workload across three tiers — device, gateway and cloud — and the question is really "which task belongs where."

Start from the constraint, not the model

Three constraints usually decide the split:

• Latency. If a decision has to be made in milliseconds — stopping a machine before a fault propagates, flagging a defect before the next part moves down the line — it has to run close to the sensor. Round-tripping to the cloud adds network latency you cannot always predict.

• Bandwidth and connectivity. Streaming raw vibration or video data from hundreds of assets is expensive and often impossible where connectivity is intermittent (a factory floor, a moving vehicle, a rural site). Edge inference lets you send events and summaries instead of raw streams.

• Data sensitivity and residency. Some data should never leave the site. Processing it locally and sending only the model's output is often the simplest way to satisfy that requirement.

If none of these apply, the cloud is usually the right default: it is easier to update models, easier to scale compute, and easier to retrain as more data arrives.

A working split we use

1. Device or PLC-level filtering. Cheap, rule-based checks that discard obviously normal readings and reduce the volume that needs further processing.

2. Gateway inference. A compressed anomaly-detection or classification model runs on a local gateway, close enough to the assets to act within seconds, far enough from the device to aggregate several sensors at once.

3. Cloud training and fleet-wide analytics. The heavier work — training, evaluating and comparing behaviour across an entire fleet — happens centrally, where you have access to the full history.

The model that runs at the edge is usually a compressed or distilled version of the one trained centrally, not a separate model built from scratch. Keeping them in the same lineage matters: if the edge model drifts from what the cloud model expects to see, you lose the ability to compare results across sites.

What breaks this in practice

The most common failure mode is not a modelling problem — it is an operations problem. Edge devices go months without a software update because nobody owns the rollout process. A model trained on six months of "normal" data starts flagging false anomalies once a machine is serviced and its baseline behaviour shifts. The fix is boring but essential: versioned models, staged rollouts, and a monitoring dashboard that treats the edge fleet the same way you would treat a cloud service — with health checks, not just alerts.

The question to ask first

Before choosing edge or cloud, ask: what decision does this model support, and how much time do we have to act on it? If the answer is "seconds," start at the edge. If the answer is "hours or days," start in the cloud and add edge inference later, once you understand which specific step actually needs the latency.