For the last few years, "adding AI to an app" has meant one thing by default: calling a cloud API and waiting for a response. That's still the right call for most features. But a growing share of mobile use cases, real-time camera filters, on-the-fly translation, keyboard suggestions, and privacy-sensitive health or finance features, are moving the inference step onto the phone itself. This is on-device AI: running a trained model directly on the device's processor instead of a remote server.
The shift matters for three practical reasons: latency drops because there's no network round trip, the feature keeps working with no internet connection at all, and sensitive data (a photo, a voice recording, a health metric) never has to leave the device to get a prediction. None of this is free, on-device models are smaller and less capable than their cloud counterparts, and getting them running well across a fragmented range of phone hardware takes real engineering effort. Understanding where that tradeoff is worth making is the core skill this space demands.
Take a field service app used by technicians to identify equipment parts from a photo before ordering a replacement. Technicians frequently work in basements, rural sites, or warehouses with poor or no signal, exactly where a cloud-dependent image recognition feature fails at the worst possible moment. By shipping a compact, quantized image classification model bundled with the app, the identification step runs entirely on the technician's phone, no signal required, no wait for a server response.
For example, a field service company in this situation could reasonably expect the on-device version of this feature to resolve identification in under a second regardless of signal strength, compared to a cloud version that might time out entirely in a dead zone, though the actual accuracy gap between a compact on-device model and a full cloud model depends heavily on how many part categories the model needs to distinguish. This kind of constraint, where reliability under poor connectivity matters more than raw model size, comes up constantly in emerging markets, a theme also explored in our guide to offline-first mobile apps for India's Tier 2 and Tier 3 towns.
Don't start with "let's put a model on the device" as the goal. Start by identifying a specific feature where cloud latency, offline reliability, or data privacy is an actual constraint. Most AI features in most apps don't need this; the ones that do usually stand out clearly.
On-device models must be compact enough to run on mid-range hardware without draining the battery or overheating the device. This usually means starting from a purpose-built small model rather than trying to shrink a large general-purpose one.
Convert the trained model into a mobile-optimized format, Core ML for iOS, TensorFlow Lite or ML Kit for Android, and apply quantization to reduce precision (and therefore size and compute cost) while monitoring the accuracy tradeoff at each step.
A model that runs smoothly on the latest flagship can stutter badly on a three-year-old mid-range device, which is often what a meaningful share of the actual user base is running. Test across a deliberately low-end device tier before shipping.
Bundle the model in a way that allows it to be updated independently of the app binary where possible, so accuracy improvements and bug fixes don't require waiting on app store review cycles.
For inputs the on-device model handles poorly, provide an optional path to a more capable cloud model when connectivity is available, rather than forcing every case through the constrained on-device path.
On-device AI isn't a replacement for cloud AI, it's a targeted tool for the specific slice of features where latency, offline reliability, or privacy genuinely change the product experience.
It's worth being honest about the limits here. Anything requiring broad general reasoning, open-ended conversation, or synthesis across large amounts of context is still far better served by a cloud-hosted large model; phone hardware simply isn't there yet for that class of task, and forcing it on-device usually means a worse feature, not a faster one. On-device AI shines specifically for narrow, well-defined tasks: classification, detection, transcription, simple recommendation, where a small specialized model can match or beat a general cloud model on the task it was built for.
Teams building the broader mobile product around these decisions often find it useful to think about this alongside their overall architecture choices, similar to how the tradeoffs are framed in our piece on WebGPU and in-browser AI, since both are really the same underlying question applied to different platforms: when does moving inference closer to the user actually pay off.
Two constraints catch teams off guard the first time they ship an on-device model: battery drain and device heat. Running inference repeatedly, especially on camera frames in real time, can noticeably drain a battery and cause a phone to warm up if the model isn't optimized or if it runs more often than the feature actually needs. Throttling inference frequency (processing every third camera frame instead of every frame, for instance) and using the device's dedicated neural processing hardware rather than the general CPU are both standard mitigations. App size is the other practical constraint: app stores increasingly enforce download size limits on cellular networks, so a bundled model needs to be weighed against that budget just like any other asset.
It also helps to profile on real hardware early rather than late. A model that looks fine in a simulator can behave very differently on an actual chip, where thermal throttling kicks in after sustained use and silently slows every subsequent inference call. Building a simple in-app benchmark that logs inference time and device temperature during internal testing catches this kind of regression well before it reaches a public release and generates a wave of one-star reviews about a phone getting hot.
Shipping on-device AI well usually requires a slightly different skill mix than shipping a cloud-connected feature. Someone on the team needs to understand model conversion and quantization, not just how to call an API, and QA needs a device lab (or at least a deliberately chosen set of low, mid, and high tier test devices) rather than relying on a single development phone. Teams that already maintain a native or cross-platform mobile codebase are usually best positioned to add this incrementally, feature by feature, rather than as a ground-up architectural rewrite. For teams weighing platform choices around this kind of work more broadly, it's worth reading alongside our mobile app development services overview, since the platform and framework choice made early on affects how much friction on-device model integration adds later.
Version management is another practical detail that's easy to underestimate. A model bundled inside an app binary is frozen until the next app store release, which can take days to clear review. Teams that expect to iterate on model accuracy frequently should plan for a separate model-delivery mechanism, downloading and caching an updated model file after install, so accuracy fixes don't have to wait on the app store release cycle every time.
On-device AI has moved from a research curiosity to a practical tool for specific, well-scoped mobile features. The right mental model is not "cloud versus on-device" as a permanent choice, but a per-feature decision: does this specific capability need to survive poor connectivity, respond instantly, or keep sensitive data local? When the answer is yes, the engineering investment in a compact, well-tested on-device model tends to pay for itself quickly in user trust and reliability. When the answer is no, a cloud API remains the faster, simpler path, and there's no need to force the alternative. Treated as a targeted decision rather than an all-or-nothing architecture choice, on-device AI is one of the more durable ways a mobile product can differentiate on speed and reliability, precisely because so few competing apps bother to get it right.