On-Device Fitness Coach #1: On-Device AI Is a Decision, Not a Trend

Before we get to SDKs, runtimes, or models, there's one question every AI feature should answer. It's not which model, which framework, or even which prompt. It starts with one architectural decision :

Where should the intelligence actually live?

It's a surprisingly easy question to skip. We get excited about the latest runtime or the newest model release and jump straight into implementation. I've done it too. But I've learned that choosing where your AI runs is often more important than how it runs.

Let me explain with a simple example.

Imagine someone opens their fitness app after a run. The app quietly looks at the last few weeks of activity and says :

“You've been most consistent on days when your workouts stayed under 35 minutes. Maybe try a shorter session tomorrow.”

It's a small insight. Helpful. Personal. It feels almost instant.

But before that feature ever existed, someone had to answer a much less glamorous question :

Did the phone generate that insight locally, or did the user's workout history travel to a server first?

That decision shapes everything that follows - performance, privacy, architecture, operational complexity, and ultimately the user experience. That's why I think every Android AI feature should begin here.

AI Is Not the Decision. Architecture Is.

One pattern I've noticed is that teams often start evaluating AI SDKs before they've decided whether the intelligence belongs on the device at all. In my opinion, that's backwards.

On-device AI isn't a feature or a marketing label. It's an architectural decision with real tradeoffs, just like choosing a database, a networking strategy, or your app architecture. Just because a model can run on-device doesn't necessarily mean it should.

Good AI architecture isn't about putting intelligence everywhere.

It's about putting intelligence in the right place.

A Framework for Making the Decision

Whenever I'm evaluating a new AI feature, I try to think through the same five questions. I've started calling it the PLACE Framework because it reminds me that before building intelligence, we need to decide where it belongs.

P - Privacy

How sensitive is the data? If the feature processes health information, financial data, personal conversations, or anything users reasonably expect to remain private, that's a strong signal toward on-device processing.

Privacy isn't just about compliance. It's about earning trust.

L - Latency

How quickly does the feature need to respond? Real-time camera analysis, gesture recognition, live translation, or contextual suggestions often need to feel instantaneous. Waiting on a network request can completely change the experience.

On the other hand, a weekly activity summary or an end-of-day report can comfortably tolerate cloud latency.

A - Availability

Should the feature continue working without an internet connection? Users don't stop expecting apps to work just because they're on a plane, hiking in the mountains, or dealing with unreliable connectivity.

If offline support is a product requirement, on-device inference becomes much more attractive.

C - Cost

Cloud inference scales with every user request. That flexibility is powerful, but it also creates ongoing operational costs. Running inference locally shifts much of that cost to the device, but introduces additional engineering responsibilities instead.

Neither approach is “free.” They simply spend resources differently.

E - Engineering Complexity

This is the dimension I think teams underestimate the most. Running AI locally means you're now responsible for :

None of these are impossible. They're simply part of owning an AI capability inside your application instead of behind a backend service.

The Most Common Mistake

The biggest mistake I see isn't choosing cloud. It isn't choosing on-device. It's choosing an AI runtime before deciding whether the feature should be local in the first place.

Architecture decisions should drive technology choices - not the other way around.

Once you've decided where the intelligence belongs, choosing the appropriate runtime becomes much easier.

When Cloud Is the Better Answer

This isn't an argument that everything should run on-device. Sometimes the cloud is objectively the better solution.

If your feature depends on :

• • Very large models
• • Frequently changing knowledge
• • Organization-wide data aggregation
• • Heavy computational workloads
• • Centralized reasoning

then cloud inference may provide a simpler, more scalable architecture.

There's no prize for making a problem harder than it needs to be. Good engineering is about making intentional tradeoffs, not following trends.

And Sometimes the Right Answer Is Both

Many of the strongest user experiences are hybrid. Imagine our fitness coach again.

Most coaching insights could be generated locally using summarized workout history, making them fast, private, and available offline. But if a user explicitly asks for a detailed training plan that combines historical activity with external research or larger reasoning models, the app might choose a cloud service instead.

The user still gets the best experience. The architecture simply uses the right tool for the right job.

Hybrid is often the most honest representation of the problem you're solving.

Applying This to Our Sample App

Throughout this series, we'll build a simple On-Device Fitness Coach. Its promise is intentionally modest :

A user opens the app and receives one short, encouraging insight based on recent activity patterns. Something like :

“You've been most consistent on days when your workouts stayed under 35 minutes. Maybe try a shorter session tomorrow.”

Here's how the PLACE framework shapes that design.

What stays on the device

• • Step counts
• • Workout duration
• • Rest days
• • Activity summaries
• • Local inference

The model receives only a bounded summary of recent activity rather than an unlimited stream of personal data.

What the app will never do

Just as importantly, we've decided what the app won't do. It won't :

• • Diagnose injuries
• • Recommend medical treatment
• • Make health claims
• • Present uncertain predictions as facts

If the model isn't confident or isn't available, the app falls back to a deterministic, rule-based coaching tip rather than guessing. Defining the boundaries of an AI feature is just as important as defining its capabilities.

A Simple Decision Matrix

Here's the mental model I use before selecting an AI stack.

It's intentionally simple. The goal isn't to replace architectural judgment.

It's to encourage asking the right questions before reaching for a particular SDK.

Looking Ahead

This article wasn't really about AI runtimes. It was about making one architectural decision deliberately instead of accidentally.

If you're evaluating an AI feature for your own app, try walking through the PLACE framework : Privacy, Latency, Availability, Cost, Engineering Complexity. Score your specific feature honestly. The answer will almost always tell you more than whichever AI framework happens to be trending this month.

In Part 2, we'll turn the PLACE framework into something you can score : an interactive decision scorecard for your specific feature. In Part 3, we'll assume you've made that placement decision and tackle the next question every Android developer eventually asks : ML Kit? MediaPipe? LiteRT? Gemini Nano? We'll compare each option and build a practical framework for choosing the right Android AI stack - not just the newest one.

TL;DR

• • Decide where intelligence lives before picking an SDK or model.
• • Use PLACE : Privacy, Latency, Availability, Cost, Engineering Complexity.
• • Cloud, on-device, and hybrid are all valid when the tradeoffs match the feature.
• • This series builds a modest On-Device Fitness Coach with clear scope and guardrails.
• • Part 2 : interactive scorecard to quantify cloud vs hybrid vs on-device.
• • Part 3 : choosing the Android AI stack (ML Kit, MediaPipe, LiteRT, Gemini Nano).