Offline AI on Android, with the limits shown clearly.
An offline assistant is not simply a cloud chatbot with cached screens. The model, tokenizer, runtime, conversation state, and requested feature all need to work without a provider connection.
At a glance
An offline AI assistant for Android runs its model and stores the active conversation on the phone after the model files have been downloaded. It should continue generating in airplane mode. Device RAM, storage, thermal limits, model architecture, quantization, and runtime support determine whether that experience is usable. Phos exposes a local route while keeping online BYOK and local-server routes visibly separate.
How Phos handles it
Three clear routes
Local model
After a compatible model is present, prompt processing and token generation occur on the Android device.
Local server
A computer can run the model, but the phone still needs network access to that machine; this is local-network AI, not phone-offline AI.
Bring your own key
Provider inference needs internet access and follows provider availability, pricing, and data handling.
Who this is for
Built for people with real private work to do.
Travelers and field workers who need assistance without dependable connectivity.
People who want sensitive drafts to avoid an inference provider.
Android users comparing GGUF, LiteRT-LM, and server-backed approaches.
Developers evaluating realistic phone limits before downloading large models.
Plain comparison
The point is control, not a louder chatbot.
These pages are for people comparing real options. Phos should win when someone wants privacy, local control, no account wall, and an assistant that still feels good to use.
| Feature | Phos | Typical cloud chatbot | Raw local app |
|---|---|---|---|
| Airplane-mode generation | Yes, when a compatible local model is loaded. | No. | Usually yes after model import. |
| Initial download | Required for the app and selected model. | Only the app; models remain remote. | Required, often through manual model files. |
| Hardware adaptation | Device-aware recommendations and fallback routes. | Phone hardware has little effect on inference. | User normally chooses model and runtime settings. |
| Long complex tasks | Depends on the local model; online routes remain optional. | Often stronger models and larger context. | Limited by model, RAM, context, and thermals. |
What must be local for offline chat
A credible airplane-mode test starts after setup: download the model, disable Wi-Fi and mobile data, reopen the app, load the same conversation, and generate a new response. If model discovery, tokenization, license checks, or inference silently depend on a service, the route is not fully offline. Features such as web search, remote speech services, link fetching, and cloud backups are separate and may still need connectivity.
The word 'offline' should describe the exact task, not the entire product forever. Phos can offer an offline local-model route and still offer BYOK or a local server for users who choose them. The honest design requirement is visible mode state: a user should not believe a remote request remained on the phone because the interface used the same chat screen.
RAM is the first practical gate
Model parameter count is not the same as runtime memory. Quantized weights occupy less storage than full precision, but the runtime also needs memory for model metadata, temporary buffers, the key-value cache, the tokenizer, the app, and Android itself. A file that technically fits in free storage can still fail to load or be killed under memory pressure. Longer context increases cache cost and can make an otherwise stable model unusable.
Start below the device's theoretical maximum. Test loading, a short prompt, a longer conversation, backgrounding, and a second load after the phone is warm. A smaller model that stays responsive is more useful than a larger model that repeatedly crashes. Published token-per-second figures are device-, model-, backend-, prompt-, and temperature-specific; they should never be copied into a generic phone promise.
Runtime and model format must match
GGUF is widely used with llama.cpp-style runtimes and offers many quantization choices. LiteRT-LM packages are prepared for Google's LiteRT stack and its supported CPU, GPU, and NPU paths. ExecuTorch uses exported PTE models and its own backend system. These formats are not interchangeable labels for the same file. A model architecture can be supported in one runtime before another, and a chat template error can make a correctly loaded model answer badly.
Google reported 52 tokens per second decode for Gemma 4 E2B on a Samsung S26 Ultra using LiteRT-LM's Android GPU backend. That is a useful proof of what one optimized combination can achieve, not a baseline for every Android phone. Unsloth separately documents about 40 tokens per second for a quantization-aware-trained Qwen3 0.6B on Pixel 8 through ExecuTorch. The model and measurement context belong beside the number.
Use an offline readiness checklist
Confirm enough free storage for both the download and unpacked model. Keep the phone charged, avoid maximum context at first, and close memory-heavy apps during the first load. Verify that the selected chat template matches the model's instruct format. Test a deterministic prompt twice so a runtime failure is not confused with ordinary sampling variation. Finally, repeat the test in airplane mode and inspect whether the response completes without a network permission prompt.
For sensitive use, also decide what happens after inference. A locally generated answer can still enter Android backups, screenshots, exported files, clipboard history, or notification previews. Offline inference narrows exposure at the model boundary; a complete privacy setup also controls storage and sharing. The strongest experience is transparent about both layers instead of treating the absence of a network call as the end of the analysis.
Direct answers
Frequently asked questions
Can every Android phone run an offline LLM?
No. Support depends on Android version, CPU features, available RAM and storage, model architecture, quantization, and runtime. Smaller models widen compatibility but still require device testing.
Is a local server the same as offline phone inference?
No. A local server can keep inference under your control, but the phone must reach the server over a network. On-device mode continues without that server connection.
What is the best model size for an Android phone?
There is no universal size. Begin with a conservative quantized model, test memory and thermals on the actual device, and increase only when the smaller model is stable.