Local AI on Android that starts with the device you have.
A polished local assistant sits above a demanding runtime problem: the model must fit, the chat template must match, the backend must support the hardware, and Android must keep the process alive.
At a glance
A local AI assistant for Android performs prompt processing and generation on the phone with a compatible model and runtime. The practical experience depends on available RAM, storage, CPU or accelerator support, model format, quantization, context length, and thermal behavior. Phos is designed to recommend a stable local path first while preserving user-chosen local-server and BYOK routes when a phone model is not the right tool.
How Phos handles it
Three clear routes
Local model
Model weights, prompt processing, generation, and local conversation state remain on the device for the local route.
Local server
The phone becomes a client to a user-controlled endpoint that may run a larger model on better hardware.
Bring your own key
A remote provider supplies inference under the user's key and current provider policy.
Who this is for
Built for people with real private work to do.
Android users who want a private assistant rather than a raw inference demo.
Developers comparing llama.cpp, LiteRT-LM, and ExecuTorch deployment paths.
People who need local tools, memory, and workflows around the model.
Users with mid-range phones who need graceful fallback instead of optimistic downloads.
Plain comparison
The point is control, not a louder chatbot.
These pages are for people comparing real options. Phos should win when someone wants privacy, local control, no account wall, and an assistant that still feels good to use.
| Feature | Phos | Typical cloud chatbot | Raw local app |
|---|---|---|---|
| Primary abstraction | Assistant workflows and visible inference routes. | Hosted conversation and account services. | Model file, sampler, and runtime controls. |
| Model compatibility | Curated recommendations plus supported import paths. | Chosen by vendor. | Broad but user-verified. |
| Failure handling | Device-aware guidance and explicit alternate routes. | Server retry or vendor fallback. | Logs and manual parameter changes. |
| Everyday continuity | Chats, memory controls, projects, and local tools. | Cloud history and integrations. | Varies by frontend. |
The runtime is part of the product
A model name alone does not specify an Android experience. llama.cpp consumes GGUF files and exposes mature CPU-oriented tooling. LiteRT-LM uses models prepared for Google's LiteRT stack and can target Android CPU, GPU, or supported NPU paths. ExecuTorch uses exported PTE programs and backend delegates. Each route has different supported operators, packaging, acceleration, and integration work.
The assistant must validate more than a filename. It needs architecture and tokenizer compatibility, the correct chat template, enough working memory, and an error path when loading fails. Phos aims to hide routine complexity without hiding the active boundary. Power users can still choose files and servers, while normal users should receive a conservative recommendation based on the actual device rather than a universal 'best local model.'
Build a memory budget before a model catalog
Start with the model's quantized weight size, then add the key-value cache for the intended context, runtime buffers, tokenizer state, application memory, and Android headroom. Available RAM at one moment is not a guarantee: the operating system can reclaim background processes, camera use can increase pressure, and vendor memory policies differ. Storage also needs temporary room during download and validation.
This is why a stable recommendation should sit below the theoretical maximum. First prove that the model loads repeatedly, generates a short answer, survives a longer thread, backgrounds safely, and unloads cleanly. Only then increase context or model size. A local assistant that protects the device from repeated out-of-memory failures earns more trust than one that advertises the largest possible checkpoint.
Acceleration claims require a complete test label
Google's May 2026 LiteRT-LM report measured Gemma 4 E2B at 52 generated tokens per second on a Samsung S26 Ultra through the Android OpenCL GPU backend. Unsloth's phone deployment guide reports roughly 40 tokens per second for Qwen3 0.6B on a Pixel 8 using an INT8-activation/INT4-weight QAT export through ExecuTorch. These results demonstrate optimized paths; they do not predict a different phone, model, prompt, context, or thermal state.
A useful benchmark label includes device model, Android version, runtime commit, backend, model and quantization, context, thread count, prompt processing rate, generation rate, repetitions, and temperature. Measure cold load separately from steady generation. Phos should present any future measurements with that context and otherwise prefer 'supported' or 'recommended' over an invented speed promise.
An assistant needs more than inference
Users need model discovery, safe downloads, storage status, cancellation, resumable transfer, chat persistence, attachment boundaries, and useful workflows. They also need to know when a tool accesses a file, when a provider receives context, and when a local server is unreachable. These behaviors determine whether local AI feels dependable after the first demo.
Phos organizes those concerns around explicit modes and Android-first flows. Local mode prioritizes privacy and availability. Local server mode uses hardware the user already controls. BYOK mode supplies optional capability without company-paid inference. The product goal is not to make every mode identical; it is to preserve a coherent assistant while clearly exposing the capability, cost, and privacy change each route introduces.
Direct answers
Frequently asked questions
Which Android runtime is best for local LLMs?
There is no universal winner. GGUF and llama.cpp offer broad model flexibility; LiteRT-LM can exploit optimized Google backends; ExecuTorch is strong for prepared deployment pipelines. Support must be verified per model and device.
Why can a model load once and fail later?
Android memory pressure, longer context, thermal state, other apps, and fragmented or temporary allocations can change between runs. Leave headroom and test repeated loads.
Does local AI need internet permission?
Inference itself need not. Model downloads, web tools, BYOK, and local-server discovery may use networking, so the active feature and route should be disclosed separately.