GGUF vs LiteRT-LM on Android

Short answer

Use GGUF when broad open-model availability, many quantization choices, direct llama.cpp compatibility, and portability across clients matter. Use LiteRT-LM when the target model is available in a supported package and Google's optimized Android backends have been validated on the device class you care about. An app can support both without pretending the same model file flows through both runtimes.

Phos treats the runtime as a device-fit decision. A stable CPU GGUF path can be the right default on broad hardware, while a validated LiteRT-LM package can exploit GPU or NPU acceleration on supported devices.

Side-by-side architecture

GGUF strengths and costs

GGUF offers direct access to many model families and precision-size tradeoffs. llama.cpp supports integer quantization from very low bit levels through eight bit and provides Android bindings and benchmark tools. The same file can often move between desktop and mobile clients, making personal-server fallback easier. Users and developers can inspect metadata and choose model-specific releases such as Unsloth Dynamic quants.

The cost of flexibility is a larger compatibility matrix. Upload names, base versus instruct variants, chat templates, reasoning formats, multimodal projectors, runtime versions, context, and device features all matter. Community files vary in calibration and quality. Conservative curation is therefore part of the product, not an artificial restriction.

LiteRT-LM strengths and costs

LiteRT-LM is engineered around optimized on-device execution across Android, iOS, web, and embedded systems. Google describes CPU, GPU, and NPU backend selection through LiteRT and uses the stack in Google products and AI Edge Gallery. A supported prepared package can deliver strong performance without asking the user to reason about every tensor quant.

The tradeoff is that arbitrary GGUF uploads are not the interface. Model and operator support, packaging, conversion, and delegate behavior must match the runtime. NPU availability can be device- and vendor-specific. A product should keep a proven fallback when the optimized path is absent rather than making accelerator support a hard requirement.

Read benchmark claims with phase labels

Google reported 52 tokens per second decode for Gemma 4 E2B on Samsung S26 Ultra through OpenCL GPU and 56 tokens per second on iPhone 17 Pro through Metal. Its earlier Gemma 3 1B comparison on Galaxy S25 Ultra showed LiteRT ahead of llama.cpp on the tested CPU/GPU setup and reported NPU prefill about three times the GPU result. Those are official, valuable, configuration-specific measurements.

They do not show that every LiteRT-LM model is faster than every GGUF, or that decode and prefill improve equally. GGUF performance depends on runtime build and backend; LiteRT performance depends on package and delegate. Report model, quant/package, phone, backend, prefill length, decode length, repetitions, and thermals. Never copy a flagship result into a mid-range-phone promise.

Quantization quality is different from runtime speed

Unsloth's Dynamic 2.0 work is useful for understanding GGUF quality. It selectively quantizes layers and evaluates perplexity, KL divergence, and harder task benchmarks. Its Qwen3.5 results show that tensors differ in sensitivity and that imatrix methods can trade quality for a five-to-ten-percent speed cost. That research measures properties of exact quantized artifacts, not LiteRT versus GGUF Android performance.

Keep the questions separate: Does the model retain useful answer quality? Does the artifact fit? How quickly does this runtime process prompts and generate? Does the phone remain thermally stable? A single number cannot answer all four.

Decision rules for an Android assistant

Choose GGUF when the user brings a specific open model, needs desktop portability, or owns a phone where a stable CPU path is broader than accelerator support. Choose LiteRT-LM when the application supports a prepared model that demonstrates meaningful performance or power benefit on the target hardware. Preserve the same privacy and memory controls around both routes.

For Phos, the model recommendation should identify a preferred path and a fallback. If LiteRT-LM inspection rejects a package or delegate, the app should explain why. If a GGUF is too large, recommend a smaller file or a user-run server. The route should fail transparently rather than cross into hosted inference.

What to test before shipping

Run cold load, short prompt, long prefill, sustained decode, cancellation, background/resume, repeated load/unload, low-storage handling, memory pressure, model deletion, and airplane-mode generation. Test at least one lower-end supported phone, not only a flagship. Verify output format and chat template with a known prompt. Record energy and temperature when the feature is expected to run for minutes.

The best runtime is the one that produces a stable, reviewable assistant experience on the device in front of the user. Format loyalty is less important than truthful compatibility and a recoverable fallback.