flutter_edge_ai runs any model a registered engine runs: .litertlm
(LiteRT-LM), .task/.bin (MediaPipe), ONNX (ORT-GenAI natively,
Transformers.js on web), GGUF (llama.cpp), or the OS model (built-in AI). The tables below list the
models we have run. On desktop (macOS, Windows, Linux) LiteRT-LM models must be
.litertlm — there is no MediaPipe engine there; ONNX models and the OS
built-in model also run on desktop.
For a small custom model, LiteTune provides the easier
end-to-end path from Hugging Face data/checkpoints through fine-tuning,
.litertlm conversion, verification, and bundling. It is alpha software, so
verify the converted model on the devices and backends you plan to ship.
Model file types#
Flutter Edge AI supports different model file formats, grouped into three types based on how chat templates are handled.
Type 1: SDK-managed templates#
-
.gguffiles: llama.cpp through the optional llama engine; the GGUF supplies its own chat template. -
.taskfiles: MediaPipe-optimized format for mobile (Android/iOS) and web. -
.litertlmfiles: LiteRT-LM format for Android, iOS, Desktop, and web (early preview). Never handled by MediaPipe — the two formats go to different engines. -
ONNX models (
ModelFileType.onnx): an ORT-GenAI model directory on native platforms, a Hugging Face repo through Transformers.js on web.
Chat templates are applied by the runtime rather than by your code — MediaPipe
for .task, LiteRT-LM for .litertlm, ORT-GenAI or Transformers.js for ONNX —
on every platform, iOS included.
Type 2: Template from the model type#
.binfiles: standard binary format..tflitefiles: LiteRT format (formerly TensorFlow Lite).
The runtime applies no template to these. Core wraps each message in the turn
markers of the ModelType you install with, so declare the right one (see the
ModelType reference).
Type 3: System OS models (no file)#
Gemini Nano (Android, via AICore / ML Kit GenAI), Apple Foundation Models
(iOS 26+/macOS), Windows AI Foundry / Phi Silica (Windows), and Gemini Nano
via the browser Prompt API (Web — Gemini Nano in Chrome, Phi-4-mini in Edge) are
built into the
OS/browser — there is no model file to bundle or download; the platform owns the
weights. Add flutter_edge_ai_builtin_ai, register
BuiltInAiEngine(), and use ModelFileType.builtIn. Availability is
device-gated — Gemini Nano needs a phone on Google's Prompt API device list, Apple FM needs Apple
Intelligence enabled on iPhone 15 Pro+/M-series, and Windows needs Windows 11
25H2+ on Copilot+-class hardware and a packaged app (nothing to configure to
build — flutter_local_ai resolves the Windows App SDK projection itself). On Android
the package requires minSdk 26 and Kotlin 2.3.21.
👉 See Built-in AI for the full setup, the availability
probe (BuiltInAi.availability() / BuiltInAi.ensureReady()), and fallback
guidance.
ModelFileType is what selects the engine — it is not inferred from the file
name. installModel defaults it to ModelFileType.task, so declare it
explicitly: ModelFileType.litertlm for .litertlm files (omitting it routes the
model to MediaPipe, which cannot read that format), ModelFileType.task for
.task, ModelFileType.binary for .bin and .tflite, ModelFileType.onnx
for ONNX models (flutter_edge_ai_onnx), and
ModelFileType.builtIn for OS-provided models. ModelFileType.gguf (core
2.3.2) selects the optional llama.cpp engine,
flutter_edge_ai_llama.
Format by platform#
| Format | Android | iOS | Web | Desktop | Use Case |
|---|---|---|---|---|---|
.gguf |
✅ | ✅ | ✅ | ✅ | Optional llama.cpp engine; model-provided templates, qualification limits |
.task |
✅ | ✅ ⁴ | ✅ | ❌ |
Every model shipped as
.task
(see the tables below). MediaPipe is maintenance-only — new apps should use
.litertlm
|
.litertlm |
✅ | ✅ ¹ | ⚠️ ² | ✅ |
Every model shipped as
.litertlm
(see the tables below); EmbeddingGemma 2 embeddings (every platform; web from litertlm 1.13.1)
|
-web.task |
❌ | ❌ | ✅ | ❌ | Web-specific MediaPipe builds (Gemma 4, Gemma 3 1B and Gemma 3 270M in the current catalog) |
.bin |
✅ | ✅ | ✅ | ❌ | Core formats the prompt from the ModelType |
.tflite |
✅ | ✅ | ✅ | ✅ |
LLM via
ModelFileType.binary
(MediaPipe, mobile and web), and LiteRT embedding models (EmbeddingGemma, Gecko) on every platform
|
onnx |
✅ | ✅ | ✅ | ✅ | ONNX models — an ORT-GenAI model directory on native, a Hugging Face repo via Transformers.js on Web — ONNX |
builtIn (no file) |
✅ | ✅ | ✅ | ⚠️ ³ | OS/browser model — Built-in AI |
¹ iOS .litertlm runs on the FFI engine — vision and audio supported on physical
devices. The Simulator stays CPU-only because Metal sim has a 256 MB
single-allocation cap.
² Web .litertlm is an early preview via @litert-lm/core. A language
model gets text, function calling and thinking, and only from a dedicated web
export (-web.litertlm) — Gemma 4 and Gemma 3n today; a native language-model
.litertlm installs on web and then fails at the first generation, and no
.litertlm language model takes images or audio on web. EmbeddingGemma 2 is
the exception: its generic .litertlm bundles embed text and images on web
through WebGPU, and the 740m audio from litertlm 1.13.2 (see
Embedding models). See the feature matrix in
Troubleshooting. For image input to a chat model on
web use a MediaPipe -web.task build. LoRA works only on MediaPipe .task.
³ On desktop the OS model exists on macOS (Apple Foundation Models) and
Windows (AI Foundry; Windows 11 25H2+ on Copilot+-class hardware, packaged
app) — not on Linux. Every platform still gates it at
runtime: probe with BuiltInAi.availability().
⁴ On iOS run .task models on PreferredBackend.cpu: on the GPU, MediaPipe is
killed for memory while loading even Gemma 3 1B. See
MediaPipe → Backends.
Model capabilities#
| Model Family | Best For | Function Calling | Thinking Mode | Vision | Languages | Size |
|---|---|---|---|---|---|---|
| Gemma 4 E2B | Next-gen multimodal chat — text, image, audio | ✅ | ✅ ⁵ | ✅ | Multilingual | 2.6GB |
| Gemma 4 E4B | Next-gen multimodal chat — text, image, audio | ✅ | ✅ ⁵ | ✅ | Multilingual | 3.7GB |
| Gemma3n | On-device multimodal chat and image analysis | ⚠️ ⁶ | ❌ | ✅ | Multilingual | 3.1-4.9GB |
| FastVLM 0.5B | Fast vision-language inference | ❌ | ❌ | ✅ | Multilingual | 1.2GB |
| Qwen2-VL 2B | Vision-language chat (image + text) | ❌ | ❌ | ✅ | Multilingual | 1.8GB |
| SmolVLM2 500M | Compact vision-language model | ❌ | ❌ | ✅ | Multilingual | 0.36GB |
| LLaVA-OneVision 0.5B | Compact vision-language model | ❌ | ❌ | ✅ | Multilingual | 0.83GB |
| VibeVoice-ASR 2.2B (BitNet) | Speech to text: a spoken clip in, its transcript out | ❌ | ❌ | ❌ | en, zh, fr, it, ko, pt, vi | 2.0GB |
| Phi-4 Mini | Advanced reasoning and instruction following | ✅ | ❌ | ❌ | Multilingual | 3.9GB |
| Phi-4 Mini Reasoning | Step-by-step reasoning | ❌ | ⚠️ ‡ | ❌ | Multilingual | 2.8GB |
| DeepSeek R1 | High-performance reasoning and code generation | ✅ | ✅ | ❌ | Multilingual | 1.8–1.9GB |
| Qwen3 0.6B | Compact multilingual chat with function calling | ✅ | ✅ | ❌ | Multilingual | 345MB |
| Qwen 2.5 | Strong multilingual chat and instruction following | ✅ | ❌ | ❌ | Multilingual | 0.5-1.6GB |
| Gemma 3 1B | Balanced and efficient text generation | ❌ | ❌ | ❌ | Multilingual | 0.6GB |
| Gemma 3 270M | Ideal for fine-tuning (LoRA on the .task build) for specific tasks |
❌ | ❌ | ❌ | Multilingual | 0.3GB |
| FunctionGemma 270M | Specialized for function calling on-device | ✅ | ❌ | ❌ | Multilingual | 284MB |
| SmolLM 135M | Ultra-compact, resource-constrained devices | ❌ | ❌ | ❌ | English | 167MB |
| LFM2.5 230M (Liquid AI) | Smallest desktop entry; no HF token needed | ❌ | ❌ | ❌ | Multilingual | 177MB |
| SmolLM3 3B | Multilingual small LLM with reasoning mode | ❌ | ⚠️ ‡ | ❌ | Multilingual | 2.0GB |
| TranslateGemma 4B † | Single-shot 55-language translation | ❌ | ❌ | ❌ | 55 languages | 2.0–3.9GB |
| Phi-3.5 Mini (ONNX) | Text chat through ORT-GenAI on native platforms | — | ❌ | ❌ | Multilingual | 2.8GB |
| Qwen 2.5 0.5B (ONNX) | Text chat in the browser through Transformers.js | — | ❌ | ❌ | Multilingual | 0.5GB |
| Gemini Nano (built-in) | The OS model on Android, and in Chrome | ✅ prompt-based | ❌ | ✅ Android | varies | system |
| Apple Foundation Models (built-in) | The OS model on iOS and macOS | ✅ prompt-based | ❌ | ⚠️ OS 27 | varies | system |
| Phi Silica (built-in) | The OS model on Windows | ✅ prompt-based | ❌ | ❌ | varies | system |
⁵ Gemma 4 Thinking Mode needs LiteRT-LM (.litertlm): Android, iOS, Desktop
and Web (measured on Gemma 4 E2B in Chrome). MediaPipe Web cannot enable it and
ONNX Web ignores it.
Qwen3 is different: core parses its emitted <think> tags on every platform,
including Web.
⁶ Gemma3n function calling: the downloadable catalog enables it only on the E4B
.litertlm build.
— not run. The ONNX engine is text-only and ignores enableThinking; see
ONNX. Built-in AI calls functions through the prompt; Apple
Foundation Models takes images only on OS 27 with an OS 27 SDK (not verified on
a device yet), and the Chrome Prompt API and Phi Silica are text-only — see
Built-in AI, which also covers Phi-4-mini in Edge.
‡ SmolLM3 runs as ModelType.general, Phi-4 Mini Reasoning as
ModelType.phi. These .litertlm bundles stream their reasoning
on a thought channel, which arrives as ThinkingResponse; a model without such a
channel keeps its raw thinking text inside the answer. See
Thinking Mode.
† TranslateGemma is CPU-only for now. Google hasn't released a
mobile/desktop .litertlm bundle
(HF discussion #5).
The community-converted bundle from
barakplasma/translategemma-4b-it-android-task-quantized
runs correctly on the CPU and returns only padding on the GPU.
Measured here on an M4 Pro (macOS, Metal), both published artifacts, same prompt and the same session code, backend the only variable:
| artifact | PreferredBackend.cpu | PreferredBackend.gpu |
|---|---|---|
int4-generic (2.0 GB, INT4 blockwise) | Guten Morgen | 997 <pad> tokens, nothing else |
dynamic_int8-generic (3.9 GB, INT8 channelwise) | Guten Morgen | 997 <pad> tokens, nothing else |
Nothing fails: the Metal engine is created, activeBackend reports gpu (so it
is not a silent fallback), generation runs and emits padding until it reaches the
context limit. A gemma-4-E2B-it.litertlm bundle on the same machine, the same code path and the same backend answers correctly with no padding, so this is not the Metal path in general. Earlier revisions of this page said the bundle crashed the LiteRT
GPU partitioner because its EMBEDDING_LOOKUP weights stay float32. Nothing in
the runs above crashes, so that description does not hold; the repository has
published these .litertlm artifacts since 2026-03-31, so it did not describe a
later change either. Reproducing across two different quantization recipes also
rules the bit width out.
Tracked at
LiteRT-LM#1748.
Use PreferredBackend.cpu for this model (≈90 s prefill on a 4 B int4 bundle on
M-series Macs). If you are converting it yourself, AI Edge Quantizer is the
supported route today — the litert-lm quantization CLI announced in that thread
never shipped — with published recipes and Model Explorer for the layer regexes
(maintainer's pointers).
ModelType reference#
When installing models, specify the correct ModelType:
| Model Family | ModelType | Examples |
|---|---|---|
| Gemma 4 | ModelType.gemma4 |
Gemma 4 E2B, Gemma 4 E4B (native function-call tokens) |
| Gemma 3 / Gemma3n | ModelType.gemmaIt |
Gemma 3 1B, Gemma 3 270M, Gemma3n E2B/E4B, TranslateGemma 4B |
| DeepSeek | ModelType.deepSeek | DeepSeek R1 |
| Qwen 2.5, Qwen3-2507 | ModelType.qwen |
Qwen 2.5 1.5B, Qwen 2.5 0.5B, Qwen3 4B Instruct/Thinking 2507 (installed from a Hugging Face manifest, 2507 still resolves to
qwen3
; pass
qwen
yourself)
|
| Qwen 3 (the original hybrid) | ModelType.qwen3 |
Qwen3 0.6B |
| Qwen 3.5 / 3.6 / 3.8 | ModelType.qwen35 |
Qwen3.5 0.8B, 2B, 4B |
| Phi-4 | ModelType.phi |
Phi-4 Mini (parses Phi's own tool-call markers), Phi-4 Mini Reasoning |
| Llama | ModelType.llama |
Llama 3.2 Instruct (its sampling defaults; not in the tables above) |
| Hammer | ModelType.hammer |
Hammer 2.1 (its sampling defaults; not in the tables above) |
| FunctionGemma | ModelType.functionGemma |
FunctionGemma 270M IT |
| General | ModelType.general |
FastVLM 0.5B, SmolLM 135M, LFM2.5 230M (Liquid AI), SmolLM3 3B, Qwen2-VL 2B, SmolVLM2 500M, LLaVA-OneVision 0.5B, VibeVoice-ASR, Phi-3.5 Mini (ONNX), the built-in OS models |
Gemma 4 (ModelType.gemma4) and FunctionGemma on a .litertlm route their native
tool-call tokens through the LiteRT-LM SDK's chat-template path. For Gemma 3 and
earlier, keep ModelType.gemmaIt; a .task FunctionGemma keeps the text format
flutter_edge_ai renders itself.
Usage example:
// Gemma models
await FlutterEdgeAi.installModel(modelType: ModelType.gemmaIt)
.fromNetwork(url).install();
// DeepSeek models
await FlutterEdgeAi.installModel(modelType: ModelType.deepSeek)
.fromNetwork(url).install();
// Phi-4 (its own type — parses Phi's tool-call markers)
await FlutterEdgeAi.installModel(modelType: ModelType.phi)
.fromNetwork(url).install();
Supported models & platforms#
The LiteRT-LM (.litertlm) language models come from the
Model Registry, the same files the catalog installs
installs. A cell shows, per backend, the best that any build of the model has
on that platform: 📝 documented to work, ⚪ the package runs it but no run has
been recorded, ❌ known broken (✅ verified once runs are recorded). The
registry page lists every build with its pinned file and the source of each
mark.
| Model | Size | Android | iOS | macOS | Windows x64 | Linux x64 | Linux arm64 | Web | HF token |
|---|---|---|---|---|---|---|---|---|---|
| DeepSeek R1 Distill Qwen 1.5B | 1.83 GB | GPU ⚪ CPU ⚪ | GPU ⚪ CPU ⚪ | GPU ⚪ CPU ⚪ | GPU ⚪ CPU ⚪ | GPU ⚪ CPU ⚪ | GPU ⚪ CPU ⚪ | no | |
| FastVLM 0.5B | 1.16 GB | GPU ⚪ CPU ⚪ NPU ⚪ | GPU ⚪ CPU ⚪ | GPU 📝 CPU ⚪ | GPU 📝 CPU ⚪ | GPU 📝 CPU ⚪ | GPU ⚪ CPU ⚪ NPU ⚪ | no | |
| FunctionGemma 270M (Flutter demo fine-tune) | 284 MB | GPU ⚪ CPU ⚪ | GPU ⚪ CPU ⚪ | GPU ⚪ CPU ⚪ | GPU ⚪ CPU ⚪ | GPU ⚪ CPU ⚪ | GPU ⚪ CPU ⚪ | no | |
| FunctionGemma 270M | 284 MB | GPU 📝 CPU ⚪ | GPU 📝 CPU ⚪ | GPU 📝 CPU ⚪ | GPU 📝 CPU ⚪ | GPU 📝 CPU ⚪ | GPU ⚪ CPU ⚪ | no | |
| Gemma 3 1B | 584 MB | GPU ⚪ CPU ⚪ NPU 📝 | GPU ⚪ CPU ⚪ | GPU 📝 CPU ⚪ | GPU 📝 CPU ⚪ | GPU 📝 CPU ⚪ | GPU ⚪ CPU ⚪ NPU ⚪ | yes | |
| Gemma 3 270M | 304 MB | GPU ⚪ CPU ⚪ NPU ⚪ | GPU ⚪ CPU ⚪ | GPU 📝 CPU ⚪ | GPU 📝 CPU ⚪ | GPU 📝 CPU ⚪ | GPU ⚪ CPU ⚪ | yes | |
| Gemma 3n E2B | 3.04–3.66 GB | GPU 📝 CPU ⚪ NPU ⚪ | GPU 📝 CPU ⚪ | GPU 📝 CPU ⚪ | GPU 📝 CPU ⚪ | GPU 📝 CPU ⚪ | GPU ⚪ CPU ⚪ | GPU 📝 | yes |
| Gemma 3n E4B | 4.28–4.92 GB | GPU 📝 CPU ⚪ | GPU 📝 CPU ⚪ | GPU 📝 CPU ⚪ | GPU 📝 CPU ⚪ | GPU 📝 CPU ⚪ | GPU ⚪ CPU ⚪ | GPU 📝 | yes |
| Gemma 4 E2B | 2.01–2.59 GB | GPU ✅ CPU ✅ NPU ✅ | GPU ✅ CPU ✅ | GPU ✅ CPU ✅ | GPU ✅ CPU ✅ NPU ✅ | GPU ✅ CPU ✅ | GPU 📝 CPU 📝 NPU 📝 | GPU 📝 | no |
| Gemma 4 E4B | 2.97–3.66 GB | GPU 📝 CPU ⚪ | GPU 📝 CPU ⚪ | GPU 📝 CPU ⚪ | GPU 📝 CPU ⚪ | GPU 📝 CPU ⚪ | GPU ⚪ CPU ⚪ | GPU 📝 | no |
| LFM2.5 230M | 177–266 MB | GPU 📝 CPU ⚪ | GPU 📝 CPU ⚪ | GPU 📝 CPU ⚪ | GPU 📝 CPU ⚪ | GPU 📝 CPU ⚪ | GPU ⚪ CPU ⚪ | no | |
| LLaVA-OneVision 0.5B | 829 MB | GPU 📝 CPU ⚪ | GPU 📝 CPU ⚪ | GPU 📝 CPU ⚪ | GPU 📝 CPU ⚪ | GPU 📝 CPU ⚪ | GPU ⚪ CPU ⚪ | no | |
| Phi-4 Mini Instruct | 3.91 GB | GPU ⚪ CPU ⚪ | GPU ⚪ CPU ⚪ | GPU 📝 CPU ⚪ | GPU 📝 CPU ⚪ | GPU 📝 CPU ⚪ | GPU ⚪ CPU ⚪ | no | |
| Phi-4 Mini Reasoning | 2.78 GB | GPU 📝 CPU ⚪ | GPU 📝 CPU ⚪ | GPU 📝 CPU ⚪ | GPU 📝 CPU ⚪ | GPU 📝 CPU ⚪ | GPU ⚪ CPU ⚪ | no | |
| Qwen2-VL 2B | 1.78 GB | GPU 📝 CPU ⚪ | GPU 📝 CPU ⚪ | GPU 📝 CPU ⚪ | GPU 📝 CPU ⚪ | GPU 📝 CPU ⚪ | GPU ⚪ CPU ⚪ | no | |
| Qwen 2.5 1.5B Instruct | 1.60–6.18 GB | GPU ⚪ CPU ⚪ | GPU ⚪ CPU ⚪ | GPU ⚪ CPU 📝 | GPU ⚪ CPU 📝 | GPU ⚪ CPU 📝 | GPU ⚪ CPU ⚪ | no | |
| Qwen3 0.6B | 345–498 MB | GPU ⚪ CPU 📝 NPU ⚪ | GPU ⚪ CPU 📝 | GPU ⚪ CPU 📝 | GPU ⚪ CPU 📝 | GPU ⚪ CPU 📝 | GPU ⚪ CPU ⚪ | no | |
| SmolLM3 3B | 2.00–3.13 GB | GPU 📝 CPU ⚪ | GPU 📝 CPU ⚪ | GPU 📝 CPU ⚪ | GPU 📝 CPU ⚪ | GPU 📝 CPU ⚪ | GPU ⚪ CPU ⚪ | no | |
| SmolVLM2 500M | 361 MB | GPU 📝 CPU ⚪ | GPU 📝 CPU ⚪ | GPU 📝 CPU ⚪ | GPU 📝 CPU ⚪ | GPU 📝 CPU ⚪ | GPU ⚪ CPU ⚪ | no | |
| TranslateGemma 4B | 2.01–3.92 GB | GPU ❌ CPU 📝 | GPU ❌ CPU 📝 | GPU ❌ CPU 📝 | GPU ❌ CPU 📝 | GPU ❌ CPU 📝 | GPU ❌ CPU ⚪ | no |
The Qwen3 0.6B build the catalog installs is
Qwen3-0.6B_dynamic_wi4b32_afp32.litertlm (345 MB). Qwen3-0.6B.litertlm
(614 MB) failed to load from flutter_edge_ai_litertlm 1.7.0 until the repo
replaced its tokenizer on 2026-10-13 at 15:53 UTC
(#659, upstream
LiteRT-LM #3865); a
copy downloaded before that still fails — uninstall it and install it again.
Other formats#
Not in the registry: MediaPipe .task files, ONNX models, the OS models, and
VibeVoice.
| Model | Format | Size | Desktop | Mobile | Web | HF token |
|---|---|---|---|---|---|---|
| Gemma 4 E2B | .task |
2.0GB | ❌ | ❌ | ✅ | no |
| Gemma 4 E4B | .task |
3.0GB | ❌ | ❌ | ✅ | no |
| Gemma3n E2B | .task |
3.1GB | ❌ | ✅ | ❌ | yes |
| Gemma3n E4B | .task |
4.4GB | ❌ | ✅ | ❌ | yes |
| Gemma 3 1B | .task |
0.5GB | ❌ | ✅ | ✅ | yes |
| Gemma 3 270M | .task |
0.3GB | ❌ | ✅ | ✅ | yes |
| DeepSeek R1 Distill Qwen 1.5B | .task |
1.86GB | ❌ | ✅ | ❌ | no |
| Qwen 2.5 1.5B | .task |
1.6GB | ❌ | ✅ | ❌ | no |
| Qwen 2.5 0.5B | .task |
0.5GB | ❌ | ✅ | ❌ | no |
| SmolLM 135M | .task |
167MB | ❌ | ✅ | ❌ | no |
| Phi-4 Mini | .task |
3.9GB | ❌ | ✅ | ❌ | no |
| FunctionGemma 270M | .task |
284MB | ❌ | ✅ | ❌ | no |
| VibeVoice-ASR-BitNet ⁹ | .litertlm |
2.0GB | ✅ macOS | ✅ Android | ❌ | no |
| Phi-3.5 Mini ⁸ | ONNX | 2.8GB | ✅ | ✅ | — | no |
| Qwen 2.5 0.5B ⁷ | ONNX | 0.5GB | ❌ | ❌ | ✅ | no |
| Gemini Nano | built-in | system | ❌ | ✅ Android | ✅ Chrome | — |
| Apple Foundation Models | built-in | system | ✅ macOS | ✅ iOS | ❌ | — |
| Phi Silica | built-in | system | ✅ Windows | ❌ | ❌ | — |
.task files run through MediaPipe, which has no desktop
engine.
⁷ On web, Qwen 2.5 0.5B runs as the ONNX build from onnx-community, through
Transformers.js (flutter_edge_ai_onnx). That build is web-only.
⁸ An ORT-GenAI model directory — fromHuggingFace installs its CPU folder,
cpu_and_mobile/cpu-int4-awq-block-128-acc-level-4 — run on macOS arm64,
Linux x64, Windows x64, Android arm64 and iOS arm64. It peaks at about 3.7 GB
of RAM, so plan for a 6 GB+ phone. — means not run.
⁹ Speech recognition, not chat: install it as ModelType.general (the
repo's litertlm_manifest.json sets maxTokens: 2048 and audio — pass
getActiveModel(defaults: r.runtime)), send the clip as audioBytes
with the
text This is a X.XX seconds audio, please transcribe it. (the clip's length),
and decode greedily (topK: 1). Leave preferredAudioBackend
unset: the audio
encoder has to run on the CPU, and on a GPU delegate it returns an empty
transcript; the language model may run on the GPU. ModelType.general sends
the audio before the text, which this model needs
(multimodal order).
Measured with the language model on the GPU of a Galaxy S24, 80 synthesized
clips of 0.8–3.9 s: word error rate 0.32 under 1.5 s, 0.18 at 1.5–2.5 s and
0.07 from 2.5 s, with 9–11 s to the first token whatever the length, because
the audio encoder always processes a 30 s window on the CPU. On macOS, with
the language model on Metal, 8 of those clips were transcribed the same way.
Web cannot send audio to an LLM.
The built-in models download nothing and need no token; availability is decided by the device at run time, and Edge runs Phi-4-mini through the same Prompt API — see Built-in AI.
Installation sources#
From the model catalog#
flutter_edge_ai 2.5.5, with the flutter_edge_ai_litertlm and
flutter_edge_ai_speech releases that ship the catalog, installs a
Model Registry build by name through the usual installers.
Each build carries its model type, file type, Hugging Face commit, file size
and install file name, so there is no URL to copy and nothing to keep in step:
// A language model, an embedder, a speech model.
await FlutterEdgeAi.installModelFromCatalog(LiteRtLmModels.gemma4E2bIt.generic)
.withProgress((percent) => print('downloading: $percent%'))
.install();
await FlutterEdgeAi.installEmbedder()
.fromCatalog(
LiteRtEmbedders.embeddinggemma300m.seq256,
token: huggingFaceToken, // the repository is gated
)
.install();
await FlutterEdgeAi.installStt().fromCatalog(LiteRtSpeechModels.whisperTiny.f32)
.install();
// Every model the registered packages offer, and the build for this device.
for (final model in FlutterEdgeAi.catalog.models) {
final variant = model.variantFor(backend: PreferredBackend.gpu);
print('${model.id}: ${variant?.id ?? 'nothing for the GPU here'}');
}
variantFor leaves out a build this runtime cannot run: a build compiled for
an NPU whose vendor's dispatch the package does not ship for this processor
architecture, a web export on a phone, any native build on a host the
package's native libraries are not published for. A build compiled for an NPU
is picked only for the SoC the app names (variantFor(backend: PreferredBackend.npu, npu: …); without
npu: it throws), because the
package does not read the device's SoC. The app builds that target in the
spelling the compiler stamps into the file, not the one Android reports:
LiteRtLmNpuTarget(vendor: 'Qualcomm', soc: 'SM8750') (Android says QTI),
'Google' with 'Tensor_G5', 'MediaTek' with
'mt6993' (any case),
'IntelOpenVINO' with 'PTL'. An install from the catalog refuses, before it
downloads anything, a build whose backendsHere is empty here; whether this
device has that NPU, and whether the app bundled the vendor's stack
(qualcomm_npu, google_tensor_npu, mediatek_npu), is known only when the
model loads. A
build compiled for an NPU names the context it was compiled for
(requiredMaxTokens); keep maxTokens at or below it. After the download,
and before the model becomes active, each file's size is checked against the
registry (not on the web): a file of another size is deleted with its record,
unless an active model reads it, and install() throws
ModelValidationException. An install that would
write over a file an active model of any kind reads, from another source (a
model installed earlier from a /resolve/main/ URL under the same name, or
an older pin), is refused with a StateError before it downloads (not on the
web). Its message names that model's kind, its name and the call that
uninstalls it (uninstallModel, uninstallEmbedder, uninstallStt
or
uninstallTts): uninstall it first, then install. On the web a file over
2 GiB is refused before it downloads unless the app initializes with
webStorageMode: WebStorageMode.streaming, the only mode that holds it.
withProgress weights the files by their pinned sizes, and reports 100 once
the model is active. A network install outside the catalog gets the same
checks from expectedSize: (bytes) on installModel().fromNetwork,
installEmbedder().modelFromNetwork/tokenizerFromNetwork and
installStt().modelFromNetwork/tokenizerFromNetwork (core 2.5.5); a
negative value throws ArgumentError. An LLM is installed under the registry's install name (for example
phi-4-mini-reasoning.litertlm), which isModelInstalled,
getModelPath and
uninstallModel take. install(activate: false) installs an LLM without
making it active. A LoRA is not installed through the catalog.
The catalog pins every file to a Hugging Face commit. A model installed
earlier from a /resolve/main/ URL is a different source, so its first
install from the catalog downloads the file again (once); later installs from
the catalog find it installed.
From a URL, an asset or a file#
// Network — .litertlm is the cross-platform default (Android/iOS/Desktop).
// For mobile-only or web-only apps you can substitute a .task URL — and drop
// the fileType, which defaults to ModelFileType.task. On iOS, run a .task
// model on PreferredBackend.cpu.
await FlutterEdgeAi.installModel(
modelType: ModelType.gemmaIt,
fileType: ModelFileType.litertlm,
)
.fromNetwork('https://example.com/model.litertlm', token: 'optional')
.install();
// Flutter assets
await FlutterEdgeAi.installModel(
modelType: ModelType.gemmaIt,
fileType: ModelFileType.litertlm,
)
.fromAsset('assets/models/model.litertlm')
.install();
// Native bundle
await FlutterEdgeAi.installModel(
modelType: ModelType.gemmaIt,
fileType: ModelFileType.litertlm,
)
.fromBundled('model.litertlm')
.install();
// External file (native only)
await FlutterEdgeAi.installModel(
modelType: ModelType.gemmaIt,
fileType: ModelFileType.litertlm,
)
.fromFile('/path/to/model.litertlm')
.install();
Hugging Face#
Install a model straight from a Hugging Face repo. The resolver that reads a repo's deployment manifest ships with the engine, so registering the engine is enough — there is no separate resolver list to maintain.
One call (manifest-driven) — omit file, and fromHuggingFace(repo)
resolves the repo's manifest (e.g. litertlm_manifest.json) at install time, installs the right revision-pinned variant for the device, and returns the manifest's overridable runtime defaults on the result:
// The engine carries its resolver — nothing else to register.
await FlutterEdgeAi.initialize(inferenceEngines: [LiteRtLmEngine()]);
final install = await FlutterEdgeAi.installModel(
modelType: ModelType.qwen, // a specific type wins over the manifest's
fileType: ModelFileType.litertlm, // selects the litertlm resolver
)
// The manifest types this repo `qwen3`; 2507 is a `qwen` model (see the
// ModelType table). `ModelType.general` would take the manifest's type.
.fromHuggingFace('litert-community/Qwen3-4B-Thinking-2507')
.install();
final model = await FlutterEdgeAi.getActiveModel(defaults: install.runtime);
// `minOutputTokens` is a FLOOR, not a cap — leave `maxOutputTokens` unset (or
// keep it >= `install.runtime?.minOutputTokens`). Passing the floor as the cap
// would truncate a reasoning model mid-thought.
final session = await model.openSession(
enableThinking: install.runtime?.thinkingDeclared ?? false,
);
await session.close();
.onnx repos install the whole ORT-GenAI directory — the ONNX resolver (carried by
OnnxEngine) lists the repo, picks a CPU execution-provider folder (the bundled runtime is CPU-only; pin one with
OnnxHuggingFaceResolver(variant: …)) and downloads every file in it; see ONNX.
.builtIn repos surface that OS models have no Hugging Face file.
Explicit file — pass file, and fromHuggingFace(repo, file:)
resolves …/resolve/<revision>/<file> and installs it directly, for any fileType
(no manifest needed; the HF token is applied to huggingface.co automatically):
await FlutterEdgeAi.installModel(
modelType: ModelType.gemmaIt,
fileType: ModelFileType.litertlm,
)
.fromHuggingFace('litert-community/Gemma3-1B-IT', file: 'gemma3-1b-it-int4.litertlm', token: 'optional')
.install();
Inspect first — resolveHuggingFace(repo, fileType:) returns the resolved identity plus overridable runtime defaults WITHOUT installing, so you can inspect the variant, its notes, and defaults before committing:
final r = await FlutterEdgeAi.resolveHuggingFace(
'litert-community/Qwen3-4B-Thinking-2507',
fileType: ModelFileType.litertlm,
);
// … inspect r.file / r.notes / r.runtime …
await FlutterEdgeAi.installModel(
// 2507 resolves to qwen3; ModelType.qwen is the right type for it.
modelType: ModelType.qwen,
fileType: r.fileType,
)
.fromNetwork(r.url) // r.url pins the resolver's revision
.install();
The resolver lives in the engine package — LitertlmManifestResolver in flutter_edge_ai_litertlm
reads litertlm_manifest.json — and rides on the engine via HuggingFaceResolverSource, so
initialize(inferenceEngines: […]) auto-registers it. Pass initialize(huggingFaceResolvers: [...])
only to override an engine's default (e.g. LitertlmManifestResolver(revision: 'abc123')
to pin a commit).
r.family is the model family the resolver found: a ModelType for a language model (r.family as ModelType? ?? ModelType.general), null when it could not tell.
Each resolver serves one role — its type: ModelManagementType.inference, embedding,
stt or tts. resolveHuggingFace(repo, type: …, fileType: …) asks only the resolvers of that role (inference by default), so an
.onnx language model and an .onnx embedder each reach their own. A backend package carries its resolver the way an engine does:
initialize registers the resolvers of inferenceEngines, embeddingBackends,
sttBackends and ttsBackends, and refuses a provider whose resolver serves another role than its list. The explicit
huggingFaceResolvers: list is registered first and wins a tie within its role.
No published backend ships an embedding resolver yet (the LiteRT-LM, ONNX and built-in AI resolvers serve the inference role). With your own
HuggingFaceResolver whose type is ModelManagementType.embedding, passed in
initialize(huggingFaceResolvers: [...]), an embedder installs the same way:
final install = await FlutterEdgeAi.installEmbedder()
.fromHuggingFace('org/embedder', fileType: ModelFileType.onnx)
.install();
final embedder = await FlutterEdgeAi.getActiveEmbedder(
defaults: install.runtime,
);
It installs the model file and, when the resolver names one (r.tokenizerFile), its tokenizer, and, on native platforms, checks the sizes the resolver gives before the model becomes active (on the web they only drive progress and the 2 GiB refusal outside streaming mode); a resolution that is a model directory is refused with
UnsupportedError for now. getActiveEmbedder, getActiveStt and
getActiveTts take defaults: like getActiveModel — an explicit argument wins over a default, which wins over the SDK's — and each reads only the fields of its role (ModelRuntimeDefaults.language
for speech).
Source capabilities#
| Source Type | Platform | Progress | Resume | Authentication | Use Case |
|---|---|---|---|---|---|
| NetworkSource | All | ✅ Detailed | ⚠️ Server-dependent | ✅ Supported | HuggingFace, CDNs, private servers |
| AssetSource | All | ⚠️ End only | ❌ No | ❌ N/A | Models bundled in app assets |
| BundledSource | All | ⚠️ End only | ❌ No | ❌ N/A | Native platform resources |
| FileSource | Native (no Web) | ⚠️ End only | ❌ No | ❌ N/A | User-selected files (file picker) |
Resume after interruption is server-dependent and not supported by the HuggingFace CDN — flutter_edge_ai uses smart retry logic with exponential backoff and automatic restart instead. See Troubleshooting.
Android foreground service (large downloads)#
Android has a 9-minute background execution limit. A foreground service shows a notification and exempts the download from battery-optimization kills, and you have to ask for it — the default configures no notification, and the platform will not start the service without one, so file size alone changes nothing:
// DEFAULT — no foreground service; pass foreground: true to get one
await FlutterEdgeAi.installModel(modelType: ModelType.gemmaIt)
.fromNetwork(url) // foreground: null
.install();
// Force foreground mode
await FlutterEdgeAi.installModel(modelType: ModelType.gemmaIt)
.fromNetwork(url, foreground: true)
.install();
iOS uses native URLSession which handles long downloads automatically — no foreground service needed.
Cancelling downloads#
import 'package:flutter_edge_ai/flutter_edge_ai.dart'; // exports CancelToken
final cancelToken = CancelToken();
final future = FlutterEdgeAi.installModel(modelType: ModelType.gemmaIt)
.fromNetwork(url)
.withCancelToken(cancelToken)
.withProgress((progress) => print('Progress: $progress%'))
.install();
// Cancel from elsewhere (e.g. user pressed a cancel button)
cancelToken.cancel('User cancelled download');
try {
await future;
} catch (e) {
if (CancelToken.isCancel(e)) {
print('Download was cancelled by user');
}
}
CancelToken cancels all files in multi-file downloads (e.g. embedding model +
tokenizer), works on mobile + web, and throws DownloadCancelledException.
Speech models#
On-device speech via the opt-in flutter_edge_ai_speech package —
selectable, profile-driven pipelines (you pick the model with SttModelType
/
TtsModelType). moonshine-tiny, Whisper, and
Parakeet (STT) and
Matcha, Qwen3-TTS (multilingual), and Inflect-Nano-v2
(fast) TTS work
end-to-end today. TtsModelType.kokoro and TtsModelType.supertonic
have no pipeline: using either throws UnimplementedError. Native only (no
Web). See Speech for usage.
A VoiceSession also chains STT → LLM → TTS into a push-to-talk voice loop —
see Speech.
Speech-to-text
| Model | Input | Builds (size) | Documented on (CPU) | Auth |
|---|---|---|---|---|
| Moonshine Tiny | raw 16 kHz PCM | f32 109 MB, int8 52 MB | Android, iOS, macOS, Windows x64, Linux x64; f32 also Linux arm64 | ❌ |
| Whisper Tiny | log-mel | f32 151 MB, int8 41 MB | Android, iOS, macOS, Windows x64, Linux x64 | ❌ |
| Whisper Base | log-mel | int8 77 MB | Android, iOS, macOS, Windows x64, Linux x64 | ❌ |
| Parakeet CTC 0.6B | log-mel | f32 2.35 GB, int8 596 MB (phones, CPU) | f32: macOS, Windows x64, Linux x64, Android (Galaxy S23); int8: Android (Galaxy S23) | ❌ |
Sizes are the model file; the tokenizer adds 0.4–2.5 MB. The registry also
lists Whisper Base f32 (290 MB), which has not been run, and per-SoC NPU
builds, which flutter_edge_ai_speech refuses. On a phone use Parakeet int8
on the CPU: on a Galaxy S23 (Adreno 740) the f32 file did not load on the GPU
and the int8 file transcribed the same text on the CPU in a third of the
memory; see Speech
(#747).
Whisper is multilingual — the shipped checkpoints are the multilingual ones
(no .en suffix), so all 99 of Whisper's languages are available. Set a default
for the recognizer, override it per transcription, or both:
// Default for every transcription on this recognizer.
final stt = await FlutterEdgeAi.getActiveStt(language: 'de');
final german = await stt.transcribe(germanPcm);
// One call in another language — same recognizer, nothing reloaded.
final french = await stt.transcribe(frenchPcm, language: 'fr');
Both are free: the language is one token in the decoder's seed prompt, and that prompt is rebuilt on every transcription. Switching languages never reloads the model or invalidates the recognizer you are holding.
The value is Whisper's own language code without the delimiters, and it defaults
to 'en'. It decides the OUTPUT language only — the weights understand the
audio either way, so asking for 'en' on German speech returns an English
translation rather than an error.
Moonshine and Parakeet have no language token in their decoder prompt and
reject the parameter with an ArgumentError rather than ignoring it; both
are English-only.
Text-to-speech
| Model | Output | Size | Documented on (CPU) | Auth |
|---|---|---|---|---|
| Matcha-TTS | 16-bit PCM @ 22050 Hz | 94 MB | Android, iOS, macOS, Windows x64, Linux x64, Linux arm64 | ❌ |
Qwen3-TTS 0.6B
(10 languages +
auto
)
|
16-bit PCM | 1.89 GB | macOS; not run elsewhere | ❌ |
| Inflect-Nano-v2 (fast) | 16-bit PCM @ 24 kHz | 36 MB | Android, iOS, macOS, Windows x64, Linux x64 | ❌ |
Embedding models#
The LiteRT embedding models below generate 768-dimensional vectors. The
numbers in names (64/256/512/1024/2048) indicate maximum input sequence length
in tokens, not embedding dimension. An ONNX model's dimension is its own —
all-MiniLM-L6-v2 is 384-dim. See
Embeddings & RAG for usage. An input longer than
the maximum is never truncated: it throws EmbeddingInputTooLongException.
| Model | Parameters | Dimensions | Max Seq Length | Size | Auth Required |
|---|---|---|---|---|---|
EmbeddingGemma 2 text
(
.litertlm
; web from litertlm 1.13.1)
|
270M | 768D | 1024 tokens | 165MB + 71MB weight cache (native) | ❌ |
EmbeddingGemma 2 text + vision
(
.litertlm
, text and images; web from litertlm 1.13.1)
|
440M | 768D | 1024 tokens | 388MB + 71MB + 218MB weight caches (native) | ❌ |
EmbeddingGemma 2 text + vision + audio
(
.litertlm
, text, images and audio, from litertlm 1.13.1; web audio from 1.13.2)
|
740M | 768D | 1024 tokens | 485MB + 385MB weight caches (native) | ❌ |
| Gecko 64 | 110M | 768D | 64 tokens | 112MB | ❌ |
| Gecko 256 | 110M | 768D | 256 tokens | 114MB | ❌ |
| Gecko 512 | 110M | 768D | 512 tokens | 120MB | ❌ |
| EmbeddingGemma 256 | 300M | 768D | 256 tokens | 179MB | ✅ |
| EmbeddingGemma 512 | 300M | 768D | 512 tokens | 179MB | ✅ |
| EmbeddingGemma 1024 | 300M | 768D | 1024 tokens | 183MB | ✅ |
| EmbeddingGemma 2048 | 300M | 768D | 2048 tokens | 196MB | ✅ |
EmbeddingGemma 2 GGUF
(
.gguf
Q8_0, text; native, through
flutter_edge_ai_llama
)
|
text tower of google/embeddinggemma-2 | 768D | 1024 tokens | 310MB | ❌ |
| all-MiniLM-L6-v2 (ONNX; native and web) | 22M | 384D | the graph's | 90MB (onnx/model.onnx) |
❌ |
| EmbeddingGemma-300M ONNX (ONNX; native only) | 300M | 768D | the graph's | 175MB–1.2GB by variant | ❌ |
EmbeddingGemma 2 is one .litertlm file with its own tokenizer (no
tokenizerFrom…()) and runs on CPU on Android, iOS, macOS, Linux and Windows,
and in the browser on WebGPU since flutter_edge_ai_litertlm 1.13.1 (the
browser needs WebGPU: the CPU path of @litert-lm/core 0.18.0 cannot load
these bundles).
All three bundles embed text (flutter_edge_ai_litertlm 1.13.0, identical text
vectors); the 440m and the 740m also embed JPEG, PNG, GIF and BMP images
(1.13.1; per image about 0.35 s on an Apple-silicon Mac and about 2 s on an
Android flagship, and about 400–500 MB more RAM from the first image); the 740m
also embeds WAV, MP3 and FLAC audio of up to about 40 seconds, natively (1.13.1;
about 0.2–0.3 s per 3-second clip on an Apple-silicon Mac) and on the web from
1.13.2, and any of them together in one vector. Keep chunks at
512 tokens or fewer, and use the 270m when you embed only text; see
Embeddings & RAG.
The same repositories hold each bundle compiled for one SoC's NPU
(…_Qualcomm_SM8550/SM8650/SM8750/SM8850.litertlm,
…_Qualcomm_QCS8275.litertlm, 188–563 MB; run on SM8650, SM8850 and QCS8275),
which flutter_edge_ai_litertlm 1.13.1 runs on Qualcomm Android and Qualcomm
Linux arm64 with preferredBackend: PreferredBackend.npu; see
EmbeddingGemma 2 on a Qualcomm NPU.
From 1.13.2 the Intel Panther Lake bundles (…_Intel_PTL.litertlm, 317–856 MB)
run on Windows x64 the same way; see
EmbeddingGemma 2 on an Intel NPU.
From 1.15.0 the Google Tensor G5/G6 bundles (…_Google_Tensor_G6.litertlm,
166–658 MB) run on matching Pixels with google_tensor_npu: true; see
EmbeddingGemma 2 on a Google Tensor NPU.
From 1.17.0 the MediaTek MT6991/MT6993 bundles (…_MediaTek_MT6991.litertlm)
are accepted with mediatek_npu: true for phones with that SoC, but not run
on such a phone yet; see LiteRT-LM.
The Gecko and EmbeddingGemma 1 rows are a .tflite model plus a tokenizer
file; the ONNX rows are an .onnx model plus its tokenizer file, run by
OnnxEmbeddingBackend (ONNX). An ONNX model's input limit is its
graph's static sequence length, or none for a dynamic-shape graph.
Performance (Android Pixel 8):
- Gecko 64: ~109 ms/doc embedding, 130 ms search (fastest — 2.6× faster than EmbeddingGemma).
- EmbeddingGemma 256: ~286 ms/doc embedding, 342 ms search (more accurate — 300M vs 110M params).
SigLIP 2 text tower (ONNX, manual wiring)#
flutter_edge_ai_embeddings ships a SigLIP 2 text profile, for putting text
into the same space as SigLIP's vision tower — image↔text retrieval rather than
document RAG. It is the only embedding profile here that is not installed
through installEmbedder().
Text in, vectors out — flutter_edge_ai does not run the vision tower. You embed the image side elsewhere (or offline) and query it with vectors this profile produces.
Its convention differs from every model above: no leading BOS, a single trailing
<eos>, lowercased text, and a fixed 64-token width that lives in the token ids
because the int8 export carries no attention_mask. It also ignores the
TaskType prefix — the vision tower encodes an image with no prefix, so adding
one moves the text vector off the space the two towers share.
SigLIP 2 reuses the Gemma BPE vocabulary, so a tokenizer.json cannot be told
apart by its vocabulary alone, and the embedding tokenizer router in
flutter_edge_ai_embeddings (GemmaEmbeddingTokenizers) refuses
such a file rather than reading it with Gemma's convention and
returning a plausible but wrong vector.
To tell whether an export is the one this profile expects, look at two blocks of
its tokenizer.json — the same two the refusal keys on:
-
"padding"declares a fixed width —"strategy": {"Fixed": …}, not"BatchLongest" -
"post_processor"appends<eos>and prepends no<bos>
The refusal needs both, so there are three outcomes rather than two. A file
matching both is SigLIP 2's convention. One that prepends <bos>
is an
EmbeddingGemma-family file and belongs on the profiles above. A file failing
either check — no padding block, "BatchLongest", or a
post_processor
that is missing, empty, or does not end in a special token — is not classified
at all: it is read with Gemma's convention and nothing is raised.
So do not read the absence of an error as approval. A SigLIP 2 export that declares both blocks is caught, but one that dropped either — a re-export, or a tool that strips them — reaches the Gemma path and produces exactly the silently wrong vector this section warns about.
Wire it yourself:
import 'package:flutter_edge_ai_embeddings/embedding_tokenizer.dart'
show loadSiglipSentencePieceEmbeddingTokenizer;
and pass it as the tokenizerFactory of your own TokenEmbeddingPass (from
package:flutter_edge_ai/engine.dart), which your backend's describe()
returns in an EmbeddingPassDescriptor. That library is native-only. See the
flutter_edge_ai_embeddings README
for the full profile, and Embeddings & RAG
for why
the tokenizer router declines to guess.
