flutter_edge_ai is on-device AI for Flutter on Android, iOS, Web, macOS, Windows and Linux. Run any open model a registered engine runs — see Models — or the model the operating system already ships, with text, vision, audio, function calling, embeddings, RAG and speech. No server, no cloud.
The core is small: it installs models and dispatches to engines, and you add only the engines, stores and features your app uses — see Packages.
Features#
- Local Execution: Run any model a registered engine runs (see Models) directly on user devices for enhanced privacy and offline functionality.
- Platform Support: Compatible with iOS, Android, Web, macOS, Windows, and Linux.
-
Desktop Support: Native desktop apps with GPU acceleration via LiteRT-LM, called directly from Dart through
dart:ffi— no JVM/JRE bundling. See Desktop Support. - Built-in / System AI: Use the OS's own on-device model, no download — Gemini Nano (Android + Chrome), Apple Foundation Models (iOS/macOS) and Windows AI Foundry. See Built-in AI.
-
Pluggable Engines: Opt-in engine packages — LiteRT-LM, MediaPipe, ONNX Runtime, llama.cpp (GGUF), built-in OS AI — registered via
FlutterEdgeAi.initialize(...). See Packages. - Multimodal Support: Text + image input with vision models (the Vision column on Models), including Gemini Nano on Android through built-in AI. See Multimodal.
- Audio Input: Record and send audio messages with Gemma 4 and Gemma3n models (Android, iOS device, Desktop).
- Function Calling: Let models call external functions and integrate with other services. See Function Calling.
-
Agent Skills: Give the model a catalog of
SKILL.mdskills it picks and runs itself — text, JS, native intents, or MCP tools. See Agent Skills. -
Speech & Voice (STT + TTS + voice loop): Transcribe audio, synthesize speech, and run a full on-device speech-to-speech voice loop (
VoiceSession) viaflutter_edge_ai_speech— STT (moonshine/Whisper/Parakeet) + TTS (Matcha/Qwen3/Inflect) today. See Speech. - Thinking Mode: View the reasoning process of thinking models — see Thinking Mode.
- Stop Generation: Cancel text generation mid-process on Android, iOS, Web, and Desktop.
-
Backend Switching: Choose CPU or GPU where supported, plus NPU for
.litertlmon Qualcomm Android devices, Qualcomm Linux arm64 boards, Google Tensor G5/G6 Pixels and MediaTek Android devices (opt-in, see LiteRT-LM) and on Intel LunarLake/PantherLake Windows PCs. Which builds have run where: the Model Registry. -
LoRA (MediaPipe
.task): Efficient fine-tuning and integration of LoRA (Low-Rank Adaptation) weights. - Enhanced Downloads: Smart retry logic with exponential backoff and automatic restart of interrupted downloads.
-
Android Foreground Service: opt in with
foreground: truefor large downloads to bypass the 9-minute timeout. -
Embeddings (text, images, audio) & RAG: Generate vectors with pluggable embedding runtimes,
then use independent
flutter_edge_ai_ragindexes with Qdrant or SQLite storage. See Embeddings & RAG. - Web Persistent Caching: Models persist across browser restarts using the Cache API (Web only).
What's new in 2.7#
-
Saving a conversation (
flutter_edge_ai2.7.0):MessageTranscriptCodecwrites a list ofMessages as versioned JSON and reads it back. See Saving a conversation. -
Hugging Face resolvers per role (
flutter_edge_ai2.7.0): a resolver declares the role it serves, andinstallEmbedder().fromHuggingFaceinstalls an embedder. See Migration. -
Web downloads (
flutter_edge_ai2.7.0): downloads stream into the Cache API, and--wasmbuilds install models over 1 GiB. -
Text and hybrid search (
flutter_edge_ai_rag1.2.0,flutter_edge_ai_sqlite2.2.0):searchText,searchHybrid(RRF) andVectorStoreSpec.enableTextSearch, with SQLite FTS5 on every platform. See Text and hybrid search. -
litertlm_dart(flutter_edge_ai_litertlm1.21.0): the LiteRT-LM runtime and its native build hook moved to this package, so NPU flags go underhooks: user_defines: litertlm_dart:. On Android, Gemma 4 on the GPU stays at FP16 above 2,048 tokens, and a GPU turn that fails names the CPU (see Troubleshooting). See Migration. -
ONNX response formats on Windows x64 (
flutter_edge_ai_onnx0.10.0, on ORT-GenAI 0.20.0). See ONNX. -
llama.cpp (
flutter_edge_ai_llama0.2.0): llamadart^0.20.1with Apple companion 0.20.1. See llama.cpp.
What's new in 2.6#
-
Typed JSON replies (
flutter_edge_ai2.6.0):openSessionandopenChatacceptresponseFormat; engines advertise exact support and core validates every completed reply. See Structured JSON replies. -
Native LiteRT-LM conversations (
flutter_edge_ai_litertlm1.20.0): verified native rows keep one conversation and KV cache per open session. -
Engine-owned turn recovery: failed and refused turns roll back consistently,
and
ContextLimitExceptiondistinguishes an input that never fit. -
Open APIs are canonical:
createSession,createChat,sessionandchatremain compatible but are deprecated until core 3.0.0. - Add your app to the showcase — submit a shipped Flutter Edge AI app for review.
What's new in 2.5#
-
Several open models (
flutter_edge_ai2.5.0,flutter_edge_ai_litertlm1.16.0):FlutterEdgeAi.openModel(spec)builds a caller-owned model on every call, so one file can be open on CPU and GPU, or two files at once. See Desktop. -
Deciders from a language model (experimental,
flutter_edge_ai2.5.0 withflutter_edge_ai_litertlm1.16.0, native,PreferredBackend.cpu; the GPU too from 1.19.1, not the NPU):InferenceModel.createDeciderscores the options on a.litertlmtrained on the decision with LiteTune. See Decisions. -
MediaTek NPU (
flutter_edge_ai_litertlm1.17.0):PreferredBackend.npuon MediaTek Android devices, opt-in withmediatek_npu: true. See LiteRT-LM. -
Intel's Gemma 4 12B on the Windows NPU (
flutter_edge_ai_litertlm1.17.0): OpenVINO 2026.4.1, and the next NPU load deletes the%TEMP%weight banks a killed 1.17.0 app left (older ones after a restart). See Desktop. -
GGUF embeddings (
flutter_edge_ai_llama0.0.3):LlamaEmbeddingBackendruns EmbeddingGemma 2.gguftext embeddings natively. See llama.cpp. -
Several ONNX sessions (
flutter_edge_ai_onnx0.8.0): nativeopenSession/openChatopen independent sessions, each with its own generator; at most 2 on Android and iOS unlessmaxConcurrentSessionssays otherwise. ORT-GenAI 0.17.0. See ONNX. -
Windows memory diagnostics (
flutter_edge_ai_diagnostics0.6.0):memorySnapshot()on Windows. See Diagnostics. -
llama.cpp on llamadart 0.17.5 (
flutter_edge_ai_llama0.0.6; superseded by 0.2.0 on^0.20.1), with Apple companion 0.17.3. See llama.cpp.
What's new in 2.4#
-
Decider (experimental,
flutter_edge_ai2.4.0):EmbeddingModel.createDeciderpicks one of N options for an input — an intent, an action, a route, a tool — or says "none of these". See Decisions. -
Order of images, audio and text (
flutter_edge_ai_litertlm1.14.0): Gemma 4 gets its audio after the text, as its model card asks, andgetActiveModel(multimodalOrder:)sets another order. See Multimodal. -
Structured tool calls (
flutter_edge_ai2.4.0):Message.toolCall(calls:)keeps the calls with their ids,Message.toolResponse(toolCallId:)names the call a result answers, and a sentence Gemma 4 writes before a call no longer streams the call's JSON as text. See Function Calling. -
Backends from the model file (
flutter_edge_ai_litertlm1.14.0): the engine reads a.litertlmbefore it loads it — a bundle compiled for an NPU runs there, each encoder runs where its section says (activeVisionBackend,activeAudioBackend), and a file that cannot run as asked is refused with the reason.readLiteRtLmBundleInfo(path)shows what it reads, andLiteRtLmEngine(backendFallback: false)makes one attempt that fails with LiteRT-LM's own reason. See LiteRT-LM. -
requireWeightCache (
flutter_edge_ai_litertlm1.14.0):LiteRtEmbeddingBackend(requireWeightCache: true)fails an EmbeddingGemma 2 load that wrote no weight cache instead of running it slower and larger. See Embeddings & RAG. -
Agent skills 0.3.0 (
flutter_edge_ai_agent): an empty answer right afterloadSkillis asked again (EmptyAnswerRetryEvent, an event an exhaustiveswitchmust now handle), intents the app adds getvalidators:, and SKILL.md parsing is fixed. See Agent Skills. -
Genkit 0.10.0 (
genkit_flutter_edge_ai): toolrefs travel as call ids both ways; it requiresflutter_edge_ai2.4.0. See Genkit. -
2.4.1 /
flutter_edge_ai_litertlm1.15.0: Google Tensor NPUs on Pixels (run on G6, G5 not run), opt-in withgoogle_tensor_npu: true; a stopped turn is rewound in place, so the chat keeps earlier images and audio; the Intel NPU on Windows loads its model again; a raw Gemma 4 tool call runs with typed arguments or not at all. See LiteRT-LM. -
Agent skills 0.4.0 (
flutter_edge_ai_agent, breaking):schedule_notificationruns only withNativeIntentExecutor(notifications:). See Migration. -
llama.cpp engine (
flutter_edge_ai_llama0.0.1): GGUF models, on pub.dev. See llama.cpp.
Upgrading from 2.3: see Migration.
What's new in 2.3#
-
2.3.2: the active model keeps its LoRA across an app restart, and
ModelFileType.ggufselects the optional llama.cpp engine,flutter_edge_ai_llama, on pub.dev since 0.0.1. -
EmbeddingGemma 2 on an Intel NPU (
flutter_edge_ai_litertlm1.13.2): the Panther Lake bundles load on Windows x64 withpreferredBackend: PreferredBackend.npu(not run on an Intel NPU at that release; the 270m has since run on a Panther Lake PC). See Embeddings & RAG. -
Audio turns (
flutter_edge_ai_speech0.5.7,flutter_edge_ai_agent0.2.10):VoiceSession.runAudioTurngives the model the recording itself, with no speech-to-text, andAgentSession.ask(audioBytes:)asks an agent with a recording. See Speech and Agent Skills. -
flutter_edge_ai_diagnostics0.3.0: macOS support andanonymousPeakBytes, the lifetime footprint peak on iOS and macOS. See Diagnostics. -
EmbeddingGemma 2 (
flutter_edge_ai_litertlm1.13.0): text embeddings from one.litertlmbundle with its own tokenizer, on Android, iOS, macOS, Linux and Windows; up to 1024 tokens per input. See Embeddings & RAG. -
EmbeddingGemma 2 audio and combined inputs (
flutter_edge_ai_litertlm1.13.1,flutter_edge_ai2.3.1): the 740m bundle embeds WAV, MP3 and FLAC clips on native platforms, andEmbeddingInput(text:, images:, audio:)puts several parts in one vector; an over-long input reports its exact token count. See Embeddings & RAG. -
EmbeddingGemma 2 on the Web (
flutter_edge_ai_litertlm1.13.1): text and images in the browser through@litert-lm/core, on WebGPU (the browser needs it), and from 1.13.2 audio and combined inputs; addwindow.EmbeddingEngine = m.EmbeddingEngine;toweb/index.html. See Installation → Web. -
EmbeddingGemma 2 images (
flutter_edge_ai_litertlm1.13.1): the 440m bundle embeds JPEG, PNG, GIF and BMP photos next to text, for text-to-image and image-to-image search in a RAG image index. See Embeddings & RAG. -
A new embedding API (breaking):
embed(EmbeddingInput.text(t), taskType: …)andembedAllreplacegenerateEmbedding(s),dimensionreplacesgetDimension(),TaskTypehas nine values, errors are typedEmbeddingExceptions delivered as failed futures, and an over-long input throwsEmbeddingInputTooLongExceptioninstead of being truncated. -
RAG 1.1 (breaking):
RagIndex.add/searchtake anEmbeddingInput,open()takes the embedder andopenVectors()a profile, and a location binds one document modality. Needsflutter_edge_ai_sqlite/flutter_edge_ai_qdrant2.1.0. -
Engine packages import
package:flutter_edge_ai/engine.dart; core's internals moved underlib/src, and the deprecated 1.x/2.0 members are gone. Upgrade every flutter_edge_ai package together — see Migration.
What's new in 2.2#
-
Sampling defaults come from the model (breaking):
temperature,topK,topPandrandomSeedare nullable, and an unset one comes from the sampler the model file ships, else its family's published defaults, instead of greedytopK: 1. PasstopK: 1to keep greedy decoding. See Migration. -
2.2.1 — installs and model lifecycle: a
fromFilemodel is active again after a restart, and core never deletes a file you installed withfromFile. Installing a name again from another source replaces it (see Installing again). With nothing installed,getActiveModelandgetActiveEmbedderthrowNoActiveModelError, aStateError.InferenceModel.isClosedis new, and web reuses the loaded model as the other platforms do.
What's new in 2.1#
-
enableThinkingreplacesisThinkingonopenChatandopenSession(breaking);ModelRuntimeDefaults.isThinkingisthinkingDeclared. NewModelType.qwen35. See Migration. -
Qualcomm NPU is opt-in (
flutter_edge_ai_litertlm1.10.0): setqualcomm_npu: trueunderhooks: user_defines: litertlm_dart:(flutter_edge_ai_litertlm:before 1.21.0), and the build hook fetches Qualcomm's QNN runtime. The same flag now covers Qualcomm Linux arm64 boards. See LiteRT-LM. -
LiteRT-LM v0.18.0 (
native-v0.18.0-d). Apple's Metal accelerator ships asLiteRtLmMetalAccelerator, so an app can useflutter_litertbeside it, and native tool calls work on Linux.
What's new in 2.0#
-
RAG is an independent module.
flutter_edge_ai_ragownsRagIndex, embedding profiles, filters, and provider selection; SQLite and Qdrant are replaceable storage packages. It can borrow core's active embedder, use a custom embedder, or run vector-only withoutFlutterEdgeAi.initialize(). - Persistent indexes are profile-safe. A location is bound to a stable embedding-pipeline identity, and app-owned indexes have explicit disposal. See Embeddings & RAG and Migration.
-
The
flutter_gemmaaliases are gone. The old Dart names no longer compile;dart fix --applystill renames them. -
ModelFileManager.setActiveModelis removed — useensureModelReadyFromSpec.
Historical release notes (1.x): Flutter Gemma became Flutter Edge AI#
-
flutter_gemma is now Flutter Edge AI. Every package has a new name —
flutter_edge_ai,flutter_edge_ai_litertlm, … Installed models, vector stores and platform setup carry over, anddart fix --applyrenames the old Dart names. See Migration.
Core through 1.11.4 (flutter_gemma_litertlm through 1.8.6) shipped under the flutter_gemma*
names; the new names continue the same numbering (the table is in Migration).
What's new in 1.9#
-
embeddingTokenizers:— embedding backends no longer bring a tokenizer. Addflutter_edge_ai_embeddingsand passembeddingTokenizers: [GemmaEmbeddingTokenizers()]besideembeddingBackends:, or the first embedding throws aStateError(since 2.3,getActiveEmbedder()throws anEmbeddingLoadExceptionwith that cause). Breaking for every app that computes embeddings. See Migration. -
Engines no longer pull the tokenizer package —
flutter_edge_ai_litertlmandflutter_edge_ai_onnxno longer depend onflutter_edge_ai_embeddings. Speech intentionally still depends on LiteRT-LM to share its native bundle. See Packages. -
Web embeddings run — the LiteRT.js bundle was rebuilt on
@litertjs/core2.5.3 and all four JS files now ship together influtter_edge_ai_litertlm/web/. See Embeddings & RAG. -
SQLite RAG on web drains for real —
flutter_gemma_rag_sqlite1.4.0 requires sqlite3 3.6.0 and Flutter 3.47; a Flutter 3.44 app stays on 1.3.2. See Embeddings & RAG.
What's new in 1.8#
-
Package Skills — flutter_edge_ai ships agent skills for coding assistants;
dart run skills@ get --allinstalls them. See Package Skills. -
Whisper output language —
getActiveStt(language:)sets a default,transcribe(…, language:)overrides one call. Breaking for customSpeechRecognizerimplementations. See Speech. -
Vector-store
flush()was added to persist an index. In 2.0 the current call isRagIndex.flush(); see Embeddings & RAG.
What's new in 1.7#
-
Install from Hugging Face —
installModel(…).fromHuggingFace(repo)reads the repo's manifest and installs the right variant for the device;resolveHuggingFaceinspects it first. See Models → Hugging Face.
What's new in 1.6#
-
flutter_edge_ai_onnx— new opt-in ONNX Runtime engine: text generation via ORT-GenAI (OnnxEngine) + embeddings via plain ONNX Runtime (OnnxEmbeddingBackend), bothdart:ffi. Device-verified on macOS, Linux, Windows, Android, and iOS (arm64) — plus Web, via Transformers.js (generation) and onnxruntime-web (embeddings). See Packages. -
flutter_edge_ai_builtin_aion Web — Gemini Nano through Chrome's Prompt API, next to Gemini Nano on Android and Apple Foundation Models on iOS/macOS. See Built-in AI. -
BREAKING (
flutter_gemma_embeddings2.0.0): the embedder is now runtime-agnostic —LiteRtEmbeddingBackendmoved toflutter_gemma_litertlm(1.5.0). See Migration.
What's new in 1.5#
-
genai_primitives support — drive an on-device chat with the Flutter team's standard
ChatMessagetypes viapackage:flutter_edge_ai/genai.dart(sendMessage/generateContent+ streams, covering text, vision, audio, thinking, and tool calls). See genai_primitives. -
Historical RAG facade: 1.5 put RAG under
FlutterEdgeAi.rag; 2.0 removed that facade in favor of independently ownedFlutterEdgeAiRagindexes. The core facade still owns model introspection, storage helpers, and per-modality uninstallers.
What's new in 1.4#
-
flutter_edge_ai_speech— new opt-in on-device speech package: STT (moonshine) + TTS (Matcha) + a full STT → LLM → TTS voice loop (VoiceSession). See Speech.
What's new in 1.3#
-
ModelFileType.builtInfor OS system models (Gemini Nano / Apple Foundation Models), plus download-reliability fixes. See Built-in AI.
What's new in 1.2#
-
flutter_edge_ai_agent— new opt-in agent-skills package (theskillExecutors:registration seam), plus the Android Mali GPU fix. See Agent Skills.
What's new in 1.1#
-
Declared-column RAG filters were introduced through core initialization.
In 2.0 the schema lives in
VectorStoreSpec; see Embeddings & RAG.
What's new in 1.0#
-
Modular package split — the monolith is now a small core (
flutter_edge_ai) plus opt-in packages, so your app ships only the native weight it uses:flutter_edge_ai_litertlm(.litertlm),flutter_edge_ai_mediapipe(.task/.bin),flutter_edge_ai_embeddings,flutter_edge_ai_qdrant,flutter_edge_ai_sqlite. See Packages. -
New
FlutterEdgeAi.initialize(...)registration originally included AI runtimes and the vector store. In 2.0 it registers AI runtimes only. -
Model/session/chat/embedding APIs stayed compatible in 1.0; RAG later moved
to
flutter_edge_ai_ragin 2.0. See Migration. -
Two on-device vector stores —
flutter_edge_ai_qdrant(qdrant-edge, fastest on native) andflutter_edge_ai_sqlite(portable across all six platforms, including Web). Since rag_sqlite 1.1.0 the SQLite store runs exact in-SQLite KNN via thesqlite-vec/vec0extension, replacing its Dart brute-force + HNSW search.
See the CHANGELOG for the full release history.
Quick Start#
Complete the platform setup before running this code.
1. Install a Model (One Time)#
import 'package:flutter_edge_ai/flutter_edge_ai.dart';
import 'package:flutter_edge_ai_litertlm/flutter_edge_ai_litertlm.dart';
// Once, in main(): core has no engine of its own. Gemma 3 1B is gated, so
// pass a token — never hard-code it.
await FlutterEdgeAi.initialize(
inferenceEngines: const [LiteRtLmEngine()],
huggingFaceToken: const String.fromEnvironment('HUGGINGFACE_TOKEN').isEmpty
? null
: const String.fromEnvironment('HUGGINGFACE_TOKEN'),
);
// Install model. The URL below uses the .litertlm variant, which runs on
// Android, iOS and Desktop (Windows/macOS/Linux). Web uses web builds — see
// the note below.
await FlutterEdgeAi.installModel(
modelType: ModelType.gemmaIt,
fileType: ModelFileType.litertlm,
).fromNetwork(
'https://huggingface.co/litert-community/Gemma3-1B-IT/resolve/main/Gemma3-1B-IT_multi-prefill-seq_q4_ekv4096.litertlm',
).withProgress((progress) {
print('Downloading: $progress%');
}).install();
MediaPipe alternative: Google's LLM Inference API is maintenance-only, so
new apps should prefer .litertlm. On mobile you can install the .task build of the
same model instead — drop fileType (it defaults to ModelFileType.task) and
register MediaPipeEngine() from flutter_edge_ai_mediapipe. On iOS, run the
.task build on PreferredBackend.cpu: on the GPU it is killed for memory
while loading (details). On Web, use the
-web.task build (gemma3-1b-it-int4-web.task) with MediaPipeEngine().
Desktop targets need the .litertlm build — .task and .bin are
MediaPipe-only.
2. Create and Use a Model (Multiple Times)#
// Create model with specific configuration
final model = await FlutterEdgeAi.getActiveModel(
maxTokens: 2048,
preferredBackend: PreferredBackend.gpu,
);
// Use model
final chat = await model.openChat();
await chat.addQueryChunk(Message.text(
text: 'Explain quantum computing',
isUser: true,
));
final response = await chat.generateChatResponse();
// Cleanup
await model.close();
Message.isUser defaults to false. Always pass isUser: true for user
messages, or the model returns an empty response. Always close() sessions and
models when you're done with them.
Sampling#
Leave temperature, topK and topP unset and each session gets the sampler
the model file ships, else the defaults its family publishes
(SamplingParams.forModelType; Gemma: temperature: 1.0, topK: 64, topP: 0.95), else
temperature: 0.8, topK: 40, topP: 0.95. A value you pass
wins, field by field. Pass topK: 1 or temperature: 0 for greedy decoding.
-
Where the model file's sampler is read: a
.litertlmbundle on native platforms, and an ONNX model'sgenai_config.json. Web.litertlmand MediaPipe.taskuse the family defaults, and MediaPipe holds an unset top-k to 40, its own default. -
A temperature or top-p without a top-k asks for sampling, so a greedy
family (Phi, Hammer) or model config contributes no
topK: 1: the session gets the next layer's top-k instead. Built-in AI is the exception: it keepstopK: 1, and says so once. -
.litertlmkeeps the sampler of an engine's first generation for every later conversation (LiteRT-LM #2080). To change it, close the model and load it again; a warning says so once when you set either sampler. A new chat on the same model still gets a new answer to the same prompt, and a freshly loaded model repeats its first answer for the same seed. -
What an engine ignores is printed once: built-in AI takes no seed and
applies top-p only on Windows, MediaPipe web takes no top-p, ONNX web
applies neither the seed nor top-p, and the NPU decodes greedily whatever
you set.
randomSeedis 1 when unset, unless a.litertlmbundle ships one; ONNX leaves it to the model config. Built-in AI sendstemperature: 0.8, topK: 1for unset values, as it always has.
System Instructions#
Control model behavior with a system-level instruction:
final chat = await model.openChat(
systemInstruction: 'You are a concise assistant. Always respond in bullet points.',
);
Platform support:
-
.litertlmon every platform, native and web: passed as a real system turn. - ONNX (native and web): passed as a
systemmessage. - Built-in AI: handed to the OS model as its instructions.
-
MediaPipe
.task(Android, iOS, web): prepended to the first user message as a fallback.
Changing a runtime parameter reloads the model#
getActiveModel caches one model per active spec, but it compares every runtime
parameter — maxTokens, preferredBackend, supportImage,
maxConcurrentSessions and the rest. A call that differs in any of them
closes the cached model and builds a new one, logging which parameter forced
the rebuild:
final quick = await FlutterEdgeAi.getActiveModel(maxTokens: 512);
final deep = await FlutterEdgeAi.getActiveModel(maxTokens: 4096);
// deep is a NEW model, and `quick` has been closed — using it now throws a
// StateError ('Model is closed…'). Drop the old handle.
So don't hold a handle across a parameter change, and don't vary the arguments on a hot path: each change costs a full weight reload. To serve several conversations from one loaded model, keep the arguments identical and open several sessions instead (below). Closing it yourself first is equivalent:
await quick.close();
final deep = await FlutterEdgeAi.getActiveModel(maxTokens: 4096);
An app that needs several models loaded at once, such as one per backend,
opens them with FlutterEdgeAi.openModel instead: each call builds a model
the app owns and closes, which a getActiveModel rebuild never touches. See
Desktop → Several models at once.
To serve several dialogues from one loaded model, do not reload it — open several sessions instead (next section).
Web follows the same rule since flutter_edge_ai 2.2.1. Before that, every
getActiveModel call on web closed the cached model and built a new one,
identical arguments or not.
A model is closed from the moment its close() starts: model.isClosed is
true. Since 2.2.1 a getActiveModel call made while that close is still
running waits for it to finish (at most 60 s, then TimeoutException) and
builds a new model; it never returns the closing one, and two models are never
loaded at once.
On native .litertlm (except the NPU) a maxTokens below 1024 is clamped up to
1024, so the 512 above would not have been honored in any case. See
Troubleshooting.
Structured JSON replies#
Use responseFormat only with openSession or openChat; it is fixed for that
session and every normally completed reply is checked.
ResponseFormat.jsonObject() accepts any JSON object.
ResponseFormat.jsonSchema(...) supports type, properties,
required,
additionalProperties: false, items, enum, const,
minItems, maxItems,
minLength, maxLength, minimum, maximum, plus the annotation-only
description, title and examples. The root must declare one scalar
type;
unions, $ref, $defs, composition keywords, pattern
and format are outside
this dialect.
final format = ResponseFormat.jsonSchema({
'type': 'object',
'properties': {'summary': {'type': 'string', 'maxLength': 200}},
'required': ['summary'],
'additionalProperties': false,
});
final refusal = model.refusalFor(format);
if (refusal != null) throw refusal;
final chat = await model.openChat(responseFormat: format);
try {
// Use the structured chat.
} finally {
await chat.close();
}
Read model.structuredOutputSupport after loading: it reports the keywords this
exact engine/platform/backend enforces and StructuredDelivery.incremental
or
.buffered. ResponseFormatUnsupportedException means the format was refused.
If generation ends cleanly but early with TurnEnd.outputLimit, contextLimit,
stopped or closed, IncompleteStructuredOutputException
carries that end
and the partial text; a prompt that never fits fails with
ContextLimitException instead. NonConformingStructuredOutputException
means
an engine accepted a format but ended normally with text that was not exactly
one matching JSON value. Incremental text is emitted before the final validation
error. Do not combine responseFormat with tools or enableThinking: true;
core throws ArgumentError before opening the engine session.
Concurrent Sessions (openSession)#
A single loaded model can serve several independent dialogues at once.
openSession() returns a session with its own conversation history;
openChat() is the same for the higher-level chat API. Native ONNX has one ORT-GenAI generator per session,
capped by maxConcurrentSessions; on the web each ONNX session keeps its own
history on the one Transformers.js pipeline. MediaPipe on the web supports it
too: its sessions share the one loaded model, take turns on it, and share the
sampling and LoRA of the session that loaded it.
The model weights (the big, expensive part — hundreds of MB to several GB) are
loaded once and shared across every session, but a session is not always
lightweight. Native .litertlm sessions have one conversation and KV cache
each on macOS CPU/Metal, Linux and Windows CPU, Android Adreno/Mali, Qualcomm or
Google Tensor NPUs on Android, and Intel NPUs on Windows. Linux Vulkan, Windows
D3D12, iOS, Android CPU/PowerVR/MediaTek, and unverified Linux arm64
accelerators instead use virtual sessions that replay full messages, including
media, into one shared conversation. The web runtime is also virtual but
remains text-only.
Hybrid models such as LFM2 rebuild from full messages when recovery is needed
and never rewind.
final model = await FlutterEdgeAi.getActiveModel(maxTokens: 1024);
final chatA = await model.openChat(); // independent context A
final chatB = await model.openChat(); // independent context B
await chatA.addQueryChunk(Message(text: 'My name is Alice.', isUser: true));
await chatA.generateChatResponse();
await chatB.addQueryChunk(Message(text: 'My name is Bob.', isUser: true));
await chatB.generateChatResponse();
// Each remembers only its own context.
await chatA.addQueryChunk(Message(text: 'What is my name?', isUser: true));
print(await chatA.generateChatResponse()); // "Alice"
model.sessions; // all live sessions
await chatA.close(); // closing one leaves the others usable
Concurrent contexts, serialized inference. The sessions are logically
independent, but only one session generates at a time — calling
getResponse() or generateChatResponse() on a second session while another is still running blocks
until the first finishes. Generation is not parallel. This is intentional:
parallel on-device inference would contend for the accelerator and risk OOM.
Memory: what an open session costs depends on the engine. A native
.litertlm session holds its own KV cache: Gemma 4 E2B on an M5 Pro added
36.8 MiB of RSS per idle session at maxTokens: 4096, from the third session
onward (the first two reused freed memory). A virtual .litertlm
session keeps full message history instead and pays replay time when it becomes
active. A MediaPipe .task session on Android or iOS holds its own KV cache (~100–500 MB depending
on model + maxTokens), so on phones with large models (Gemma 4 E2B+) several
concurrent sessions can OOM. GGUF sessions share one runtime with no separate
native KV cache.
An ONNX session on the web keeps only its history, resent on every turn.
.litertlm on the web and built-in AI are not measured. Cap the count with
maxConcurrentSessions: on getActiveModel(...) — openSession()
throws
StateError past the cap. A MediaPipe web session keeps only its history, in
Dart, and resends it every turn; one session's turn waits for the other
sessions' turns, however long they run, unless they are stopped.
For LiteRT-LM, MediaPipe, ONNX and llama.cpp, the legacy createSession() /
createChat() lane does not consume the detached-session cap. Built-in AI
instead applies the cap to every live underlying flutter_local_ai session,
including its legacy singleton lane.
createSession() and createChat() are deprecated in 2.6.0 and will be
removed in 3.0.0. Do not use the legacy lane in new code.
If you only ever have one conversation at a time, open one session or chat, keep its handle for that conversation, and close it when done.
Turn outcomes and chat history#
For a core-managed chat turn, a successful exchange stays in
chat.fullHistory. A clean stopped turn also stays after it produced non-empty
text; if it reaches the chat as stopped before the first non-empty text chunk,
its staged input is taken back. Cancelling the response subscription directly
does not itself apply that empty-stop rollback, so do not use cancellation as a
history-reset operation.
Any engine error on a core-managed turn takes the whole staged turn back, including a tool call emitted before the error. A hook-backed engine owns its transcript and balancing instead; the shipped llama.cpp hook rolls failed turns back, but its current plain stopped or cancelled turn is also rolled back even after text. Structured llama.cpp turns use core's guarded path and keep their partial reply.
ContextLimitException means the engine refused the input because it did not
fit the remaining context; nothing from that turn was committed. A clean
TurnEnd.contextLimit is different: decoding filled the window, so the partial
exchange stays. Reporting engines also use endOfSequence (natural end),
outputLimit (maxOutputTokens), stopped, and
closed; an engine that cannot
distinguish ends may expose no EngineTurn.end.
A reported close keeps partial text, then a plain core-managed chat throws
StateError. A structured session reports every non-endOfSequence
end with
IncompleteStructuredOutputException, including outputLimit, a clean
contextLimit, stopped, and closed.
Saving a conversation#
MessageTranscriptCodec writes a list of messages, such as chat.fullHistory,
as versioned JSON and reads it back; a decoded message is == to the one saved
when its tool-call arguments are JSON values. It saves the messages only, not
the chat's system instruction, tools or response format: open the chat with
your own configuration and replay the messages. The model's context is rebuilt
from them, not restored.
import 'dart:convert';
const codec = MessageTranscriptCodec();
final json = jsonEncode(await codec.encode(chat.fullHistory));
// later, on a chat opened with the same system instruction and tools:
final messages = await codec.decode(jsonDecode(json));
await chat.clearHistory(replayHistory: messages);
Replay what fullHistory returned, not a display list: systemInfo and
thinking messages are not model input, and the llama engine refuses
systemInfo. Numbers come back as JSON numbers: compiled to JavaScript,
1.0
reads as 1, and integers past 2^53 lose precision. encode
throws
ArgumentError for a tool-call argument JSON cannot hold (a BigInt, a
DateTime, NaN, a non-string key, a cycle).
Media is written inline as base64, which makes the JSON about a third larger
than the media. Pass writeMedia to store each buffer yourself and return a
reference, and readMedia to resolve it on decode. The writer gets the media
type and the field's path, such as messages[3].images[0]; the path changes
when a message is deleted, so build the reference from a content hash of the
bytes such as SHA-256 (not bytes.hashCode, which hashes the object), or the
path plus an id new for each save. Decoding is strict: another format version,
a missing or an extra key is a FormatException.
Managing Installed Models#
Installing again#
Installing a file name again from the source it came from does nothing. From
another source — another URL (query and token aside), or fromFile with
another path — it replaces the install, and core deletes its own earlier copy
once the new one is active. Core never deletes or overwrites a file you
installed with fromFile: it is used in place, it is active again after a
restart, and a download that would land on it throws ModelStorageException.
Uninstalling#
For the active embedder / STT / TTS model, one call deletes every file it
owns (e.g. an embedder's model and tokenizer) and clears its persisted
identity so it isn't auto-restored on the next launch. Files you installed with
fromFile are yours: they are uninstalled but stay where they are.
await FlutterEdgeAi.uninstallEmbedder(); // model + tokenizer, clears identity
await FlutterEdgeAi.uninstallStt();
await FlutterEdgeAi.uninstallTts();
Each first closes the model getActiveEmbedder(), getActiveStt() or
getActiveTts() built from those files, including the one your app holds: a
request already running finishes, and later calls on it throw. The next
getActive*() call throws until a model is installed again, and then builds a
new one.
To delete an inference model, or any model by filename, use uninstallModel().
When the file belongs to the active inference model (the model or its LoRA), it
also clears the active inference identity: hasActiveModel() is false and
getActiveModel() throws NoActiveModelError instead of handing out the model
of a deleted file. A directory model (ORT-GenAI, installed with
fromHuggingFace) is uninstalled by its name, the spec.name
its install
returned: every file of it and its directory go.
It first closes the model getActiveModel() built from that file, with its
sessions and chats: a generation running on it stops early, with or without an
error depending on the engine, and later calls on it throw. Models you opened
with openModel() stay open; close them yourself:
// Closes the model getActiveModel() built from this file, deletes the
// downloaded file + metadata (a fromFile file stays) and, if this was the
// active model, clears the active inference identity.
await FlutterEdgeAi.uninstallModel('Gemma3-1B-IT_multi-prefill-seq_q4_ekv4096.litertlm');
clearActiveInferenceIdentity() wipes both the in-memory active spec and the
persisted preference without deleting any file. uninstallModel() now does it
for you when it removes a file of the active model; calling it afterwards, as
earlier releases required, is still harmless. The uninstallEmbedder/Stt/Tts
helpers clear their identity too.
Inspecting what's active#
Cheap, synchronous getters return the active model's identity (its spec)
without loading the engine — use these to render UI, not getActiveModel()
(which loads the runtime). Each is typed to its modality:
final InferenceModelSpec? model = FlutterEdgeAi.activeModelSpec;
final EmbeddingModelSpec? embedder = FlutterEdgeAi.activeEmbedderSpec;
final SttModelSpec? stt = FlutterEdgeAi.activeSttSpec;
final TtsModelSpec? tts = FlutterEdgeAi.activeTtsSpec;
// Where an installed file is read from on the device: your own file for a
// fromFile install, the app bundle for fromBundled on iOS, else core's copy.
// On web it returns the file name itself, not a path or a URL.
final path = await FlutterEdgeAi.getModelPath(
'Gemma3-1B-IT_multi-prefill-seq_q4_ekv4096.litertlm',
);
Storage & cleanup#
Inspect on-device usage and reclaim space left by interrupted downloads:
final info = await FlutterEdgeAi.getStorageInfo(); // StorageStats: files + bytes
final orphans = await FlutterEdgeAi.getOrphanedFiles(); // fragments with no metadata
final removed = await FlutterEdgeAi.cleanupStorage(); // delete orphans, returns count
await FlutterEdgeAi.performCleanup(); // cancel stale tasks + sweep
Message Types#
// Text only
final textMessage = Message.text(text: "Hello!", isUser: true);
// Text + Image
final multimodalMessage = Message.withImages(
text: "What's in this image?",
imageBytes: [imageBytes],
isUser: true,
);
// Image only
final imageMessage = Message.imagesOnly(imageBytes: [imageBytes], isUser: true);
// Tool response (for function calling)
final toolMessage = Message.toolResponse(
toolName: 'change_background_color',
response: {'status': 'success', 'color': 'blue'},
);
// System information message
final systemMessage = Message.systemInfo(text: "Function completed successfully");
// Thinking content (DeepSeek, Gemma 4, Qwen3, …)
final thinkingMessage = Message.thinking(text: "Let me analyze this problem...");
// Check if a message contains an image
if (message.hasImage) {
print('This message contains an image');
}
Response Types#
The model can return different types of responses depending on its capabilities:
chat.generateChatResponseAsync().listen((response) {
if (response is TextResponse) {
// Regular text token from the model
print('Text token: ${response.token}');
} else if (response is FunctionCallResponse) {
// Model wants to call a function
print('Function: ${response.name}');
print('Arguments: ${response.args}');
_handleFunctionCall(response);
} else if (response is ParallelFunctionCallResponse) {
// Several calls in one reply
for (final call in response.calls) {
_handleFunctionCall(call);
}
} else if (response is ThinkingResponse) {
// Model's reasoning process
print('Thinking: ${response.content}');
_showThinkingBubble(response.content);
}
});
-
TextResponse— contains a text token (response.token) for regular model output. -
FunctionCallResponse— contains function name (response.name) and arguments (response.args). See Function Calling. -
ParallelFunctionCallResponse— several calls in one reply (response.calls, each aFunctionCallResponse). -
ThinkingResponse— contains the model's reasoning process (response.content). See Thinking Mode.
Next Steps#
- Installation — per-platform setup and engine registration.
- Models — supported models, file formats, and capabilities.
-
Migration — upgrade from
flutter_gemmaor Flutter Edge AI 1.x.
Writing this with a coding assistant? dart run skills@ get --all installs
flutter-edge-ai-inference, the skill that teaches it engines, installing a model, sessions and chats, streaming, and the platform setup for all six targets.
