flutter_edge_ai's engines are pluggable: you register them in
FlutterEdgeAi.initialize(...), and the registry picks one per model by its
declared ModelFileType. flutter_edge_ai_onnx adds two of them from the
ONNX Runtime family:
OnnxEngine— text generation via ORT-GenAI.-
OnnxEmbeddingBackend— embeddings via plain ONNX Runtime (no generation).
On native platforms both are pure dart:ffi — no JVM, no gRPC — and each
drives the native library from a long-lived worker isolate, so no FFI pointer
crosses an isolate boundary. On Web there is no FFI: generation runs through
Transformers.js and embeddings through onnxruntime-web, behind the same public
API. Either arm can be registered on its own — they don't depend on each other.
Platforms#
Native support is arm64/x64-gated. Windows ARM64 generation, embeddings, and the example build were verified on a GitHub-hosted Windows 11 ARM64 runner (Azure Cobalt 100):
| Platform | Runtime | OnnxEngine / OnnxEmbeddingBackend |
|---|---|---|
| macOS (Apple Silicon) | ORT-GenAI + ORT via dart:ffi |
✅ ~54 tok/s (M4 Pro) |
| Linux x64 | ORT-GenAI + ORT via dart:ffi | ✅ ~5.3–5.8 tok/s |
| Windows x64 | ORT-GenAI + ORT via dart:ffi | ✅ ~3.3 tok/s |
| Windows ARM64 | ORT-GenAI + ORT via dart:ffi |
✅ 10.9 tok/s (Azure Cobalt 100) |
| Android (arm64) | ORT-GenAI + ORT via dart:ffi |
✅ ~10.4 tok/s (Pixel 8 Pro) |
| iOS (arm64) | ORT-GenAI + ORT via dart:ffi | ✅ device-verified |
| Web | Transformers.js + onnxruntime-web (WebGPU/WASM) | ✅ both arms |
On an unsupported native host (macOS Intel, or any other native ABI)
OnnxEnginepolitely declines and logs why, letting another registered engine take over. Android needsminSdk 24(ORT's AAR declaresminSdkVersion=24, and the bundled ORT-GenAI is built for API 24; the build fails below it). The native library co-location — ORT loaded next to ORT-GenAI — is handled for you: the build hook bundles the two libraries side by side (on iOS ORT-GenAI's framework keeps its own ORT private, and embeddings use a separateonnxruntime.framework), and on macOS and Linux the package points ORT-GenAI at the bundled ORT at run time. You don't configure anything. On iOS the package needs a 15.1 deployment target: Microsoft compiles ONNX Runtime for iOS 15.1.
Setup#
Add the package and register whichever arm(s) you use at startup:
dependencies:
flutter_edge_ai: ^2.7.0
flutter_edge_ai_onnx: ^0.10.0 # ONNX Runtime engines
flutter_edge_ai_embeddings: ^2.3.3 # tokenizers, for embeddings
import 'package:flutter_edge_ai/flutter_edge_ai.dart';
import 'package:flutter_edge_ai_embeddings/flutter_edge_ai_embeddings.dart';
import 'package:flutter_edge_ai_onnx/flutter_edge_ai_onnx.dart';
await FlutterEdgeAi.initialize(
inferenceEngines: [OnnxEngine()], // text generation
embeddingBackends: [OnnxEmbeddingBackend()], // embeddings
embeddingTokenizers: [GemmaEmbeddingTokenizers()], // flutter_edge_ai_embeddings
);
Install a model#
ONNX models install with fileType: ModelFileType.onnx. What that means differs
by platform.
Native — a directory, not a file. An ORT-GenAI model is a directory:
genai_config.json, model.onnx (plus model.onnx_data
for external weights),
and tokenizer files. OnnxEngine takes the tracked genai_config.json's
parent directory as the model directory.
fromHuggingFace(repo) installs that whole directory for you: the ONNX resolver
(carried by OnnxEngine, so registering the engine is enough) lists the repo,
picks an execution-provider folder, and downloads every file in it. A repo with
several EP variants resolves to a CPU/mobile folder — the bundled runtime is
CPU-only — unless you pin one with OnnxHuggingFaceResolver(variant: …):
await FlutterEdgeAi.installModel(
// ONNX repos declare no family — the caller's modelType is used as-is.
modelType: ModelType.general,
fileType: ModelFileType.onnx,
).fromHuggingFace('microsoft/Phi-3.5-mini-instruct-onnx').install();
Or ship the bundle yourself and point fromFile at its genai_config.json:
await FlutterEdgeAi.installModel(
modelType: ModelType.general,
fileType: ModelFileType.onnx,
).fromFile('/path/to/my-model/genai_config.json').install();
Web — fileless. The model identity is a Hugging Face repo id; Transformers.js resolves, fetches, and caches it the first time you run inference. Install just marks the repo id active — core never downloads model bytes:
await FlutterEdgeAi.installModel(
modelType: ModelType.general,
fileType: ModelFileType.onnx,
).fromNetwork('https://huggingface.co/onnx-community/Qwen2.5-0.5B-Instruct').install();
From here the code is identical to any other engine:
final model = await FlutterEdgeAi.getActiveModel(maxTokens: 4096);
final session = await model.openSession();
await session.addQueryChunk(const Message(text: 'Hello!', isUser: true));
final response = await session.getResponse();
await session.close();
Generation — OnnxEngine#
Text-only (v1): no vision, no audio, no LoRA. openSession / openChat open independent sessions on the
same loaded model, natively since 0.8.0 and on Web since 0.9.0. On Web each
keeps its own history on the one Transformers.js pipeline, holds no KV cache
and is not capped unless you set maxConcurrentSessions.
Natively each session has its own generator and so its own context; turns of different sessions run one at a time and nothing is
replayed. A generator allocates its whole KV cache at its first turn,
maxTokens × the cache size per token (2 × layers × KV heads × head size ×
dtype size; 0.75 MiB for Phi-3.5-mini int4, on Linux CPU 771 MiB at 1024, 3077
at 4096), so opening more than maxConcurrentSessions throws a StateError;
unset, that is 2 on Android and iOS and unlimited on desktop. Pass a larger
maxConcurrentSessions to allow more on a phone. Windows gives a closed
session's memory back (commit −383 MiB at 512, −768 MiB at 1024); on Linux the RSS did not
fall at 512 to 2048. Sampling
follows the model's own config (genai_config.json natively,
generation_config.json on Web), which is greedy unless it sets do_sample;
the temperature, topK and topP you pass override it, and
temperature: 0
or topK: 1 keeps decoding greedy. Natively, a temperature or topP
without
topK over a config with top_k: 1 also gets a top-k (the family's, else 40),
so it samples. On Web the seed is not applied yet and topP is ignored
(Transformers.js samples with top-k only); each is said once when set. Prompts use the model's own chat template
(ORT-GenAI's OgaTokenizerApplyChatTemplate natively; the model's
chat_template on Web) — the engine never builds turn markers itself.
On native, preferredBackend is not applied at all: neither ORT client appends an
execution provider. For inference, ORT-GenAI picks the provider from the model
directory's genai_config.json, whose execution-provider section
flutter_edge_ai_onnx never reads (it reads only search.top_k), so the inference
model's activeBackend is null rather than a guess. Embeddings run on plain
ORT with no provider appended, which is CPU, and the embedder reports cpu.
PreferredBackend.cpu pins WASM on Web; anything else tries WebGPU first and
falls back to WASM.
Structured JSON output#
This section requires core 2.6.0 and flutter_edge_ai_onnx 0.9.0 (Windows
x64: 0.10.0).
Native ORT-GenAI can apply llguidance to each independent generator opened by
openSession(responseFormat: ...) or openChat(responseFormat: ...). It is
enabled on macOS arm64 (0.9.0) and Windows x64 (0.10.0), after each host's run
passed, and only when the model directory contains tokenizer.json. Linux
x64, Windows ARM64, Android, iOS and ONNX Web return a reason from
structuredOutputSupport and throw ResponseFormatUnsupportedException
if a
format is requested; the Linux x64 and Windows ARM64 archives contain guidance
but stay disabled until their own runs pass.
final format = ResponseFormat.jsonSchema({
'type': 'object',
'properties': {'answer': {'type': 'string'}},
'required': ['answer'],
'additionalProperties': false,
});
final refusal = model.refusalFor(format);
if (refusal != null) throw refusal;
final session = await model.openSession(responseFormat: format);
try {
// Use the structured session.
} finally {
await session.close();
}
ONNX advertises the whole core dialect and delivers structured text
incrementally. Core validates the completed reply. A turn cut during generation
therefore emits any generated chunks and then throws
IncompleteStructuredOutputException; a normally completed non-matching value
throws NonConformingStructuredOutputException. A prompt that does not fit the
remaining context fails with ContextLimitException and is not kept.
Embeddings — OnnxEmbeddingBackend#
A plain ONNX Runtime forward pass over an .onnx/.ort embedding model. The
tokenizer provider you register (GemmaEmbeddingTokenizers from
flutter_edge_ai_embeddings) handles both tokenizer families, and the
output
dimension is model-dependent (not a fixed 768):
- WordPiece / BERT-style models (e.g. all-MiniLM-L6-v2, 384-dim) — mean-pooled + normalized client-side.
-
SentencePiece models (e.g. EmbeddingGemma-300M-ONNX) — the model's own
pooled
sentence_embeddingoutput. Native only — on Web, embeddings are WordPiece/BERT-style only in this release (a pure-Dart SentencePiece parser is pending).
A SigLIP 2 tokenizer.json is a third convention the provider does not
select yet, and it is BPE — so it would otherwise be read as Gemma and return a
plausible but wrong vector. The provider refuses it instead, with a message
pointing at loadSiglipSentencePieceEmbeddingTokenizer, which you can wire into
your own TokenEmbeddingPass (from package:flutter_edge_ai/engine.dart) — see
SigLIP 2 text tower
for
the profile and how to recognise a matching export.
The output contract and mask requirements are discovered from the session's
actual graph once it opens — no per-model configuration. It registers at
priority 10 (above LiteRtEmbeddingBackend's catch-all priority 0), so with
both registered an .onnx/.ort model (any letter case) routes here. See
Embeddings & RAG for the shared embedding API.
ONNX embeddings are text only (getActiveEmbedder(supportImage: true)
throws EmbeddingLoadException). maxInputTokens is the graph's static
sequence length, or null for a dynamic-shape graph; a static graph is padded,
never truncated, so a longer input throws EmbeddingInputTooLongException.
The pass is OnnxEmbeddingPass (OnnxEmbeddingForwardPass before 0.5.8).
Web setup#
Web needs a small web/index.html shim before FlutterEdgeAi.initialize() runs
(the same readiness-handshake pattern the other web arms use). Add the shim for
whichever arm(s) you register, in <head>, ahead of flutter_bootstrap.js:
<!-- Transformers.js v4 — OnnxEngine web generation. -->
<script type="module">
window.transformersReady = (async () => {
const m = await import('https://cdn.jsdelivr.net/npm/@huggingface/transformers@4.3.1');
window.transformers = m;
return m;
})();
</script>
<!-- onnxruntime-web — OnnxEmbeddingBackend web embeddings. -->
<script type="module">
window.ortReady = (async () => {
const m = await import('https://cdn.jsdelivr.net/npm/onnxruntime-web@1.30.0/dist/ort.bundle.min.mjs');
m.env.wasm.wasmPaths = 'https://cdn.jsdelivr.net/npm/onnxruntime-web@1.30.0/dist/';
window.ort = m;
return m;
})();
</script>
Dart awaits window.transformersReady / window.ortReady before touching
either module, so the shim must run before the Flutter app boots.
See also#
- Embeddings & RAG — the shared embedding API this backend plugs into.
- Models — the full supported-model matrix.
-
Packages — every opt-in engine and backend, including
flutter_edge_ai_onnx.
Writing this with a coding assistant? dart run skills@ get --all installs
flutter-edge-ai-onnx, the skill that teaches it ORT-GenAI generation and ONNX embeddings, native and through Transformers.js.
