LogoFlutter Edge AI

Desktop Support

Setup and reference for running flutter_edge_ai on macOS, Windows, and Linux via dart:ffi.

Detailed setup and reference for running flutter_edge_ai on macOS, Windows, and Linux. Desktop platforms run LiteRT-LM directly via dart:ffi — no Java, no separate process, no IPC.

LiteRT-LM (.litertlm) is the primary desktop engine (core registers no engine, so register LiteRtLmEngine()), but not the only one. flutter_edge_ai_onnx (ONNX Runtime — ORT-GenAI generation + ORT embeddings) also runs on desktop (macOS arm64, Linux x64, Windows x64 and Windows ARM64), flutter_edge_ai_llama (llama.cpp, .gguf) runs on all three desktops, and the OS built-in model is available through flutter_edge_ai_builtin_ai (Built-in AI) on macOS (Apple Foundation Models) and, since 0.3.0, on Windows (AI Foundry / Phi Silica — nothing to configure to build: flutter_local_ai resolves the Windows App SDK projection itself; running needs Windows 11 25H2+ on Copilot+-class hardware and a packaged app). Linux has no OS built-in model. What holds across all of desktop is the narrower statement: there is no MediaPipe engine on desktop. See Installation and Packages.

Architecture#

┌─────────────────────────────────────────────────────┐
│              Flutter Desktop App                     │
│                                                      │
│   ┌──────────────────────────────────────────────┐ │
│   │  FlutterEdgeAiDesktop (lib/src/desktop/)       │ │
│   │           ↓                                    │ │
│   │  LiteRtLmFfiClient                            │ │
│   │  (litertlm_dart/lib/src/ffi/)                   │ │
│   │           ↓ dart:ffi                           │ │
│   │  ───────────────────────────────────           │ │
│   │  libLiteRtLm.{dylib,dll,so}                    │ │
│   │  + libLiteRt.{dll,so} (Linux/Windows)          │ │
│   │  + LiteRtLmMetalAccelerator.framework (macOS)  │ │
│   │  + libLiteRtWebGpuAccelerator.{dll,so}         │ │
│   │  + libwebgpu_dawn.{dll,so} (Linux/Windows GPU) │ │
│   │  + dxil.dll + dxcompiler.dll (Windows GPU)     │ │
│   └──────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────┘

Native libraries are fetched at build time by the package's hook/build.dart from the GitHub release, SHA256-verified, and bundled by Flutter Native Assets into the application bundle. The Dart FFI layer is shared with mobile — Android and iOS use the same LiteRtLmFfiClient against the same C API. Only the dynamic library loading sequence differs per platform.

Model format (LiteRT-LM engine): the LiteRT-LM engine on desktop accepts only .litertlm files. MediaPipe .bin / .task models used on web won't load on desktop. A generic .litertlm build runs on every desktop. A build compiled for an NPU runs only on its vendor's platform: Intel on Windows x64, Qualcomm on Linux arm64 with qualcomm_npu. MediaTek builds are refused on every desktop (they run on the Android NPU only), and the -web.litertlm exports are for the web. The Model Registry shows where each build runs. (The ONNX engine uses its own .onnx model directories instead.)

Supported platforms#

Platform Architecture GPU backend Vision Audio Notes
macOS arm64 (Apple Silicon) Metal ✅ ✅ Vision verified on Gemma 4 + Gemma 3n (text decoder on Metal; the vision encoder runs where the bundle says: CPU on Gemma 4, GPU on Gemma 3n)
macOS x86_64 — — — Not supported (Apple Silicon only)
Windows x86_64 DirectX 12 (via Dawn/WebGPU) ✅ ✅ Nothing to install
Windowsarm64———Not supported
Linux x86_64 Vulkan (via Dawn/WebGPU) ✅ ✅ glibc ≥ 2.34, libstdc++ ≥ 6.0.30 (Ubuntu 22.04+, Debian 12+, RHEL 10+; RHEL 9 ships GCC 11's older libstdc++)
Linux arm64 Vulkan (via Dawn/WebGPU) ✅ ✅ Same glibc and libstdc++ requirement

Qualcomm Linux arm64 boards also run PreferredBackend.npu on the Hexagon DSP when the app opts in with qualcomm_npu: true — see LiteRT-LM.

Windows NPU. PreferredBackend.npu on Windows requires Intel LunarLake/PantherLake silicon — the Windows native archive ships LiteRtDispatch.dll + the OpenVino runtime + TBB to drive it. Other Windows hardware has no NPU to run on: use PreferredBackend.gpu or .cpu. The load does not check for one — a bundle compiled for the Intel NPU is not refused on a PC without it, and what LiteRT does then is not measured. LLMs on the Intel NPU need flutter_edge_ai_litertlm 1.15.0: every release from flutter_gemma_litertlm 1.7.0 (native-v0.17.0) through 1.14.0 failed the load or ended the process with 0xC0000409 (LiteRT-LM#4023).

Intel's Gemma 4 12B for Panther Lake needs flutter_edge_ai_litertlm 1.17.0: its NPU blobs carry metadata 2.7, which the OpenVINO 2026.3 of earlier releases refuses. 1.17.0 ships OpenVINO 2026.4.1. Measured with Intel NPU driver 32.0.100.4778 on both PCs.

A bundle whose NPU blobs carry no weights makes the dispatch copy them to %TEMP%\litert_ov_bank_<hex>.bin at every load: 5.55 GB for Intel's 12B, 74.5 MB for EmbeddingGemma 2 270m _Intel_PTL. Keep that much free on the system drive. The dispatch deletes the file when the model is closed, so an app that exits without closing the model leaves it behind. From 1.17.0 each app marks its banks with a <bank>.<pid>.owner file it keeps open while the model is loaded. The next NPU load deletes the banks whose owner is gone, and the unmarked ones written before the PC last started, and prints how much it freed. Windows' Fast Startup shutdown does not count as a start. An unmarked bank written since the start is kept, because an app on an older release or another LiteRT program may still be using it.

Run Gemma 4 there. On the Intel NPU a Gemma 3 bundle loses every prefill chunk after the first — the reply is fluent, answers from the opening of your prompt and never mentions the rest, with no error raised (LiteRT-LM#3508).

EmbeddingGemma 2's Panther Lake bundles (…_Intel_PTL.litertlm) load on the same stack from flutter_edge_ai_litertlm 1.13.2. The 270m is enabled in code and was measured on a Panther Lake PC, 2026-10-21; see EmbeddingGemma 2 on an Intel NPU.

Requirements#

  • Flutter ≥ 3.44.0
  • macOS: Apple Silicon (arm64); 14.0 with flutter_edge_ai_litertlm (and so flutter_edge_ai_speech) or flutter_edge_ai_llama, 12.0 with flutter_edge_ai_builtin_ai
  • Windows: 10/11 64-bit. LiteRT-LM needs no Visual C++ Redistributable (see below); flutter_edge_ai_llama's Windows runtime comes from llamadart; whether its current archive carries the Visual C++ runtime is not checked (see llama.cpp).
  • Linux: glibc ≥ 2.34, libstdc++ ≥ 6.0.30 (Ubuntu 22.04+, Debian 12+, Fedora 36+, RHEL 10+ — RHEL 9 ships GCC 11's libstdc++, which is too old)
  • GPU drivers: any vendor driver with WebGPU/Vulkan/Metal/DX12 support; falls back to CPU if not available (LiteRtLmEngine(backendFallback: false) turns the fallback off)

No Java/JVM/JRE required.

Quick Start#

import 'package:flutter_edge_ai/flutter_edge_ai.dart';
import 'package:flutter_edge_ai_litertlm/flutter_edge_ai_litertlm.dart';

Future<void> chat() async {
  // Once, in main(): core has no engine of its own.
  await FlutterEdgeAi.initialize(inferenceEngines: const [LiteRtLmEngine()]);

  // Install model (downloads on first run, cached after). Gemma 4 from
  // litert-community is public — no token needed.
  await FlutterEdgeAi.installModel(
    modelType: ModelType.gemma4,
    fileType: ModelFileType.litertlm,
  ).fromNetwork(
    'https://huggingface.co/litert-community/gemma-4-E2B-it-litert-lm/resolve/main/gemma-4-E2B-it.litertlm',
  ).install();

  // Create model with full capabilities — keep it for the app's lifetime.
  final model = await FlutterEdgeAi.getActiveModel(
    maxTokens: 4096,
    preferredBackend: PreferredBackend.gpu,
    supportImage: true,
    supportAudio: true,
  );

  // Each chat / conversation is a session. Sessions are cheap to create
  // and destroy; the engine is reused across them.
  // Sampling left unset: the model's own sampler, else its family's defaults.
  final session = await model.openSession();
  await session.addQueryChunk(Message(text: 'Hi!', isUser: true));
  await for (final chunk in session.getResponseAsync()) {
    print(chunk);
  }
  await session.close();
}

For the high-level chat API with history + thinking + tool calling, use model.openChat(...) and chat.generateChatResponseAsync().

Platform-specific setup#

macOS#

macOS 14 or later. LiteRT-LM's companion libraries are built for macOS 14, so the app's deployment target must be 14.0: in Xcode, Runner target → General → Minimum Deployments (MACOSX_DEPLOYMENT_TARGET = 14.0 in macos/Runner.xcodeproj/project.pbxproj), and platform :osx, '14.0' in macos/Podfile if the app has one. Below it the build stops with The package product 'flutter-edge-ai-litertlm-product' requires minimum platform version 14.0, or CocoaPods' required a higher minimum deployment target.

Native libs are fetched and bundled via Native Assets, except two companion libraries: GemmaModelConstraintProvider, which LiteRtLm links, and the Metal accelerator LiteRtLmMetalAccelerator. Native Assets cannot relocate them (#247), so they reach the app one of two ways:

  • Swift Package Manager on (Flutter's default since 3.44): nothing to set up. flutter_edge_ai_litertlm's Swift package brings them from the native release, and Xcode embeds them and signs them with the app's identity. An app that also has a macos/Podfile (another plugin uses CocoaPods) needs no block in it.
  • Swift Package Manager off (enable-swift-package-manager: false under flutter: config: in pubspec.yaml): a post_install block in macos/Podfile copies them. Replace that Podfile's post_install block with the one below, keeping any lines of your own inside it; the next build runs pod install itself.

Without them the first model load fails with LiteRT-LM did not load on macOS, naming the missing library.

post_install do |installer|
  installer.pods_project.targets.each do |target|
    flutter_additional_macos_build_settings(target)
  end

  # flutter_gemma: stage the upstream Apple companion dylibs into the built
  # .app. `hook/build.dart` deliberately skips them from Native Assets on macOS
  # (#247 — Google ships them without `-Wl,-headerpad_max_install_names`, so the
  # JIT bundling path cannot rewrite their install_name), which leaves this
  # build phase to stage them.
  #
  # The phase only LOCATES and RUNS a script; the staging logic itself lives in
  # flutter_edge_ai_litertlm and is delivered next to the dylibs it stages. That
  # is deliberate: this block is frozen into your Xcode project, and a copy of
  # the logic frozen there cannot be fixed by upgrading the package.
  installer.aggregate_targets.each do |aggregate_target|
    aggregate_target.user_targets.each do |user_target|
      phase_name = '[flutter_gemma] Setup LiteRT-LM macOS'

      # Only the app target embeds the Frameworks/ this phase patches.
      # RunnerTests inherits Runner's framework search paths and has no
      # Contents/Frameworks of its own — having the phase there creates a
      # cross-target dependency on Runner's framework output that Xcode reports
      # as "Cycle inside Flutter Assemble" (#300). Remove any stale copy from
      # non-app targets and skip them.
      unless user_target.name == 'Runner'
        user_target.build_phases
          .select { |p| p.respond_to?(:name) && p.name == phase_name }
          .each { |p| user_target.build_phases.delete(p) }
        next
      end

      existing = user_target.shell_script_build_phases.find { |p| p.name == phase_name }
      phase = existing || user_target.new_shell_script_build_phase(phase_name)
      # The embedded LiteRtLm binary is an INPUT so the phase re-runs whenever
      # Flutter's always-out-of-date `embed` phase re-copies the raw, unpatched
      # binary over the patched one. Without it Xcode caches the phase after the
      # first build and the second incremental build ships an unpatched
      # LiteRtLm that fails dlopen at runtime (#368).
      phase.input_paths = [
        '$(BUILT_PRODUCTS_DIR)/$(PRODUCT_NAME).app/Contents/Frameworks/LiteRtLm.framework/Versions/A/LiteRtLm',
      ]
      # A declared output lets Xcode order the phase in its dependency graph
      # instead of treating it as "runs every build with no outputs" — the other
      # half of the cycle warning (#300). The script touches this file.
      phase.output_paths = ['$(DERIVED_FILE_DIR)/flutter_gemma_litertlm_macos.stamp']
      phase.shell_script = <<~SHELL
        set -e
        STAGER="${HOME}/Library/Caches/flutter_gemma/native/macos_arm64/stage_macos_companions.sh"
        if [ ! -f "${STAGER}" ]; then
          echo "[flutter_gemma] ERROR: ${STAGER} not found." >&2
          echo "  flutter_edge_ai_litertlm installs it there from its build hook." >&2
          echo "  Upgrade the package, then: flutter clean && flutter pub get" >&2
          exit 1
        fi
        sh "${STAGER}" "${BUILT_PRODUCTS_DIR}/${PRODUCT_NAME}.app/Contents/Frameworks"
        mkdir -p "$(dirname "${SCRIPT_OUTPUT_FILE_0}")"
        touch "${SCRIPT_OUTPUT_FILE_0}"
      SHELL
    end
  end
end

Flutter 3.44 with Xcode 27. With Swift Package Manager off, the first build stops with The macOS deployment target 'MACOSX_DEPLOYMENT_TARGET' is set to 10.15, but the range of supported deployment target versions is 12.0 to 27.0.x on the Pods targets, FlutterMacOS among them. This is Flutter 3.44.0 on Xcode 27, not this package: an app with only shared_preferences and Swift Package Manager off fails the same way. Add one line inside the installer.pods_project.targets.each loop of the block above, after flutter_additional_macos_build_settings(target):

target.build_configurations.each { |c| c.build_settings['MACOSX_DEPLOYMENT_TARGET'] = '14.0' }

Flutter 3.47.8 builds the same Podfile without it (measured on Xcode 27.1, 2026-10-28; see also Troubleshooting).

From Flutter 3.50 a build with Swift Package Manager off prints Enabling Swift Package Manager will be required in a future version of Flutter.

Turning Swift Package Manager back on. An app that turned it off for an earlier flutter_edge_ai_litertlm can delete the enable-swift-package-manager: false line. It builds and loads LiteRT-LM with the Podfile still there; Flutter then prints the steps that remove CocoaPods (pod deintegrate, the two Pods-Runner includes). The [flutter_gemma] Setup LiteRT-LM macOS build phase stays in the Xcode project after them. It leaves the Swift package's frameworks alone; delete it from the Runner target's Build Phases.

Entitlements — add to both macos/Runner/DebugProfile.entitlements and Release.entitlements:

<key>com.apple.security.network.client</key>
<true/>
<key>com.apple.security.app-sandbox</key>
<true/>

network.client lets the sandboxed app download a model.

With Swift Package Manager off, also add:

<key>com.apple.security.cs.disable-library-validation</key>
<true/>

It takes effect once Hardened Runtime is on, which notarization requires: the Podfile block signs the companion libraries ad hoc, and library validation refuses code not signed by Apple or by the app's own team. With Swift Package Manager on, Xcode signs them with the app's team, and a team-signed build with Hardened Runtime loads them without it.

Do not add the iOS com.apple.developer.kernel.* memory entitlements — they are iOS-only. Without a signing team the build fails with "Runner" has entitlements that require signing with a development certificate, and a team-signed build drops them. A .litertlm model loads on macOS without them.

Windows#

flutter_edge_ai_litertlm bundles every required DLL — no manual setup. The bundle includes:

  • LiteRtLm.dll, LiteRt.dll, libGemmaModelConstraintProvider.dll, StreamProxy.dll
  • libLiteRtWebGpuAccelerator.dll
  • webgpu_dawn.dll (Dawn WebGPU backend — split into a shared lib in LiteRT-LM v0.14.0; the accelerator DLL imports it, so GPU fails without it)
  • dxil.dll + dxcompiler.dll (DirectX Shader Compiler runtime — required for WebGPU/DX12 shader compilation; from microsoft/DirectXShaderCompiler v1.9.2602)
  • LiteRtDispatch.dll, the OpenVINO runtime (openvino*.dll) and TBB (tbb*.dll) — the Intel NPU dispatch behind PreferredBackend.npu on Lunar Lake / Panther Lake

Most companion DLLs ship under two names (LiteRt.dll and libLiteRt.dll, and so on): Native Assets drops the lib prefix on Windows, while LiteRtLm.dll imports them with it.

StreamProxy.dll exposes a LoadLibraryExA(LOAD_WITH_ALTERED_SEARCH_PATH) helper that litertlm_dart uses to pre-load LiteRt.dll and libLiteRtWebGpuAccelerator.dll, then LiteRtLm.dll itself. Without this, modern Windows DLL search order doesn't always include the application directory for secondary LoadLibrary calls — they would fail to find the GPU accelerator DLL and silently fall back to CPU.

End-users need nothing installed. LiteRtLm.dll is linked against the static CRT and imports no C++ runtime at all; measured on native-v0.19.0-a, 15 of its 23 DLLs import none.

The other eight are the Intel NPU stack behind PreferredBackend.npu: our own LiteRtDispatch.dll, which links OpenVINO's C++ API and keeps the dynamic CRT, plus three of Intel's six openvino* (the NPU compiler, its loader and the NPU VM runtime import none) and its four tbb*. Each imports some of msvcp140, vcruntime140 and vcruntime140_1 — the runtimes any Flutter Windows app already resolves — and none imports vcruntime140_threads.dll, the Visual Studio 2022 17.8 one that used to make this page ask for a redistributable and that failed a Microsoft Store certification VM (#456). Nothing statically imports LiteRtDispatch.dll either, and litert_dispatch_lib_dir is set only for that backend, so an app that never asks for the Intel NPU never loads them at all.

Linux#

The bundle includes:

  • libLiteRtLm.so, libLiteRt.so, libGemmaModelConstraintProvider.so
  • libLiteRtWebGpuAccelerator.so, libStreamProxy.so
  • libwebgpu_dawn.so (Dawn WebGPU backend — split into a shared lib in LiteRT-LM v0.14.0; the accelerator loads it via $ORIGIN rpath, so GPU fails without it)

libStreamProxy.so exposes stream_proxy_load_global (an RTLD_GLOBAL dlopen). flutter_edge_ai_litertlm uses it to pre-load libLiteRt.so before libLiteRtLm.so so the WebGPU accelerator's runtime dlsym(RTLD_DEFAULT, "LiteRt*") resolves — without RTLD_GLOBAL, Dart's default RTLD_LOCAL would hide the symbols.

Build dependencies:

sudo apt install clang cmake ninja-build libgtk-3-dev lld

Linux GPU uses Dawn/WebGPU on top of Vulkan, so you need a working vendor Vulkan driver. On NVIDIA install the proprietary driver; on Intel/AMD the open-source Mesa driver works on most distros.

sudo apt install vulkan-tools libvulkan1
# Plus your vendor driver, e.g. NVIDIA:
sudo apt install nvidia-driver-535-server

Mesa's llvmpipe software fallback caps maxStorageBufferRange at 128 MB, which is below Gemma 4's per-buffer requirement — Gemma 4 will not run on llvmpipe. Install a vendor driver before running on GPU. For headless / server-side use, Xvfb is enough as a fake display surrogate.

Model lifecycle#

One model, many sessions#

The recommended pattern for one model:

// At app startup, ONCE:
final model = await FlutterEdgeAi.getActiveModel(
  maxTokens: 4096,
  preferredBackend: PreferredBackend.gpu,
  supportImage: true,
  supportAudio: true,
);

// During app runtime, MANY TIMES:
final session = await model.openSession(...);
// ... chat, generate, etc.
await session.close();   // cheap

// At app shutdown:
await model.close();

Sessions do not reload the model weights, so they are cheap to create/destroy compared with engine_create (2–10 s depending on backend and model size), which happens once when the model is first opened. Native sessions still have the per-session memory cost below.

For independent conversations, use openSession or openChat. LiteRT-LM gives each session its own native conversation and KV cache on macOS CPU/Metal and on Linux/Windows CPU. Linux Vulkan and Windows D3D12 use virtual sessions instead: they replay full messages into one shared conversation when sessions switch. Hybrid models such as LFM2 rebuild from full messages when recovery is needed and never rewind. Native sessions consume memory while idle: Gemma 4 E2B on an M5 Pro added 36.8 MiB of RSS per idle session at maxTokens: 4096, from the third session onward. Bound the count with maxConcurrentSessions on getActiveModel or openModel.

Several models at once#

getActiveModel gives the one active model: cached, and rebuilt (the previous one closed) when a runtime parameter or the active model changes. To hold several models at once, such as one file on CPU and on GPU, or two files, use FlutterEdgeAi.openModel (flutter_edge_ai 2.5.0) with an engine that supports it: LiteRtLmEngine (flutter_edge_ai_litertlm 1.16.0) on Android, iOS and desktop.

final spec = (await FlutterEdgeAi.installModel(
  modelType: ModelType.gemma4,
  fileType: ModelFileType.litertlm,
).fromFile(path).install(activate: false)).spec;

final cpu = await FlutterEdgeAi.openModel(spec, preferredBackend: PreferredBackend.cpu);
final gpu = await FlutterEdgeAi.openModel(spec, preferredBackend: PreferredBackend.gpu);
// ... generate on either ...
await gpu.close();
await cpu.close();
  • Each open model holds its own weights and KV cache, and the caller closes it: the plugin has no memory policy. On iOS a process past its memory limit is killed, with no exception to catch; read FlutterEdgeAiDiagnostics.memorySnapshot() (flutter_edge_ai_diagnostics) before opening another model.
  • The models share the engine cache directory getActiveModel uses (the cache files of one model file serve every engine on it), the native libraries the process loaded, and LiteRT-LM's log level. Engine creations run one at a time; generation on open models does not wait for them.
  • An opened model is never cached: a getActiveModel rebuild leaves it open, and closing it leaves the active model alone. Uninstalling a model does not close the models opened on it.
  • install(activate: false) registers the file without making it the active model, and openModel takes the spec it returns. A model is its file name without the extension, so two files with the same name are one model; an activate: false install that would replace the active model's file throws a StateError.
  • Measured on macOS (Apple silicon): CPU and GPU engines on one file, two GPU engines, and both created at the same time on an empty cache. On Windows Lunar Lake, CPU, GPU and NPU engines ran together in the Intel NPU demo (LiteRT-LM clients on native-v0.18.0-f). Not measured: Android, iOS, Linux, and two GPU engines on Windows.

To swap the active model, call model.close() first, then getActiveModel(...) again. Switching its backend (CPU ↔ GPU) works the same way.

Known limitations#

Windows discrete GPU (historical)#

flutter_gemma_litertlm 1.2.0–1.3.1 crashed on Windows discrete GPUs: the Windows build passed a Bazel define upstream had removed, so the LiteRt runtime was linked statically and conflicted with the separately shipped WebGPU accelerator. Fixed in 1.4.0, before the rename — every flutter_edge_ai_litertlm release has the fix.

Per-token sampler: CPU on Linux and Windows#

When preferredBackend: PreferredBackend.gpu, the forward pass (prefill + decode) runs on the GPU accelerator (Metal, DX12, Vulkan). Where the per-token sampler (top-k / top-p / argmax) runs depends on the platform; on CPU it costs roughly 1–5 ms per token, small next to the forward pass. An unset vision or audio encoder backend follows the bundle — the section's backend_constraint, else the NPU it is compiled for, else CPU (the GPU delegate can't prepare the vision encoder's ops); set preferredVisionBackend: / preferredAudioBackend: on getActiveModel(...) only where the section allows it ((await readLiteRtLmBundleInfo(path)).audioEncoder?.backendConstraint).

  • Linux and Windows — CPU. From native-v0.19.0-a no GPU sampler ships there (our build of the fused WebGPU sampler crashed the first GPU decode), so LiteRT-LM samples on the CPU while the logits stay on the GPU.
  • macOS — LiteRT-LM's samplers are linked into libLiteRtLm (from native-v0.19.0-a).

randomSeed / temperature / topK / topP — only the first session's values apply#

Only the first generation on an engine sets the sampler. Every later session on that same engine keeps those values, whatever it asks for. Upstream defect (LiteRT-LM #2080, open), reproducing on v0.14.0, v0.15.0 and v0.16.0 on CPU as well as GPU, and on v0.17.0 (checked on CPU) and v0.18.0 (macOS).

So the common shape is: an app loads a model, runs one generation with the defaults (the model's own sampler, else its family's), and from then on the engine keeps them. A later temperature: 1.5 changes nothing, and a later topK: 1 will not give you argmax. Since flutter_edge_ai_litertlm 1.12.0 a warning is printed, once per loaded model, when a later session's values differ and either set was chosen by the caller. The seed is not re-applied either — the sampler keeps its RNG state across sessions, so two identical requests on one engine produce different text.

Workaround: close and recreate the engine when you need different sampler settings — model.close() then FlutterEdgeAi.getActiveModel(...) — at the cost of a model reload. If your app uses one fixed configuration throughout, as most chat apps do, this never surfaces: the first session already set the values you wanted.

Before flutter_gemma_litertlm 1.2.0 we carried a build-time patch that fixed this downstream (offered upstream as PR #2081). v0.14.0 added a native session-config sampler API and the patch was dropped, but the underlying baking was never fixed.

Audio modality requires LiteRT-LM models#

Audio input only works with .litertlm models that include the audio adapter (Gemma 3n E2B/E4B, Gemma 4 E2B/E4B). See Multimodal.

iOS Simulator: GPU disabled#

iOS Simulator's Metal has a 256 MB single-allocation cap that LLM weight tensors exceed. Use CPU on the simulator, or test on a physical iPhone for GPU validation.

Troubleshooting#

Engine create fails with no native log on Linux#

In debug builds flutter_edge_ai_litertlm redirects native stderr to <tmpdir>/litertlm_native.log and dumps it via debugPrint after a failed engine_create. Stdout stays where it was, so print keeps reaching flutter run (from litertlm 1.17.0; earlier debug builds on macOS and Linux sent it to the same file after the first model load). In release builds stderr goes to the systemd journal / app's own stderr.

glibc 2.38 not found on Linux#

The loaded libLiteRtLm.so was built against a newer glibc than your system has. The released Linux bundle needs glibc 2.34, so this is a stale copy in the build cache. Clear it and let hook/build.dart re-fetch the release:

rm -rf ~/.cache/litertlm_dart/native
flutter clean && flutter run

Working on the package itself? A local native/litert_lm/prebuilt/linux_x86_64/ takes precedence over the release without saying so — delete that too.

Windows GPU shaders fail to compile#

Symptom: engine_create returns null with no Dart-side error, app silently falls back to CPU. Verify dxcompiler.dll and dxil.dll are next to your app.exe (Native Assets bundles them). Neither imports the Visual C++ runtime, so a missing redistributable is not the cause — look at the GPU driver instead.

macOS: generation or embedding slows down while the app is in the background#

App Nap throttles a backgrounded macOS app after about a minute, and on-device models then run several times slower (measured: an EmbeddingGemma 2 image went from 0.35 s to 2–7 s, with memory flat). An app that embeds or generates while it is in the background should hold an activity for the duration of that work, from its macOS runner:

let activity = ProcessInfo.processInfo.beginActivity(
  options: .userInitiated, reason: "Indexing photos")
// … the work …
ProcessInfo.processInfo.endActivity(activity)

An NSAppSleepDisabled key in Info.plist did not turn App Nap off in our measurements; the activity does. The example app holds one for its whole lifetime, because its integration tests run in the background.

Model file not found#

On desktop the model is downloaded to a flutter_gemma/ folder in the platform's standard "app support" directory on macOS and Linux, and in %LOCALAPPDATA% on Windows (see Troubleshooting → desktop storage). Use FlutterEdgeAi.installModel(...).fromNetwork(...).install() to download, or .fromFile(absolutePath) if you already have it locally.

Pre-cached engine + new code = stale cache#

LiteRT-LM caches compiled GPU shaders in the app's support directory (what getApplicationSupportDirectory() returns — not the flutter_gemma/ folder the model sits in), as <model>.litertlm_<mtime>_<size>_mldrift_program_cache.bin. The name is keyed on the model file's timestamp and size, so a new model build gets a fresh cache by itself (the old file stays behind). After upgrading the plugin, delete the file and the engine rebuilds the cache on first run.