Decisions

A decider maps an input onto one of the options your app defines — an intent, an action, a screen, a label, a route, the few tools worth offering a small function-calling model — and says "none of these" when nothing fits. It runs on device, costs one embedding per input, and needs no training.

Packages#

dependencies:
  flutter_edge_ai: ^2.4.0
  flutter_edge_ai_litertlm: ^1.14.0 # EmbeddingGemma 2 (.litertlm) and 1 (.tflite)

Any embedding model works: EmbeddingModel.createDecider is implemented once in core over embed, so every embedding backend has it. The numbers on this page are EmbeddingGemma 2 text 270m (embeddinggemma-2-text-270m.litertlm).

Quick start#

// ignore_for_file: experimental_member_use
import 'package:flutter_edge_ai/flutter_edge_ai.dart';
import 'package:flutter_edge_ai_litertlm/flutter_edge_ai_litertlm.dart';

Future<void> decide() async {
  await FlutterEdgeAi.initialize(
    embeddingBackends: const [LiteRtEmbeddingBackend()],
  );
  await FlutterEdgeAi.installEmbedder()
      .modelFromNetwork(
        'https://huggingface.co/litert-community/'
        'embeddinggemma-2-text-270m-litert-lm/resolve/'
        '9be6e8b90982095dc05c2bd162e4b954ee4dbac7/'
        'embeddinggemma-2-text-270m.litertlm',
      )
      .install();
  final embedder = await FlutterEdgeAi.getActiveEmbedder();

  final decider = await embedder.createDecider(
    options: [
      DeciderOption(
        'set_timer',
        description: 'Start a countdown timer',
        examples: [
          EmbeddingInput.text('set a timer for ten minutes'),
          EmbeddingInput.text('wake me in half an hour'),
          EmbeddingInput.text('countdown 5 minutes'),
        ],
      ),
      DeciderOption(
        'call_contact',
        description: 'Phone someone from the contacts',
        examples: [
          EmbeddingInput.text('call mum'),
          EmbeddingInput.text('ring Alex on his mobile'),
          EmbeddingInput.text('phone the office'),
        ],
      ),
    ],
  );

  final answer = await decider.choose(
    DeciderInput(EmbeddingInput.text('remind me in 20 minutes')),
  );
  if (answer.choice == null) {
    print('none of these: ${answer.abstainReason}');
  } else {
    print('${answer.choice} (margin ${answer.margin})');
  }
}

How it decides#

1. Every option input — its description, then its examples — is embedded once, at createDecider, with TaskType.sentenceSimilarity. The input to decide on gets the same prefix: EmbeddingGemma 2's model card asks for one prefix on every compared input. Why this prompt: see Measured accuracy.
2. An option is the centroid of its inputs (their normalised mean). A description alone is zero-shot.
3. choose embeds the input and scores every candidate:
   • DeciderMethod.centroid (default): cosine to the option's centroid (ScoreKind.cosine);
   • DeciderMethod.linear: a softmax regression fitted on the frozen embeddings at createDecider, weight decay fixed at 1e-5 (ScoreKind.logprob).
4. The abstain signal is the input's highest cosine to a candidate's centroid — for both methods. Below the calibrated threshold the answer has choice: null and AbstainReason.belowThreshold.

answer.ranked holds the best topK candidates (default 5), best first; on a tie the option given first ranks first. answer.margin is best minus runner-up. choose(input, among: [...]) restricts the candidates to some option ids: an unknown id throws ArgumentError, an empty set abstains with AbstainReason.noCandidates without embedding anything (a closed embedder or an input the model was not loaded for still fails, as with candidates).

ScoreKind, AbstainReason and DeciderMethod are classes with constant values, not enums, so a value added in a later release does not break your switch — give it a default branch. A DeciderAnswer checks itself: exactly one of choice and abstainReason, choice first in ranked, ranked highest first, confidence in [0, 1] and only with probabilities.

A raw score is not a probability#

A cosine of 0.83 does not mean 83 %. RankedOption.probability, DeciderAnswer.probabilities and DeciderAnswer.confidence are null unless the decider was calibrated on held-out labelled requests (DeciderCalibration.fromLabelled), which fits a softmax temperature. One formula decodes every method's raw scores: softmax(score / T) over the candidates. confidence is Jev's spread measure, (N · pmax − 1) / (N − 1): 0 for a uniform distribution, 1 when one option takes everything. It is not an accuracy.

Calibration: when to say "none of these"#

fromLabelled refuses (ArgumentError) rows it cannot calibrate on: when no threshold keeps their error within maxError, and when the best temperature is an end of its grid — rows all answered right with room to spare would make a near-tie look certain (one row always ends up refused one way or the other). The rows are copied when the calibration is made.

A softmax probability cannot abstain: with two options any request looks confident. Measured with EmbeddingGemma 2 on CLINC150 (k = 10 examples per option), a 5 %-budget threshold on the softmax rejects 2.9–18.4 % of out-of-subset requests at 2–20 options; the cross-fitted cosine rejects 92–99 %.

fromLabelled sees only requests that belong to an option, so its threshold is set by in-scope errors alone, and it falls as the decider gets more accurate. On CLINC150 at 150 options (centroid) it refused 94 % of the out-of-scope requests with 5 examples per option, 85 % with 10 and 4 % with 30; the cross-fit refused 89 / 90 / 94 %. When requests that fit no option matter, keep the cross-fit, or check that the threshold fromLabelled picked still refuses them.

An input of another modality than the options needs its own threshold. A spoken request compared with text options reaches lower cosines than the same request typed: in the example's Tiny Island (EmbeddingGemma 2 740m, synthetic speech) typed requests calibrate at 0.866 and spoken ones at 0.745 for about the same error. One decider has one threshold, so Tiny Island keeps a second decider over the same options for spoken requests, with minSimilarity(0.745) measured on spoken requests.

Measured accuracy#

All numbers: Google's int4 embeddinggemma-2-text-270m.litertlm through LiteRT-LM 0.18.0's EmbeddingEngine on a CPU (Apple M4 Pro), the decider's sentence similarity prompt, CLINC150 (its own out-of-scope rows as "none of these"). The measurement code is the one this API reproduces numerically; English, one-sentence requests.

Which inputs are answered and refused depends only on the abstain signal and its threshold, the same for both methods; the error on the answered ones was measured with linear, and centroid makes more (it is less accurate). At 150 options with 5 examples, the cross-fit's error (6.0 %) is over a 5 % budget: calibrate with fromLabelled there. On Tiny Island the cross-fit stays within it (3.0 %); the app calibrates with fromLabelled anyway, which answers more of the requests (99 % against 94 %) at 4.2 % error.

The prompt. Sentence similarity was the most accurate EmbeddingGemma 2 prompt on CLINC150 and on the example app's Tiny Island catalog (184 actions), against classification, question answering, a document prompt for the options and Google Decision Maker's title: <option> | text: candidates:

On HWU64 classification was ahead with every method, and so it is not best everywhere. Not measured yet: requests longer than a sentence, a large set in languages other than English, and accuracy on a phone or an NPU.

Speed#

choose costs one embedding plus one dot product per candidate.

Measured on an Apple M4 Pro (macOS 26.5.1), EmbeddingGemma 2 text 270m on the CPU, the example app's integration_test/decider_benchmark_test.dart (debug build), zero-shot options from descriptions, two runs of 60 choose calls per N; medians:

The embedding of the input dominates, and it does not depend on N: the fastest single query took 30–31 ms at every N. The N = 500 rows were slower in both runs (fastest query 47–62 ms), which this run does not explain. createDecider embeds one input per option, one after another (35–60 ms each). Phones and NPUs are not measured yet.

Fitting linear (softmax regression, float64 L-BFGS on an isolate of its own) on an Apple M4 Pro, release (AOT) build:

centroid fits nothing; its default cross-fit took 0.2 s at 150 × 30. The optimiser keeps a history of 20 pairs of parameter-sized vectors: about 2 · 20 · (768 · C + C) · 8 bytes — 37 MB at 150 options, ~0.5 GB at 2000 — on top of the inputs.

On web there is no isolate to spare: the fit runs on the page's thread in slices of 4 ms and yields to the event loop between them (and checks the cancel token there), so the page keeps drawing and taking input while it fits. Measured on an Apple M4 Pro in Chrome 155 (headless), release builds (flutter build web, dart2js and --wasm), synthetic unit rows (768 dimensions, 10 per option), cross-fit calibration; two runs:

Run without yielding, the same fits held the page for their whole length — 10.1–11.0 s (dart2js) and 11.4–11.7 s (wasm) at 150 linear; the yields cost less than the run-to-run spread (±2 %). The web fit computes the same numbers as the VM: test/web/decider_web_test.dart holds it to the VM's results in the browser.

Many decisions over one input#

Decider.chooseAll(input, deciders) answers several deciders, in order. Deciders created from one EmbeddingModel share one embedding of the input:

// ignore_for_file: experimental_member_use
import 'package:flutter_edge_ai/flutter_edge_ai.dart';

Future<void> triage(EmbeddingModel embedder) async {
  final intent = await embedder.createDecider(options: [
    DeciderOption('buy', examples: [
      EmbeddingInput.text('I want to order this'),
      EmbeddingInput.text('add it to my cart'),
    ]),
    DeciderOption('return', examples: [
      EmbeddingInput.text('this arrived broken'),
      EmbeddingInput.text('how do I send it back'),
    ]),
  ]);
  final tone = await embedder.createDecider(
    options: [
      DeciderOption('calm', description: 'calm, neutral wording'),
      DeciderOption('upset', description: 'angry or upset wording'),
    ],
    calibration: const DeciderCalibration.none(),
  );
  final answers = await Decider.chooseAll(
    DeciderInput(EmbeddingInput.text('the box was crushed, I am furious')),
    [intent, tone],
  );
  print('${answers[0].choice} / ${answers[1].choice}');
}

Tool routing for function calling#

FunctionGemma 270M has a 1024-token context: hundreds of tool declarations do not fit. A decider narrows them to the few likely ones, and only those go to the chat:

// ignore_for_file: experimental_member_use
import 'package:flutter_edge_ai/flutter_edge_ai.dart';

Future<InferenceChat> chatWithLikelyTools(
  EmbeddingModel embedder,
  InferenceModel model,
  List<Tool> tools,
  String request,
) async {
  final router = await embedder.createDecider(
    options: [
      for (final tool in tools)
        DeciderOption(tool.name, description: tool.description),
    ],
    calibration: const DeciderCalibration.none(),
  );
  final picked = await router.choose(
    DeciderInput(EmbeddingInput.text(request)),
    topK: 3,
  );
  return model.createChat(
    tools: picked.choice == null
        ? <Tool>[]
        : [
            for (final r in picked.ranked)
              tools.firstWhere((t) => t.name == r.id),
          ],
    supportsFunctionCalls: true,
    modelType: ModelType.functionGemma,
  );
}

Descriptions alone cannot cross-fit, so this router never abstains: it always offers the three nearest tools. To let it say "no tool", give each tool a few example requests and keep the default cross-fit; choice == null then offers none. A fixed minSimilarity threshold belongs to one model file and one prompt: use one only after measuring it for yours.

Store and restore#

toArtifact() returns a JSON-encodable, versioned map: the options (ids, descriptions, text examples, metadata), each option's centroid (and linear weights) as base64 little-endian float32, the method, the calibration and the embedding-space id. EmbeddingModel.loadDecider(artifact) rebuilds the decider without embedding anything.

// ignore_for_file: experimental_member_use
import 'dart:convert';

import 'package:flutter_edge_ai/flutter_edge_ai.dart';

Future<Decider> restore(EmbeddingModel embedder, String stored) =>
    embedder.loadDecider(jsonDecode(stored) as Map<String, Object?>);

The artifact is pinned to its embedding space: the model's installed files, the dimension, the backend and the task prefix. The files are named by their install names and, natively, their sizes in bytes — file names and sizes, not the bytes themselves, so install new weights under a new name (on web, where core does not read sizes, the names alone). A decider fitted on the CPU file is refused on a per-SoC NPU bundle with DeciderSpaceMismatchException: EmbeddingGemma 2's NPU vectors against its CPU file's measured cosine 0.83 on average for short texts, rising to 0.995 at 400 words, and 0.86–0.88 for images. A decider fitted on at least one image (an option example or a labelled request) also carries the embedder's image budget — EmbeddingGemma 2 encodes an image into 140 tokens, or 70 with LiteRtEmbeddingBackend(visionTokensPerImage: 70), and the same photo gives another vector under the other budget — so it is refused at the other budget too; a decider fitted on text alone is not, and an embedder widened to images does not move its space. A malformed or newer artifact, or one that contradicts itself, is a FormatException. Media examples are not stored (their count is); a restored option lists its text examples.

A model that FlutterEdgeAi.getActiveEmbedder() did not build (a custom subclass, a test fake) has no installed files to name its space, and a model that reports no backend (activeBackend null: LiteRT.js and ONNX on web) cannot tell WebGPU from WASM — both are refused with ArgumentError unless you pass embedderId: to createDecider and loadDecider. An embedderId replaces the installed-file check: you own the identity, so change it whenever the weights (or, for a model without a backend, the backend) change.

A decider borrows its embedder and never closes it. On an embedder getActiveEmbedder() returned, it follows core: when that embedder is closed — core widened it for images or audio, another backend, a re-install, a worker that died, or the app closed it — the next choose asks core for the active embedder again (with the backend and media the old one was built with, under the cache's usual reuse rule, so it may load the model again) and retries there. If that embedder is another embedding space, choose throws DeciderSpaceMismatchException. On a custom model it fails with EmbedderClosedException and decider.isClosed is true: load the artifact on the new model.

Progress, cancel and your own deciders#

createDecider(onProgress: (done, total) => …) counts the inputs it embeds — each option's description and examples, then fromLabelled's rows — from (0, total) to (total, total). It does not cover the fit after them, which for linear at 150 options takes seconds. cancelToken: stops the embedding and the fit (natively the fit's isolate is killed, on web it stops at the next yield): EmbeddingCancelledException, and no decider. A failed option input names its option and input: Option "returns", example 3: …, with the exception's type and fields kept.

A decider of your own, or a test fake, extends Decider (an abstract base class: members added later reach it with a default). toArtifact() defaults to throwing DeciderUnsupportedException.

Jev Choice JSON#

DeciderOption.fromJevChoice(question) turns a Jev (TypeSafe AI) Choice question — {"type": "choice", "criteria": {name: description}} — into options; a null criterion uses the name as its description. toJevJson() returns {"type": "choice", "choice": …, "probabilities": …, "confidence": …}, with probabilities and confidence only when calibrated. Choice questions only: noul and score throw ArgumentError. Jev has no abstention; an abstained answer has "choice": null.

Errors#

Every error arrives as a failed future.