LogoFlutter Edge AI

Thinking Mode

View a model's reasoning, from a runtime thought channel or think tags, as separate thinking responses.

Thinking mode exposes the model's internal reasoning process as a separate response channel, so you can show a "thinking" bubble in your UI before the final answer.

Supported models#

Reasoning reaches you as ThinkingResponse in one of two ways:

  • A runtime thought channel — any ModelType. A .litertlm bundle that declares the channel (Gemma 4 E2B/E4B, SmolLM3 3B, Phi-4 Mini Reasoning, Qwen3 4B Thinking 2507, …) streams its reasoning there, and the runtime splits it out.
  • <think> tags — parsed by core for ModelType.deepSeek, qwen, qwen3 and qwen35 (e.g. DeepSeek R1, Qwen3).

Qwen3 0.6B (ModelType.qwen3) thinks by default. With enableThinking: false flutter_edge_ai sends enable_thinking: false and appends /no_think to each text message — the only off switch for a bundle whose template ignores the flag. Use Qwen3-0.6B_dynamic_wi4b32_afp32.litertlm from litert-community/Qwen3-0.6B, the build the Model Registry offers. Qwen3-0.6B.litertlm from the same repo, downloaded after the repo replaced its tokenizer (2026-10-13, 15:53 UTC), loads on macOS (other platforms not run); a copy downloaded before that does not load (#659).

Enable it with enableThinking: true and the matching ModelType. See Models for which models think.

responseFormat and thinking cannot be enabled on the same session/chat. openSession or openChat with both responseFormat: and enableThinking: true throws ArgumentError before the engine is called: thought chunks are not values of the requested format. Use a non-thinking structured session, or keep thinking enabled and validate a prompt-requested JSON answer yourself.

Some models always reason — DeepSeek R1, Qwen3 4B Thinking 2507. With enableThinking: false their reasoning is hidden, not skipped: it still costs time and output tokens, and flutter_edge_ai prints a one-time NOTE per chat when it hides any.

Reasoning tags are parsed per ModelType, and only deepSeek and the Qwen types have a tag parser — general, phi and the others have none. On a .litertlm whose bundle declares a thought channel — SmolLM3 3B (general) and Phi-4 Mini Reasoning (phi) among them — the runtime splits the reasoning out and it arrives as ThinkingResponse whatever the type. Without a channel, a model that reasons but runs as such a type emits no ThinkingResponse, and its raw thinking blocks arrive inside the answer as ordinary TextResponse tokens. Strip them yourself, or don't advertise a thinking UI for those models.

Thinking budget#

LiteRtLmEngine(thinkingTokenBudget: 512) (flutter_edge_ai_litertlm 1.18.0) caps how many tokens each generation with enableThinking: true spends thinking, in every model that engine builds, on the CPU, the GPU and the NPU. Null, the default, sets no budget. At the budget LiteRT-LM closes the thought channel and the model goes on to its answer or tool call:

await FlutterEdgeAi.initialize(
  inferenceEngines: const [LiteRtLmEngine(thinkingTokenBudget: 512)],
);
final model = await FlutterEdgeAi.getActiveModel(maxTokens: 4096);
final chat = await model.openChat(
  enableThinking: true,
  maxOutputTokens: 2048,
  modelType: ModelType.gemma4,
);

Without a budget a model can think until maxOutputTokens and end the turn with an empty answer: Intel's Gemma 4 26B MoE for the Panther Lake NPU did (6144 tokens of thinking, no tool call, #721), and so did Qwen3-0.6B_dynamic_wi4b32_afp32.litertlm on the CPU (4507 characters of thinking, no answer; with a budget of 32, 107 characters, then the answer).

  • The thinking counts towards maxOutputTokens, and the reply, with the prompt and the history, towards the context (maxTokens, 1024 by default): keep the budget below maxOutputTokens and below what the context has left, with room for the answer, or the budget is never reached.
  • It applies only with thinking on. A model that reasons with enableThinking: false too, such as Qwen3 4B Thinking 2507, is not capped then.
  • It is per generation: every send with thinking on carries it, so in a tool loop a generation after a tool result has its own budget. Gemma 4 E2B did not think again after a tool result in the measured runs, so only the first generation of such a turn was seen capped.
  • LiteRT-LM caps the first channel the bundle declares (measured on Gemma 4 E2B and Qwen3-0.6B_dynamic_wi4b32_afp32.litertlm). A bundle that declares none is not capped, and loading one with a budget prints a warning.
  • It is fixed when the engine is registered. To change it, close the model, call FlutterEdgeAi.reset() (it drops every registration, not only the engine), initialize again with every provider the app uses, then install and load the model again.
  • A value outside 1..0x7fffffff throws ArgumentError when a model loads.
  • On the GPU it slows decoding for the whole generation, after the budget is reached too. Gemma 4 E2B with a budget it never reached, same output: 87–97 tokens per second instead of 117–118 on an M5 Pro, 68 instead of 111 on an NVIDIA L4 (Linux). On the CPU no cost was measured (55 tokens per second either way on the M5 Pro, 14.4 on the L4 machine).
  • The web engine accepts it and ignores it: @litert-lm/core has no such setting.

Handling thinking responses#

The model emits a ThinkingResponse (with response.content) for its reasoning, alongside regular TextResponse tokens for the final answer:

chat.generateChatResponseAsync().listen((response) {
  if (response is ThinkingResponse) {
    // Model's reasoning process
    print('Thinking: ${response.content}');
    _showThinkingBubble(response.content);
  } else if (response is TextResponse) {
    // The final answer
    print('Text token: ${response.token}');
  }
});

You can also create a thinking message manually:

final thinkingMessage = Message.thinking(text: "Let me analyze this problem...");

Platform support#

PlatformThinking Mode
Android ✅ Full with .litertlm; tag-based models only with .task
iOS ✅ Full with .litertlm; tag-based models only with .task
Desktop (macOS/Windows/Linux)✅ Full
Web ✅ Full with .litertlm (Gemma 4 measured in Chrome); not on MediaPipe

On Web, the .litertlm engine sends the same enable_thinking as native, and Gemma 4 E2B returns ThinkingResponse in Chrome; core also splits the <think>...</think> tags a model emits there (e.g. through ONNX on web). MediaPipe Web has no thinking API, ONNX ignores enableThinking (on Web and native alike, though core still parses <think> tags a model emits), and the catalog's DeepSeek R1 .task model has no Web entry.

Advanced: ModelThinkingFilter#

For custom inference implementations, ModelThinkingFilter cleans model outputs — removing model-specific tokens. This is handled automatically by the chat API, but is available if you need it:

import 'package:flutter_edge_ai/flutter_edge_ai.dart';

String cleanedResponse = ModelThinkingFilter.cleanResponse(
  rawResponse,
  enableThinking: true,
  modelType: ModelType.deepSeek,
  fileType: ModelFileType.task,
);

// It removes the reasoning blocks (even when enableThinking is false):
// - <|channel>thought\n...<channel|> (a thought channel, every model type)
// - <think>...</think> (DeepSeek and the Qwen types); for DeepSeek, which
//   starts inside its reasoning, everything up to the first </think>
// and trims whitespace. Turn markers (<end_of_turn>, <|im_end|>) are stripped
// only for .bin / .tflite and .gguf files — on .task and .litertlm the runtime
// already ends the turn.