Thinking mode exposes the model's internal reasoning process as a separate response channel, so you can show a "thinking" bubble in your UI before the final answer.
Supported models#
Reasoning reaches you as ThinkingResponse in one of two ways:
-
A runtime thought channel — any
ModelType. A.litertlmbundle that declares the channel (Gemma 4 E2B/E4B, SmolLM3 3B, Phi-4 Mini Reasoning, Qwen3 4B Thinking 2507, …) streams its reasoning there, and the runtime splits it out. -
<think>tags — parsed by core forModelType.deepSeek,qwen,qwen3andqwen35(e.g. DeepSeek R1, Qwen3).
Qwen3 0.6B (ModelType.qwen3) thinks by default. With enableThinking: false
flutter_edge_ai sends enable_thinking: false and appends /no_think
to each
text message — the only off switch for a bundle whose template ignores the flag.
Use Qwen3-0.6B_dynamic_wi4b32_afp32.litertlm from
litert-community/Qwen3-0.6B,
the build the Model Registry offers. Qwen3-0.6B.litertlm
from the same repo, downloaded after the repo replaced its tokenizer (2026-10-13,
15:53 UTC), loads on macOS (other platforms not run); a copy downloaded before
that does not load (#659).
Enable it with enableThinking: true and the matching ModelType. See
Models for which models think.
responseFormat and thinking cannot be enabled on the same session/chat.
openSession or openChat with both responseFormat:
and
enableThinking: true throws ArgumentError before the engine is called:
thought chunks are not values of the requested format. Use a non-thinking
structured session, or keep thinking enabled and validate a prompt-requested
JSON answer yourself.
Some models always reason — DeepSeek R1, Qwen3 4B Thinking 2507. With
enableThinking: false their reasoning is hidden, not skipped: it still costs
time and output tokens, and flutter_edge_ai prints a one-time NOTE per chat
when it hides any.
Reasoning tags are parsed per ModelType, and only deepSeek and the Qwen
types have a tag parser — general, phi and the others have none. On a
.litertlm whose bundle declares a thought channel — SmolLM3 3B (general)
and Phi-4 Mini Reasoning (phi) among them — the runtime splits the
reasoning out and it arrives as ThinkingResponse whatever the type. Without a
channel, a model that reasons but runs as such a type emits no ThinkingResponse, and its
raw thinking blocks arrive inside the answer as ordinary TextResponse tokens.
Strip them yourself, or don't advertise a thinking UI for those models.
Thinking budget#
LiteRtLmEngine(thinkingTokenBudget: 512) (flutter_edge_ai_litertlm 1.18.0)
caps how many tokens each generation with enableThinking: true spends
thinking, in every model that engine builds, on the CPU, the GPU and the NPU.
Null, the default, sets no budget. At the budget LiteRT-LM closes the thought
channel and the model goes on to its answer or tool call:
await FlutterEdgeAi.initialize(
inferenceEngines: const [LiteRtLmEngine(thinkingTokenBudget: 512)],
);
final model = await FlutterEdgeAi.getActiveModel(maxTokens: 4096);
final chat = await model.openChat(
enableThinking: true,
maxOutputTokens: 2048,
modelType: ModelType.gemma4,
);
Without a budget a model can think until maxOutputTokens and end the turn
with an empty answer: Intel's Gemma 4 26B MoE for the Panther Lake NPU did
(6144 tokens of thinking, no tool call, #721), and so did
Qwen3-0.6B_dynamic_wi4b32_afp32.litertlm on the CPU (4507 characters of
thinking, no answer; with a budget of 32, 107 characters, then the answer).
-
The thinking counts towards
maxOutputTokens, and the reply, with the prompt and the history, towards the context (maxTokens, 1024 by default): keep the budget belowmaxOutputTokensand below what the context has left, with room for the answer, or the budget is never reached. -
It applies only with thinking on. A model that reasons with
enableThinking: falsetoo, such as Qwen3 4B Thinking 2507, is not capped then. - It is per generation: every send with thinking on carries it, so in a tool loop a generation after a tool result has its own budget. Gemma 4 E2B did not think again after a tool result in the measured runs, so only the first generation of such a turn was seen capped.
-
LiteRT-LM caps the first channel the bundle declares (measured on Gemma 4 E2B
and
Qwen3-0.6B_dynamic_wi4b32_afp32.litertlm). A bundle that declares none is not capped, and loading one with a budget prints a warning. -
It is fixed when the engine is registered. To change it, close the model,
call
FlutterEdgeAi.reset()(it drops every registration, not only the engine),initializeagain with every provider the app uses, then install and load the model again. - A value outside 1..0x7fffffff throws
ArgumentErrorwhen a model loads. - On the GPU it slows decoding for the whole generation, after the budget is reached too. Gemma 4 E2B with a budget it never reached, same output: 87–97 tokens per second instead of 117–118 on an M5 Pro, 68 instead of 111 on an NVIDIA L4 (Linux). On the CPU no cost was measured (55 tokens per second either way on the M5 Pro, 14.4 on the L4 machine).
-
The web engine accepts it and ignores it:
@litert-lm/corehas no such setting.
Handling thinking responses#
The model emits a ThinkingResponse (with response.content) for its reasoning,
alongside regular TextResponse tokens for the final answer:
chat.generateChatResponseAsync().listen((response) {
if (response is ThinkingResponse) {
// Model's reasoning process
print('Thinking: ${response.content}');
_showThinkingBubble(response.content);
} else if (response is TextResponse) {
// The final answer
print('Text token: ${response.token}');
}
});
You can also create a thinking message manually:
final thinkingMessage = Message.thinking(text: "Let me analyze this problem...");
Platform support#
| Platform | Thinking Mode |
|---|---|
| Android | ✅ Full with .litertlm; tag-based models only with .task |
| iOS | ✅ Full with .litertlm; tag-based models only with .task |
| Desktop (macOS/Windows/Linux) | ✅ Full |
| Web | ✅ Full with .litertlm (Gemma 4 measured in Chrome); not on MediaPipe |
On Web, the .litertlm engine sends the same enable_thinking as native, and
Gemma 4 E2B returns ThinkingResponse in Chrome; core also splits the
<think>...</think> tags a model emits there (e.g. through ONNX on web). MediaPipe Web has no
thinking API, ONNX ignores enableThinking (on Web and native alike, though
core still parses <think> tags a model emits), and the catalog's DeepSeek R1
.task model has no Web entry.
Advanced: ModelThinkingFilter#
For custom inference implementations, ModelThinkingFilter cleans model outputs —
removing model-specific tokens. This is handled automatically by the chat API,
but is available if you need it:
import 'package:flutter_edge_ai/flutter_edge_ai.dart';
String cleanedResponse = ModelThinkingFilter.cleanResponse(
rawResponse,
enableThinking: true,
modelType: ModelType.deepSeek,
fileType: ModelFileType.task,
);
// It removes the reasoning blocks (even when enableThinking is false):
// - <|channel>thought\n...<channel|> (a thought channel, every model type)
// - <think>...</think> (DeepSeek and the Qwen types); for DeepSeek, which
// starts inside its reasoning, everything up to the first </think>
// and trims whitespace. Turn markers (<end_of_turn>, <|im_end|>) are stripped
// only for .bin / .tflite and .gguf files — on .task and .litertlm the runtime
// already ends the turn.
