We recently set out to create a coding agent that could run on our local hardware without resorting to using any third-party APIs. This would be a great way to keep it all under our control. We thought Qwen3.6-27B would be the perfect choice, as it could run well on our graphics cards, was decently fast, and perhaps most importantly of all, had a thinking mode for tackling complex logic.
However, when we turned the thinking mode on, the evaluation score ds4-eval-92 dropped from 72.8 to 48.9. We knew the model was capable, so sudden quality degeneration was out of the question. We figured the token limits were fine too. As we found out, a serving control solved the problem with three conditions: there had to be a reliable mode switch hardcoded into the prompt, a way to reinforce the reasoning budget to prevent the thinking mode from consuming the entire token limit, and finally, a way to close the reasoning before the system exhausted the reasoning budget. Ultimately, this was about engineering a resource-management model-serving layer for Qwen.
The baseline metrics
We first ran Qwen without the thinking mode to set baselines. The model met our expectations with the nothink mode and was stable across hardware and quantization values (the values you see in the table below: Q4_K_M, opaque quant, and MLX 8-bit). When you quantize, you store a model’s numerical weights at lower precision so it consumes less memory and runs smoothly on smaller hardware. It can affect quality, but the results below were close enough to show that the basic model behaviour was consistent.
| Serving path | Mode | ds4-eval-92 |
|---|---|---|
| lucebox, RTX 5090 Laptop, Q4_K_M | nothink | 70.7 |
| lucebox, RTX 3090 Ti, Q4_K_M | nothink | 71.7 |
| OpenRouter, opaque quant | nothink | 72.8 |
| MLX 8-bit, Mac Studio M2 Ultra | nothink | 73.9 |
We now enabled thinking mode and ran the same benchmark again because our evaluation contained tasks where additional reasoning should have helped.
| Mode | ds4-eval-92 | Note |
|---|---|---|
| nothink | 72.8 | clean nothink, zero thinking tokens |
| think, unbudgeted | 48.9 | 84/92 reasoned, 32 hit the length cap |
| think, budgeted | 76.1 | force-close on 44/92 |
The unbudgeted run makes the failure apparent. While the system activated thinking mode correctly on most examples, the problem was that the reasoning process would continue until it would consume the entire output allowance. The evaluator was still left with an incomplete response and an unclosed reasoning block, so there was no parseable final answer.
Once the serving system began controlling the reasoning budget, the score rose to 76.1. Note that this was higher than the non-thinking baseline. So we could tell reasoning had been useful all along, but it simply had not been metered.
Creating a reliable non-thinking mode control
Our first task was to create a dependable non-thinking mode. When we looked at the response tokens, we saw that some nothink runs still had thinking tokens anyway; the model was still reasoning despite explicitly telling it not to.
The reason becomes clear when you take a moment to see how the model request passes through several layers before the model generates a response. There may be an application, an OpenAI-compatible API, a hosted inference provider, a chat template, a model runtime, and, finally, the model itself. Now each layer interprets control parameters differently, may ignore fields it does not recognise, or translate them into another internal format.
This effectively means there is no single, portable command that disables reasoning across every Qwen serving stack.
Depending on the system, a request might include one of the following:
"chat_template_kwargs": {
"enable_thinking": false
}
"thinking": {
"type": "disabled"
}
"reasoning_effort": "none"These settings look as though they express the same intention, but deep down, they aren’t necessarily traveling through the same code paths.
MLX, for example, respected the chat-template flag. lucebox respected its own thinking configuration. In the OpenRouter path we tested, none of these controls reliably prevented the model from producing reasoning tokens. And as we mentioned, this became visible only after we inspected the response itself.
We found that the most reliable method in that environment was to include /no_think directly in the prompt. Qwen’s own chat template recognises this instruction, so the signal does reach the layer that ultimately determines the model’s behaviour. Once it was added, the number of thinking tokens fell to zero, and the evaluation score returned to the expected range.
It’s important to remember that you cannot assume a control works merely because it was present in the request. It’s crucial to check the response; that should be your single source of truth. So, if a supposedly non-thinking run contains thinking tokens, it was not a non-thinking run, regardless of which API field was set.
Enforcing a reasoning budget
Our second control was an enforced reasoning budget. Fields such as reasoning_effort and budget_tokens have no effect unless the serving system actively counts generated tokens and intervenes when the specified limit is reached, even though they may sound authoritative. Our unbudgeted OpenRouter run failed badly for this very reason. A parameter can be accepted by an API, preserved in a request log, and still not influence the decode loop that produces the model’s response.
In our unbudgeted run, this happened repeatedly; Qwen continued reasoning until the response reached its length cap, and the grader was often unable to find a complete, parseable answer.
Here are the category-level results that make the pattern clearer.
| Area | nothink | think, unbudgeted | think, budgeted |
|---|---|---|---|
| hellaswag | 86 | 34 | 88 |
| longctx | 100 | 33 | 100 |
| gsm8k | 93 | 77 | 96 |
| truthfulqa | 80 | 51 | 77 |
HellaSwag and longctx (a separate long-context test that’s part of luce-bench) often reward a concise, committed answer. Their scores collapsed when we ran the thinking mode without enforcement, as they do not necessarily benefit from an extended reasoning trace (especially when that trace consumes the space needed to state the result).
GSM8K, which contains mathematical word problems, behaved differently. Reasoning was more useful there, so the unbudgeted score declined less severely. However, even here, the budgeted thinking run improved the score (even outperforming non-thinking mode).
Our conclusion here is that reasoning is critical, but it has a cost, and the results drastically improve once it is metered by a serving system that determines how much of that cost a particular request can afford. So unmetered thinking was the failure mode.
Metering the reasoning space
The third and final control was a mechanism to protect enough output space for the final answer.
Qwen’s reasoning output is normally divided into two sections. The model first writes an internal reasoning trace between <think> and </think> tags, then writes the visible reply that the user (or evaluator) is expected to read.
A conventional max_tokens setting imposes one universal limit over the entire response, but the caveat is that it does not inherently distinguish between the reasoning section and the final answer.
This means that if the model spends 8,000 tokens reasoning in <think> with the total response limit at 8,000 tokens, the request may end before a single answer token appears after </think>. From the model’s perspective, sure, it has done a substantial amount of work. But from the evaluator’s perspective, it has returned nothing useful.
The opposite scenario also holds: if the cap is too tight, the model can close the thought block without any room left for the final answer.
A production-quality reasoning system needs to track at least three values that cannot safely be collapsed into a single response cap:
- the maximum number of tokens that may be spent inside the reasoning phase
- the maximum number of tokens allowed for the entire response
- the minimum number of tokens that must remain available for the visible reply
The Qwen3.6 model-card sidecar defines several reasoning tiers:
| Tier | Reasoning budget (tokens) |
|---|---|
| low | 4,032 |
| medium | 16,128 |
| high | 32,256 |
| x-high | 56,832 |
| max | 81,408 |
The same configuration reserves 4,096 tokens for the visible answer, which is easy to underestimate.
A force close that stops the reasoning process but still leaves no answer budget will still fail. The crucial thing here is that the intervention has to happen while the model still has room to change modes and finish the request.
Let us see an example. Suppose the total response allowance is 20,000 tokens and the serving system preserves 4,096 tokens for the final answer. The reasoning phase must be stopped before it passes roughly 15,904 tokens, and potentially earlier if the closing sequence itself consumes additional tokens.
As you can see, “think less” is not a dependable control. A natural-language instruction does not create a hard boundary. Once generation has entered a long reasoning trace, the model may continue elaborating because that behaviour is statistically likely given the text already in context. The server has to count the generated tokens and pull Qwen out of <think> when the budget approaches its limit.
Closing reasoning without breaking the answer
We could insert a bare </think> tag to establish a boundary that a parser can recognise, but it does not always provide enough context for the model to understand that it should now summarise its work and produce the final response. In some cases, the model may continue as though the reasoning process had merely been interrupted, or it may generate an awkward continuation that does not resemble a direct answer.
The Qwen3 technical report describes a phrase used during training when the model needs to end its reasoning because of a time constraint:
Considering the limited time by the user, I have to give the solution based on the thinking directly now.
That phrase is followed by the closing </think> tag.
The model has encountered this transition pattern during training, so the phrase matters. The serving system gives Qwen a familiar instruction: stop extending the derivation and provide the best answer available from the work already completed rather than presenting an arbitrary closing tag.
Lucebox stores this sequence in the Qwen3.6 sidecar and tokenizes it when the server starts. When the reasoning budget is reached, the decoding loop overrides the model’s next sampled tokens with the trained closing sequence.
A decoding loop is the repeated process through which a language model selects one token, adds it to the context, and then selects the next. The server intervenes directly in this loop to guarantee that the closing sequence is emitted at the correct moment.
This approach also preserves the model’s KV cache, which is the runtime memory that stores information derived from the tokens the model has already processed. Without this cache in place, the model would have to recompute the entire conversation and reasoning trace every time it generated another token. Qwen is able to move from the interrupted reasoning phase to the final answer without forgetting or reprocessing the work it has already done because it has kept that state intact.
The model therefore sees its full reasoning trace, receives the familiar instruction to wrap up, closes the thinking block, and continues directly into the visible reply.
This is considerably more reliable than merely terminating generation and hoping the text produced so far contains an answer.
When the backend cannot be changed
It is not always possible to directly control the decode loop. Hosted inference providers sometimes expose streaming output and a small set of configuration parameters and keep the actual generation system inaccessible.
In that case, luce-bench uses a different fallback, which is not as elegant, though. It watches the streamed response and counts tokens as they arrive, and when the reasoning allowance has been exhausted, it aborts the request and sends a second request containing the existing context together with the trained closing instruction.
This method requires another round trip and may repeat some computation because the second request cannot necessarily reuse the first request’s KV cache. It is slower and more expensive than intervening inside the original decode loop.
However, we’d still prefer it to allowing the first request to consume the entire output limit inside <think> and return no usable answer.
When you design systems around hosted models, this distinction can be useful. If the backend exposes a genuine reasoning budget, test that control without assumptions. If it does not expose one, the application may need to enforce the limit by monitoring the stream and continuing the interaction explicitly.
The results after all 3 serving layer controls
When we began treating reasoning as a separate, metered phase, Qwen again began behaving as we expected it to.
On OpenRouter, the benchmark score rose from 48.9 with unbudgeted thinking to 76.1 with budget enforcement. On the Mac Studio, MLX 8-bit reached 83.7 with budgeted thinking, compared with 73.9 in non-thinking mode.
The force-close mechanism activated on 44 of the 92 OpenRouter examples and 50 of the 92 MLX examples. Both runs recorded zero continuation failures, which meant the serving system successfully moved the model from the interrupted reasoning phase into a final answer every time it intervened.
Those numbers also show that force-closing was required on roughly half the benchmark. If we’d have designed a deployment assuming the model would voluntarily finish reasoning within the desired allowance, it would have behaved inconsistently. So a force-close is definitely not an unusual recovery path only reserved for extreme cases.
Non-thinking mode is still valuable as it is cheaper, steadier, and easier to compare across providers, and many tasks do not benefit from a long internal derivation. You can use a model that answers immediately for a request that needs a classification label, a small transformation, or a direct extraction.
For harder problems, however, Qwen’s thinking mode can produce better results, provided that the serving system treats reasoning as a limited resource that is strictly metered rather than an open-ended behaviour. Otherwise, “thinking” simply means spending the entire response budget in scratch space and disappearing before the answer.
Conclusion
We had started out wanting a coding agent that ran reliably on our own hardware. We dove deep to map the entire path between the prompt and the text that finally appeared in the terminal and was finally able to optimize the result: Qwen needed a serving stack to know when thinking had gone on long enough.
Our fix required control at three points in the serving stack:
- A mode switch that worked consistently
- A reasoning budget that was actually enforced
- A token limit that kept enough space for the final answer
We like to think of Qwen’s thinking mode as a resource that needs to be metered by the serving stack rather than a switch that you simply turn on or off. It is a separate stage of computation that has to be measured, limited, and ended deliberately. The server needs to know when reasoning begins, how much of the available output it has consumed, and when to close the reasoning block so the model could still produce a complete reply.
The model could spend most of its response working through the problem and leave little or no room for the answer if that control is not in place.
Notes
Benchmark results are from ds4-eval-92, using a single seed and one pinned grader, version v0.2.7.dev0.
The mode-switching and reasoning-budget implementation draws on the Qwen3 technical report and the Qwen3.6 model-card values transcribed into lucebox.
The hellaswag, gsm8k, and truthfulqa areas are drawn from the public HellaSwag, GSM8K, and TruthfulQA benchmarks; longctx is the long-context test from luce-bench.