Reasoning techniques: when thinking longer helps and when it only costs money

A finance assistant has one tool, get_budget(account), and a thinking budget of 16,000 tokens, because someone read that reasoning makes models more accurate. A manager asks: "How much is left in the marketing budget this month?" The model thinks for 3,900 tokens about how marketing budgets are usually set, how spend tends to track through a month, and what a reasonable figure would be. It answers: "Roughly 8,000 should be left, assuming even spend." Forty seconds, a few cents, and a number that came from nowhere. The tool was never called.

Nothing in the run is broken in the usual sense. The model followed its instructions and used its budget. The design is what is wrong: thinking was applied to a question that thinking cannot answer.

Thinking rearranges what is in the context. It cannot add a fact that is not there. For a question whose answer lives in a tool, a database or a document, more thinking produces a better written guess, not a better answer. Line 3 is 3,900 tokens of estimation about a number that one tool call would have returned.

Where thinking does help is well documented. Wei and others showed in 2022 that adding eight worked examples with reasoning steps took PaLM 540B from about 18 percent to about 57 percent on GSM8K, a set of grade school math word problems. Kojima and others showed the same year that the single phrase "Let's think step by step" took text-davinci-002 from 17.7 to 78.7 percent on MultiArith and from 10.4 to 40.7 percent on GSM8K. Those are multi-step problems where every fact needed is in the prompt and the work is in combining them. That is the pattern: thinking pays when the answer requires steps over known facts.

Reasoning models since OpenAI's o1 in September 2024 do this at training time and expose a budget. Anthropic's extended thinking takes a budget_tokens value with a minimum of 1,024, and the thinking tokens are billed as output tokens. So the budget is a cost line, and the question is where to spend it. Apple's June 2025 paper, The Illusion of Thinking, found three regimes on puzzle tasks: at low complexity standard models did as well or better, at medium complexity thinking helped, and at high complexity both collapsed, with thinking effort falling even when budget was left. The method is disputed, and a rebuttal argued the collapse was partly the test setup. Two things are not disputed: thinking has a sweet spot, and it is not free.

Sort the calls. Lookups, extraction, classification and formatting get no thinking budget, or the minimum. The answer is in the input or in a tool, and the job is to fetch it and shape it. Multi-step problems get a budget: planning a sequence of tool calls, reconciling three documents that disagree, writing code against constraints, math. Put that sort in code, by route, not in the prompt, so a cheap question does not pay a reasoning model's price.

For the finance assistant the fix has two parts. Tool first: a question about a current value must call get_budget before it answers, and the run fails if it gives a figure with no tool result behind it. Then the budget: this route gets the minimum, because the hard part is the lookup, not the reasoning. The eval that catches it is cheap. For every answer containing a number, check that a tool returned that number.

Measure, do not assume. Run the same eval set at a few budgets and plot accuracy against cost. On most extraction and lookup sets the line is flat and the cost is not. On a planning set it usually climbs and then flattens, and the flat part is where your budget should stop. The field disagrees on whether the visible reasoning is a faithful account of what the model did. It agrees that the accuracy curve is the thing to measure.

Give the rule and the exception. "Thinking helps when the answer needs steps over facts that are already in the context: math, planning, reconciling sources. It does not help when the fact is missing, and no budget fixes a missing tool call. I set the budget per route and check it against an eval, because the gain flattens and the cost does not." If they show you this trace, say line 3: a guess dressed up as reasoning, one tool call away from the real number.