When is more
reasoning worth it?

We wanted to understand model capabilities on analytical work beyond coding: interpreting spreadsheets, performing calculations and building forecasts. Should you use a cheaper model with more reasoning, or a more expensive model with less? Does the answer change with the complexity of the task?

On mathematical problems, Snell and colleagues found that the benefit of additional computation depended on problem difficulty and how that computation was used. We explored a related practical question: when is it worth paying for more reasoning on analytical work?

We chose business-analysis questions from DSBench, a benchmark designed around realistic data-science tasks. Models could use code as a tool, but we evaluated the final analytical answer, not the code itself. By testing the same questions across three models and three reasoning settings, we examined when extra reasoning improves results and whether the improvement justifies the cost. Our question selection.

We evaluated 134 business-analysis questions using three OpenAI GPT-5.6 models: Luna, Terra and Sol. Each model used reasoning settings of none, low and high. Every setting faced the same questions and tools. The tasks came from DSBench’s business-analysis material, including operating calendars, tax calculations, financial forecasts and loan schedules.

Easy means mostly direct calculations; medium means several linked steps; hard means more complex, interacting steps, often with conditions to track. We assigned these labels from the work required, not the models’ scores.

Imagine the same shop, with increasingly complex questions (illustrative examples):

• Easy: How much did we sell this month?
• Medium: How much profit remains after discounts, product costs and expenses?
• Hard: Will we have enough cash for the next three months if customers pay late and a loan repayment is due?

Full rubric and benchmark examples.

Success means returning the correct final answer; a wrong answer or no answer counts as unsuccessful.

Exact counts · Examples of unsuccessful answers · Research code.

Was reasoning useful on easy work?

Yes. Some reasoning helped every model across easy, medium and hard questions. The benefit wasn’t limited to difficult work. Even straightforward tasks benefited from a little reasoning. Checks on operational failures.

This is not a universal rule. Ma and colleagues found that skipping explicit thinking could work well in tested reasoning models under tight token budgets. Their setup differs from ours; here, low reasoning improved success over none in every model and difficulty group.

What this means

Don’t switch reasoning off just because a task looks easy. Start with some reasoning; then decide whether turning it up is worth the extra cost.

When was turning it up worthwhile?

Compared with low, high reasoning helped Luna most on easy and hard questions, with little change on medium work. Terra gained most on hard questions. Sol benefited most on medium questions.

On hard questions, Sol high solved four that low missed. However, it also missed four that low solved. The totals tied, even though the answers differed. That is not evidence that the two settings are interchangeable. The tie and workload mix.

Research by Chen and colleagues on overthinking shows that models can spend extra computation for little benefit, particularly on simpler problems. That offers context for our mixed returns from high reasoning, but does not establish why particular answers failed in our study.

What should you pay for?

Kapoor and colleagues argue for evaluating agent accuracy and cost together. We use that lens below: what does an upgrade cost, and how many more correct answers does it deliver?

Pay for more correct answers, not simply a higher setting. Cost per correct answer leaves out what a wrong or missing answer costs you. Whether an upgrade is worthwhile depends on how much those failures matter in your work.

The easy-question upgrade cost very little. The medium and hard comparisons bought more correct answers at a larger premium. These figures include model charges and pending reservations, not the full cost of running an agent. Runtime and the cost of mistakes can change which option is worthwhile. Full cost breakdown.

A cheaper model with more reasoning can be a useful alternative to a more expensive model, but only if its accuracy is sufficient for your work. The table pairs a lower-cost option with the highest-scoring setting in each group, not a single best-value choice for everyone. How we chose these options.

1. Group your own tasks by complexity before testing, based on the work required.
2. Compare the same questions across candidate settings. Record correct final answers, failures and total costs.
3. Choose the least expensive option that meets your accuracy needs, then check it on fresh tasks before adopting it.

Use this study to shortlist model and reasoning settings and understand when extra spending may buy better results. The findings come from selected questions, some sharing the same workbook, with difficulty labels assigned by us. We don’t know whether the models saw these public questions during training. Small differences don’t prove one setting is better, so test the options on your own work rather than treating these results as a universal ranking or a guarantee of accuracy or cost.

Some reasoning helped across easy, medium and hard analytical work. More reasoning was not always better, and the most expensive setting did not always return more correct answers. Choose the cheapest setting that meets the accuracy your work needs.