Does a good review rule travel?
In Part 1, we asked how to choose financial AI answers for review when we cannot check everything. For the GPT-5.4 mini setup we tested, checking its calculations and cited figures found more mismatches with the answer key than relying on its confidence.
But was that a useful general rule, or a result specific to that setup and those questions?
For Part 2, we collected new answers from four model setups, using a separately selected set of FinQA questions and adding a second financial dataset, TAT-QA. We did not reuse Part 1’s answers. After exclusions, all four setups were compared on the same 942 questions. Question selection and exclusions.
The result changed with the setup: verification still led for Mini, but confidence led for Sol and Gemini.
The same two ways to choose what to check
- Confidence: ask the AI. “How sure are you?” Review the answers it is least confident about first.
- Verification: check the work. Our rule-based checker recalculates the answer, looks for cited figures at their source locations, and checks for inputs the calculation did not use. Answers that fail these checks move up the review queue.
These checks test consistency, not whether the AI chose the right figures to answer the question. Neither strategy sees the answer key when choosing what to review. Exact checking and scoring rules.
Mini used low reasoning; the other three used medium. These are comparisons of complete setups, not a test that isolates model intelligence or reasoning effort.
1. The better strategy depended on the setup
Start with the same review budget as Part 1: one in five answers. Below, the percentage shows how many of each setup’s mismatches we caught. A mismatch means its answer did not match the dataset’s answer key within our small numerical allowance.
| Model setup | Mismatches caught · combined | Difference per 100 reviews Verification − confidence (95% CI) | Better strategy | ||
|---|---|---|---|---|---|
| Confidence | Verification | FinQA | TAT-QA | ||
| GPT-5.4 mini215 total mismatches | 29.3%63 of 215 | 51.2%110 of 215 | +27.5(+14.6, +46.0) | +22.7(+10.3, +35.4) | Verification |
| GPT-6.1 Sol115 total mismatches | 50.4%58 of 115 | 17.4%20 of 115 | −27.5(−38.0, −14.1) | −13.4(−22.7, −5.0) | Confidence |
| Claude Sonnet 5.5183 total mismatches | 47.5%87 of 183 | 56.3%103 of 183 | +4.4(−7.5, +17.3) | +12.4(−1.0, +24.7) | No clear winner |
| Gemini 4 Argon103 total mismatches | 58.3%60 of 103 | 22.3%23 of 103 | −25.3(−34.2, −14.4) | −14.4(−24.5, −8.0) | Confidence |
Reading the table. Each setup has 188 answers selected for review. The first two columns show the percentage of all mismatches caught (recall), with counts underneath, summed across datasets. The FinQA and TAT-QA columns show the strategy difference within each dataset, with separate 95% confidence intervals. Positive favors verification; negative favors confidence. An interval spanning zero leaves the lead uncertain. “Better strategy” names a lead supported on both datasets at this review budget.
Positive: verificationNegative: confidenceNo combined confidence interval is reported.
These totals add the original FinQA and TAT-QA review queues. The direction agrees across both datasets for Mini, Sol and Gemini. Sonnet’s higher verification count was not a clear lead in either dataset’s uncertainty analysis. Separate results and uncertainty.
Why do the totals differ? Each setup produced different answers and made a different number of mistakes. We compare the two strategies on the same answers within each setup, not Mini’s raw count against Sol’s. The percentages show the share of that setup’s mistakes found.
2. Does the result change when we review more?
As in Part 1, we tested 5%, 10%, 20% and 50% review capacity. Mini continued to favor verification; Sol and Gemini favored confidence. Sonnet was less consistent.
The practical result is also visible in the answers left outside review. Selecting more wrong answers for checking leaves a more accurate remaining batch.
How accurate were the answers left outside review?
3. Our checks missed most of Sol’s and Gemini’s wrong answers
Our verification rules look for specific problems: a calculation that does not agree with the stated answer, a cited figure that cannot be found at its source location, or an input that the calculation does not use.
For Sol on FinQA, these rules identified a problem in only 2 of its 73 mismatches. The other 71 did not trigger those checks. For Gemini, only 4 of 67 triggered them.
How can a wrong answer pass? An AI could use last year’s revenue instead of this year’s and calculate correctly. The figure is in the report and the arithmetic works, but it answers the wrong question. This is an illustration of what our checks can miss, not a confirmed explanation of these errors.
Confidence was more useful than these particular checks for Sol and Gemini. That gives us a concrete next step: improve what verification checks for, then test it again. Warning coverage and missed cases.
A better verification strategy is the next question
The next step is to inspect the missed cases and test checks for whether the chosen figures, dates and calculation actually answer the question. We can also test whether combining confidence with verification helps, especially when the checker finds no problem. Any new rule should be evaluated on fresh held-out questions.
This does not yet show that confidence gets better as models get smarter. Different error types and reasoning settings could contribute, and Sonnet did not show a clear winner. Related work on confidence and ranking.
The takeaway
Part 1 showed that verification could help. Part 2 shows that this particular verification strategy does not work equally well for every AI setup.
When you change the AI in your workflow, test how you choose its answers for review too.
Methods and supporting material
Appendix
Questions, configurations and scoring
The pilot began on ; the larger study began on October 18. This article was published on October 21.
The final set was selected separately from development questions and screened against recorded prior exposure, including known earlier-study material. FinQA questions came from a screened training-split pool; Part 1 used test-split questions. This is freshness relative to our study records, not a guarantee against model pretraining exposure.
We collected 3,772 responses. Of 1,000 selected questions, 57 could not be processed under our preparation rules, including interface and mapping limitations. They were not replaced. One further question was excluded from all four matched sets after a Sonnet response was truncated. The matched set contains 942 questions and 3,768 model answers. Missingness is not assumed random.
Configurations: GPT-5.4 mini, low reasoning; GPT-6.1 Sol, Claude Sonnet 5.5 and Gemini 4 Argon, medium reasoning. Reasoning labels are not an equalized compute budget. We did not vary effort within models.
Confidence was elicited in the answer as a probability from 0 to 1, not derived from token probabilities. Both strategies first include the same automatic-review cases: responses that decline to answer or whose format, calculation or units fail the usability checks. These use part of the same budget, not extra slots. Where numerical answers remain gradable, they contribute to match or mismatch totals. Abstentions and ungradable failures remain separate.
Scoring allows an absolute difference of 0.005 in each question’s preassigned output coordinate, with exact agreement for counts. For percentages this means 0.005 percentage points. Unit and scale conversions precede comparison. This is not 0.5% relative error or a threshold for financial harm.
Neither policy uses the answer key when ranking. Verification prioritizes arithmetic, evidence and unused-input warnings, then the fixed seeded tie order. An arithmetic warning means the recalculated result differs from the stated answer beyond the fixed allowance. An evidence warning means a cited number cannot be located at the cited source location. An unused-input warning means a supplied input was not used in the calculation. A warning is generated by our rule-based checker, not by the answer key or another model. The prompt, checker, scoring and ranking were frozen before final generation.
Uncertainty and interpretation
| Model | Total mismatches | Confidence caught | Verification caught |
|---|---|---|---|
| GPT-5.4 mini | 127 | 39 | 64 |
| GPT-6.1 Sol | 73 | 38 | 13 |
| Claude Sonnet 5.5 | 100 | 48 | 52 |
| Gemini 4 Argon | 67 | 36 | 13 |
| Model | Total mismatches | Confidence caught | Verification caught |
|---|---|---|---|
| GPT-5.4 mini | 88 | 24 | 46 |
| GPT-6.1 Sol | 42 | 20 | 7 |
| Claude Sonnet 5.5 | 83 | 39 | 51 |
| Gemini 4 Argon | 36 | 24 | 10 |
The chart reports verification minus confidence: additional mismatches per 100 review slots. Positive favors verification; negative favors confidence. Each line is a separate 95% pointwise interval from 10,000 paired resamples of source groups. It is not simultaneous coverage across eight comparisons or a causal test of model differences. Groups are not guaranteed independent company reports.
Sonnet’s intervals cross zero on both datasets. Mini’s favor verification; Sol’s and Gemini’s favor confidence. We also checked 1,000 tie-order seeds. The plot uses interval endpoints rounded to one decimal; the saved analysis retains full precision.
Public benchmark training exposure cannot be ruled out. Results concern these numerical tasks and this particular checker. No people corrected answers, and we did not measure labor savings or financial impact.
All review capacities, precision and recall
The main table and plots combine the two original dataset-specific queues by adding counts. They do not rerank a pooled dataset or introduce a new combined significance test. At 5%, 10%, 20% and 50%, the combined review-slot totals are 46, 93, 188 and 470 out of 942 answers. Recall divides combined catches by combined mismatches. Remaining-answer accuracy divides combined matches outside review by all answers outside review, including any ungradable answers. We weight by the actual remaining counts, not an unweighted average of dataset percentages.
| Setup / review capacity | Mismatches caught | Precision | Recall | Accuracy outside review |
|---|---|---|---|---|
| GPT-5.4 mini / 5% | 24 / 39 | 52.2% / 84.8% | 11.2% / 18.1% | 78.7% / 80.4% |
| GPT-5.4 mini / 10% | 40 / 80 | 43.0% / 86.0% | 18.6% / 37.2% | 79.4% / 84.1% |
| GPT-5.4 mini / 20% | 63 / 110 | 33.5% / 58.5% | 29.3% / 51.2% | 79.8% / 86.1% |
| GPT-5.4 mini / 50% | 117 / 147 | 24.9% / 31.3% | 54.4% / 68.4% | 79.2% / 85.6% |
| GPT-6.1 Sol / 5% | 23 / 6 | 50.0% / 13.0% | 20.0% / 5.2% | 89.7% / 87.8% |
| GPT-6.1 Sol / 10% | 36 / 11 | 38.7% / 11.8% | 31.3% / 9.6% | 90.7% / 87.8% |
| GPT-6.1 Sol / 20% | 58 / 20 | 30.9% / 10.6% | 50.4% / 17.4% | 92.4% / 87.4% |
| GPT-6.1 Sol / 50% | 86 / 58 | 18.3% / 12.3% | 74.8% / 50.4% | 93.9% / 87.9% |
| Claude Sonnet 5.5 / 5% | 26 / 37 | 56.5% / 80.4% | 14.2% / 20.2% | 82.5% / 83.7% |
| Claude Sonnet 5.5 / 10% | 50 / 78 | 53.8% / 83.9% | 27.3% / 42.6% | 84.3% / 87.6% |
| Claude Sonnet 5.5 / 20% | 87 / 103 | 46.3% / 54.8% | 47.5% / 56.3% | 87.3% / 89.4% |
| Claude Sonnet 5.5 / 50% | 132 / 126 | 28.1% / 26.8% | 72.1% / 68.9% | 89.2% / 87.9% |
| Gemini 4 Argon / 5% | 22 / 7 | 47.8% / 15.2% | 21.4% / 6.8% | 91.0% / 89.3% |
| Gemini 4 Argon / 10% | 37 / 11 | 39.8% / 11.8% | 35.9% / 10.7% | 92.2% / 89.2% |
| Gemini 4 Argon / 20% | 60 / 23 | 31.9% / 12.2% | 58.3% / 22.3% | 94.3% / 89.4% |
| Gemini 4 Argon / 50% | 82 / 54 | 17.4% / 11.5% | 79.6% / 52.4% | 95.6% / 89.6% |
Each pair below is confidence / verification mismatches caught. Capacities are rounded down. FinQA slots: 22,45,91,227. TAT-QA slots: 24,48,97,243. These are fixed-seed descriptive counts; 20% was the primary budget.
| Dataset / model | 5% | 10% | 20% | 50% |
|---|---|---|---|---|
| FinQA / GPT-5.4 mini | 12 / 18 | 23 / 40 | 39 / 64 | 72 / 88 |
| FinQA / GPT-6.1 Sol | 12 / 3 | 21 / 7 | 38 / 13 | 57 / 38 |
| FinQA / Claude Sonnet 5.5 | 11 / 17 | 24 / 34 | 48 / 52 | 73 / 65 |
| FinQA / Gemini 4 Argon | 11 / 4 | 21 / 8 | 36 / 13 | 53 / 36 |
| TAT-QA / GPT-5.4 mini | 12 / 21 | 17 / 40 | 24 / 46 | 45 / 59 |
| TAT-QA / GPT-6.1 Sol | 11 / 3 | 15 / 4 | 20 / 7 | 29 / 20 |
| TAT-QA / Claude Sonnet 5.5 | 15 / 20 | 26 / 44 | 39 / 51 | 59 / 61 |
| TAT-QA / Gemini 4 Argon | 11 / 3 | 16 / 3 | 24 / 10 | 29 / 18 |
Precision is mismatches caught divided by all review slots. Recall is mismatches caught divided by the total mismatches for that setup and dataset. Pairs below are confidence / verification at 20%.
| Dataset / model | Precision | Recall |
|---|---|---|
| FinQA / GPT-5.4 mini | 42.9% / 70.3% | 30.7% / 50.4% |
| FinQA / GPT-6.1 Sol | 41.8% / 14.3% | 52.1% / 17.8% |
| FinQA / Claude Sonnet 5.5 | 52.7% / 57.1% | 48.0% / 52.0% |
| FinQA / Gemini 4 Argon | 39.6% / 14.3% | 53.7% / 19.4% |
| TAT-QA / GPT-5.4 mini | 24.7% / 47.4% | 27.3% / 52.3% |
| TAT-QA / GPT-6.1 Sol | 20.6% / 7.2% | 47.6% / 16.7% |
| TAT-QA / Claude Sonnet 5.5 | 40.2% / 52.6% | 47.0% / 61.4% |
| TAT-QA / Gemini 4 Argon | 24.7% / 10.3% | 66.7% / 27.8% |
A random baseline with the same review budget
The baseline first includes the same automatic-review cases as both strategies, then selects uniformly at random from the remaining answers to fill the remaining slots. Expected mismatches caught = mismatches in automatic review + remaining slots × the mismatch share among remaining answers. This exact expectation was saved in the original analysis; it is not one randomly sampled queue.
| Dataset / model | Confidence | Verification | Random expectation |
|---|---|---|---|
| FinQA / GPT-5.4 mini | 39 | 64 | 26.3 |
| FinQA / GPT-6.1 Sol | 38 | 13 | 14.1 |
| FinQA / Claude Sonnet 5.5 | 48 | 52 | 19.9 |
| FinQA / Gemini 4 Argon | 36 | 13 | 12.9 |
| TAT-QA / GPT-5.4 mini | 24 | 46 | 19.2 |
| TAT-QA / GPT-6.1 Sol | 20 | 7 | 8.3 |
| TAT-QA / Claude Sonnet 5.5 | 39 | 51 | 17.5 |
| TAT-QA / Gemini 4 Argon | 24 | 10 | 7.1 |
Expected counts are rounded to one decimal and can be fractional. Near or below an expectation does not establish equivalence or a statistically supported disadvantage. The uncertainty intervals elsewhere compare verification with confidence, not with this baseline.
Warning coverage, ranking and false alarms
This is an additional post-study descriptive analysis of the same saved answers. It changes no primary result, scoring rule, warning threshold or queue. “Warned” means a usable response with at least one arithmetic, evidence or unused-input warning. Mandatory unusable cases are separate; warning types can overlap.
| Dataset / model | Warned mismatches / all | Warned matches | Warned mismatches selected |
|---|---|---|---|
| FinQA / GPT-5.4 mini | 68 / 127 | 38 | 61 / 68 |
| FinQA / GPT-6.1 Sol | 2 / 73 | 5 | 2 / 2 |
| FinQA / Claude Sonnet 5.5 | 47 / 100 | 24 | 47 / 47 |
| FinQA / Gemini 4 Argon | 4 / 67 | 14 | 4 / 4 |
| TAT-QA / GPT-5.4 mini | 45 / 88 | 73 | 43 / 45 |
| TAT-QA / GPT-6.1 Sol | 3 / 42 | 23 | 3 / 3 |
| TAT-QA / Claude Sonnet 5.5 | 47 / 83 | 24 | 47 / 47 |
| TAT-QA / Gemini 4 Argon | 3 / 36 | 21 | 3 / 3 |
All warned Sol and Gemini responses, including both matches and mismatches, fit into their 20% queues. The gap is low warning coverage, not exclusion of warned cases at this budget. Warnings on matches are false alarms only relative to numerical scoring; they do not prove that the reasoning or citations are sound. The diagnostic does not identify the causes of unflagged errors or prove a confidence-verification combination will help.
Cost and reproducibility
Total external spending, including the pilot, was $23.06, with no unresolved charges. The saved-answer diagnostic added $0. Subscription and human effort are excluded.
The original analysis and new diagnostic were independently checked. Frozen answers and scientific rules remain unchanged. Download dataset results · Download combined plot data · Download warning diagnostic · Diagnostic code.
Download the code and data package, including a portable diagnostic script and its de-identified input, the original saved estimates and intervals, and code for the article’s combined figures. Reproduction instructions and scope. The package reproduces the warning diagnostic and displayed summaries; it does not regenerate model answers or rerun the original bootstrap analysis. No prompts, source reports, raw model responses or provider credentials are included.