Code needs checking.
AI-generated code can contain mistakes. A coding agent can check its work using a test runner, a tool that runs the code and checks whether it behaves as expected.
The tool returns a report: the tests passed, failed or could not run. The agent uses that report to decide whether to fix the code or finish.
But what if the report is wrong? Real-world examples →
Would a warning help, or should the workflow check again?
We started by comparing three ways a coding agent could respond to a test report:
- Baseline: let the agent decide whether to revise its code, check again or finish.
- Warning: give it the same choices, but add a reminder that tool reports may be unreliable.
- Verification: automatically test its first solution again and give it any differing result before its next decision.
But we didn’t want to stop there. Would these workflows behave differently when the report said “passed” rather than “the test could not run”? To find out, we also varied what the first report told the agent:
- Clean: the actual test result, whether pass or fail.
- Testing-tool error: a message saying no test result was available.
- Misleading success: a “passed” report, even when the code failed.
- Stale report: an old result from a placeholder solution, rather than the current code.
We deliberately controlled these first reports; later reports were truthful. This let us ask whether a warning or an automatic check helped more with some kinds of feedback than others.
Each workflow-and-report combination had 60 function-writing tasks. We used 240 distinct tasks, randomly split into four report groups. Each task was attempted under all three workflows with the same assigned report condition, giving 720 recorded outcomes.
The tasks came from BigCodeBench-Instruct v0.1.4, using GPT-6 Luna with medium reasoning. We judged the final code against the unaltered official tests, whose source was hidden from the agent. Success meant passing every test. Full methods →
Same code. Different evidence.
Imagine an addition function that returns 6 for 2 + 3. The code is wrong, but the agent’s next move depends on what its report says and whether the workflow checks again. Change either selector below to follow the difference.
Automatic verification runs once on the first solution, regardless of its report. It uses the same code and tests in a fresh process. The new result is truthful in this experiment; matching reports are not shown twice.
One task, four reports, three workflows
A fresh check gives the agent more evidence. Our question was how far a warning gets us compared with that check, and which kinds of reports make the biggest difference.
The warning stayed close to baseline.
Compare the three workflows within each report condition. Each bar shows how often the final code passed all tests.
What changed when the first report changed?
Each bar shows final success on 60 tasks. Tap a bar, or use the buttons below, for counts.
All conditions and exact counts
- The warning changed little. Its results stayed close to baseline in all four groups. With misleading success, it solved 33/60 tasks versus baseline’s 31/60; the paired interval did not establish a clear improvement.
- Verification helped most after false reassurance. It solved 53/60 tasks under misleading success, 22 more than baseline.
- An obvious error looked less damaging than a false “passed.” Baseline solved 49/60 tasks after a testing-tool error, versus 31/60 after misleading success. The groups were randomly assigned, but contain different tasks; chance imbalance remains possible.
“No result” may encourage another check, while “passed” may suggest the job is done. That is a plausible explanation, not proof of the agent’s reasoning. Recovery counts →
All 720 outcomes were recorded after two operational recoveries; this is an amended completion. Collection history → · Review and evidence →
The benefit depended on the report.
The chart shows the differences between workflows and the uncertainty around them. An interval crossing zero leaves room for either improvement or decline.
Where did the workflows differ?
Dots are differences; lines are intervals. Zero means no observed difference. Read exact values in the table below.
Exact differences, intervals and paired denominators
- Overall: baseline solved 185/240 tasks, warning 187/240 and verification 212/240. Of verification’s 27 net additional successes over baseline, 22 came from misleading success.
- Elsewhere, gains were smaller and uncertain. Verification finished two tasks ahead of baseline with clean reports and five ahead after a testing-tool error; both intervals touched zero. With stale reports it finished two behind, with an interval spanning improvement and decline.
Each condition has equal weight here. That does not reflect how often these errors occur in real products. The 95% intervals are unadjusted for multiple comparisons.
What did the extra check cost?
Across 240 tasks, verification added about 2 cents in model charges and 11.2 minutes of recorded testing time compared with baseline.
What each workflow cost
The extra check runs the testing tool, not another AI call. The workflows shared the same maximum AI allowance, but used different numbers of calls. Verification used fewer calls overall, but more total input and output tokens, and its model bill was higher.
These are partial costs. Actual infrastructure invoices and human/Codex costs remain unknown or unmeasured. Full accounting →
A warning alone did little here.
Telling the agent that a report might be wrong barely changed the results. Giving it a fresh, truthful test result helped more, especially when the first report falsely said “passed.” In this experiment, building a check into the workflow worked better than relying on the warning alone.
What this does not tell us:
- Whether every warning would perform this way. We tested one warning with one AI model. Different wording or models could behave differently.
- Whether a second check would be reliable in a real system. Ours was truthful by design. A real tool could repeat the same error, so simply running it twice may not help.
- Whether always checking again is worth the cost. We supplied extra testing and reliable feedback. We did not compare cheaper checking policies or measure how often these report errors happen in real use.
A useful next test: compare checking every first solution with checking only when its report says “passed.” Because most of the benefit here came from misleading success, that comparison could show whether fewer rechecks retain the benefit. We have not tested that yet.
For now, the lesson is to test whether your workflow can recover from a wrong report. Adding a warning alone did not provide much protection in this study. Full limitations →
Appendix · Optional details
You can stop reading here. The sections below contain the methods, all comparisons, accounting and review evidence.
Each section stays collapsed until selected. Links from the article open the relevant section; return links take you back.
A. Design, limits and outcome definitions
⌄
BigCodeBench-Instruct v0.1.4; GPT-6 Luna, medium reasoning, OpenAI-only/default service. Each task–workflow assignment has one run, with at most four total model calls including finish/rerun/revise decisions. Each call permits 8,192 inclusive output tokens and a 180-second timeout. The first verification in C is a fresh test execution, not another model call.
240 tasks are divided among four assigned conditions, 60 each. The same condition applies to all three workflows for a task. Final all-hidden-tests success is measured against all assigned tasks; ordinary failures stay in the denominator. The separate 40-task pilot is not pooled. No task replacement or missing-outcome imputation is used.
Misleading success changes the verdict only when the first candidate actually fails. A cached report remains stale even if its verdict agrees with the fresh result. Thus assigned condition and realized misleading exposure are different quantities.
All 720 assignments reached a recorded outcome and contained code. Completion did not mean correctness: the final reasons were 584 all-tests passes, 127 test failures, seven import errors and two timeouts. Syntax was verified for 718 candidates, zero were classified syntax-invalid and two retained unknown syntax after timeout grades. There were no missing final outcomes to exclude or impute.
← Return to what we testedB. All 15 paired comparisons and intervals
⌄
The difference and interval are in percentage points. Assigned and observed paired denominators are equal throughout. These are the frozen descriptive 95% intervals, unadjusted for multiplicity. The chart does not calculate new intervals.
Each interval uses 20,000 task-paired bootstrap draws and the frozen nearest-rank percentile endpoints. Overall resampling is stratified within assigned condition. Keeping task pairs together preserves the within-task comparison. These intervals do not establish generalization to other models or real-world error frequencies.
← Return to the paired-difference chartC. First failures that became final successes
⌄
In misleading success, the transition counts are baseline 0/29, warning 0/27 and verification 20/27. Across all conditions, they are 55/110, 59/112 and 76/104 respectively.
These describe each workflow’s own initially failing candidates. Selecting that subset can change which tasks are included, so it is not the primary paired comparison or an estimate of a causal recovery rate. The assigned 60-task condition denominators and 240-task overall denominators remain primary. Final success does not prove awareness of the manipulated report.
← Return to the resultsD. Why this is an amended completion
⌄
The original main collection stopped at 576 observed outcomes when retrieval of a completed grading result failed. The first continuation added 52 observations before a billing-receipt issue involving a saved response stopped it at 628. Separately reviewed engineering repairs completed the remaining 92 while preserving all 628 earlier observations exactly.
The B/131 candidate received one final-only recovery grade in the first continuation, with no new model call. Its recovery agreed with the old command summary and was not repeated. C/533 consumed its exact saved first response once after the late receipt was reconciled; the original model request was not repeated and its charge was counted once. The saved response counted toward the original four-call limit.
The original incomplete report remains the historical primary report. Operational recovery and retrieval policies changed after earlier outcomes were known. The combined analysis is therefore a post-HOLD amended completion, not an uninterrupted preregistered experiment. All historical claims, failures and evidence remain preserved.
← Return to the first resultsE. Calls, tokens, time and financial accounting
⌄
All 2,049 main calls have known usage and reconciled charges. Total input is 1,510,313 tokens; output is 893,894, including 636,580 reasoning tokens. Reasoning must not be added again. The saved evidence includes 2,385 grades and one historical ambiguous evaluation intent. Observed grader time totals 7,891.29809 seconds; ambiguous durations are unknown.
| Quantity | Amount |
|---|---|
| Original main API | 0.469631345 |
| First continuation API, including late receipt | 0.037691825 |
| Final completion API | 0.072240875 |
| Main API total | 0.579564045 |
| Separate pilot API | 0.091279770 |
| Combined known API | 0.670843815 |
| Retained infrastructure reserve | 38.000000000 |
| Historical buffer | 0.050000000 |
| Conservative commitment / cap | 38.720843815 / 50 |
No pending API charge reserve remains at completion. The historical $0.05 buffer is not another unresolved charge. All launched resources have closure evidence. Actual infrastructure invoices, original host10 usage and human/Codex costs remain unknown or unmeasured. Provider latency, grader time and end-to-end runtime are different measures.
← Return to resource tradeoffsF. Evidence, downloads and computational review
⌄
The independent computational review accepted the study with no material objections. It reproduced all 15 paired differences and interval endpoints, checked preservation of prior outcomes, and reconciled calls, costs and resource closure. The original report’s “review pending” wording is preserved; the separate acceptance closes that step.
- Research code ↗. Historical code archive; private inputs and runtime evidence are not bundled.
- Download the chart data (JSON) ↓
- Download all 15 comparisons (CSV) ↓
The downloads contain the aggregate values used in this article. The complete operational evidence archive is retained separately and is not bundled with the site. Download the review summary and source hashes ↓. Charts and selectors change the view, not the evidence.
← Return to the resultsG. Can a real system show misleading success?
⌄
Yes, a downstream success signal can differ from what happened during testing. These examples are related mechanisms, not measurements of how often our experimental condition occurs.
- A reported tool bug: Jest issue #13233 describes version 29.0.2 returning a success exit code after tests failed to run under a particular coverage configuration. It is a historical report with reproduction steps, closed as not planned; it is not evidence that current Jest routinely behaves this way.
- A documented workflow setting: GitHub Actions documentation explains that a step allowed to continue on error can have an underlying failure outcome but a final success conclusion. This is intentional behavior. Treating that final conclusion as proof that tests passed would be a mistake.
Neither example is a replay of our experiment. We deliberately altered the first report and kept subsequent reports truthful. Repeating a real check would not necessarily remove the underlying reporting problem.
← Return to the research questionH. Scope and limitations
⌄
- Not every warning: we tested one wording and one model configuration.
- Not every verification policy: we did not compare always rechecking against checking only after a pass.
- Not every real-world failure: the recheck was truthful by design. A real system could repeat the same reporting fault.
This was a public coding benchmark with deliberately altered reports. It does not measure real-world error frequency, rule out training exposure or show that the agent understood the manipulation.
Technical interruptions also required changes to recovery procedures after earlier results were available. This is an amended completion, with the original incomplete report retained. Independent computational review checked the results and preserved evidence; it is not external academic peer review. Recovery history →
All workflows had the same four-call model ceiling, but verification supplied extra truthful evidence and testing work. This was not an equal-total-compute comparison. Random assignment makes condition groups comparable in expectation; the frozen paired intervals compare workflows within conditions, not the cross-condition error-versus-false-success gap.
The stale-report condition used an old test report from a placeholder solution. It does not establish robustness to stale files or stale data generally.
← Return to the takeawayI. Related research on feedback and self-correction
⌄
These papers help explain why the source and quality of feedback matter. None directly tests our three workflows across the four first-report conditions, and none establishes that warnings generally fail.
Better feedback can improve code repair
Olausson et al., Is Self-Repair a Silver Bullet for Code Generation? (ICLR 2024) studied Code Llama, GPT-3.5 and GPT-4 on HumanEval and APPS. Gains from self-repair were often modest or absent once repair costs were considered. Supplying feedback from a stronger model produced larger gains.
Connection to our study: both suggest that the feedback available during repair matters. Their intervention improved feedback from a critic; ours supplied a fresh test result after a potentially misleading report. The effects and costs are not directly comparable.
Reconsidering an answer is not always enough
Huang et al., Large Language Models Cannot Self-Correct Reasoning Yet (ICLR 2024) examined correction without external feedback. In the reasoning settings they studied, models struggled to improve their answers and sometimes made them worse.
Connection to our study: it supports caution about expecting a model to correct itself merely because it is asked to reconsider. It did not test our warning or coding workflow, and its findings should not be read as a universal claim about current models.
Feedback can help agents improve on later attempts
Shinn et al., Reflexion: Language Agents with Verbal Reinforcement Learning (2023) used task feedback to generate reflections that agents retained for subsequent trials. The authors reported improvements across tasks including coding.
Connection to our study: both use feedback within an agent workflow. Reflexion adds reflection and memory across attempts; our intervention was one automatic test rerun. It is a related approach, not evidence for our exact result.
← Return to the research contextPublished November 1, 2026 · Amended completion. Charts and filters change the view, not the evidence.