More correct answers. Should we switch?
Imagine your team uses an AI assistant to answer customer questions. You change its instructions and test both versions on the same 100 questions.
The current version answers 70 correctly; the new version answers 75. That is a five-percentage-point improvement on this test. Is it enough evidence to switch?
The new version could be better, or its lead could be due to chance. We need enough evidence to avoid an unnecessary switch without delaying a useful improvement.
Our research asks: when does an apparent improvement provide enough evidence to switch? We examined both the mistakes a decision rule can make and the useful improvements it can leave unconfirmed.
Three reasons to look closer
- Related questions: Five questions about one refund policy may share the same failure mode. Treating them as independent evidence can overstate certainty.
- Repeated checking: Inspecting results after every batch and stopping at the first apparent win creates more opportunities to act on chance.
- Missing results: If the new version’s unsuccessful outcomes are more likely to lose their recorded grades, its remaining results look better than they should.
Study design: simulate a known difference
We simulated evaluation results with a known underlying performance difference. This let us measure whether an analysis reached the right conclusion, rather than infer which version was truly better from the same test we were evaluating.
A represented the current version and B the new version. We defined three performance scenarios:
- B is worse: its success rate is 4 percentage points lower than A’s.
- B has a small advantage: its success rate is 2 points higher.
- B has a substantial advantage: its success rate is 10 points higher.
Each simulated test contained correct or incorrect outcomes for A and B on 1,000 questions, arranged in 200 groups of five. We also varied the available records, including whether all grades were present or some were missing.
Each comparison used 2,000 simulated tests. Within a setting, the underlying performance difference stayed fixed while the recorded results varied by chance. To isolate the effect of a changed assumption, comparisons across settings reused outcomes or random draws where possible.
We analyzed every test six ways. All six approaches received the same recorded results; only the way they used that evidence to make a decision changed. We then counted how often each approach recommended switching.
- Approach 1: One check. Treat every question as independent evidence. Compare A and B once, after all 1,000 questions.
- Approach 2: Five checks. Use the same calculation as Approach 1, but check after 200, 400, 600, 800 and 1,000 questions. Make no adjustment for these extra opportunities to decide.
- Approach 3: Related questions, one check. Compare A and B once at the end, as in Approach 1, but account for questions that share a policy, document or other context when calculating uncertainty.
- Approach 4: Related questions, five checks. Account for related questions, as in Approach 3, and check at the same five points as Approach 2. Do not adjust for repeated checking.
- Approach 5: Related questions, five adjusted checks. Start with Approach 4, but require stronger evidence at each check to account for the five opportunities to decide.
- Approach 6: Adjusted checks plus record checks. Start with Approach 5, remove identical duplicate records, and withhold a decision at any checkpoint with missing or invalid records.
We also checked how the findings changed when grades went missing less often, questions were more or less related, and the improvement required to switch was smaller or larger. This let us see how much each result depended on our assumptions.
Technical details: simulation design and data conditions · duplicate-record stress test · reproducibility checks.
What counted as a reason to switch?
Our main comparisons use an illustrative requirement: B must improve success by more than 2 percentage points. Each analysis estimates the improvement and a confidence interval, a range expressing uncertainty around that estimate. Each simulated test gets its own interval; the 2,000 repetitions measure how often the resulting decision is right or wrong.
- Switch: the entire interval is above +2 points.
- Gain below the requirement: the entire interval is below +2 points.
- Not enough evidence: the interval includes or touches +2 points.
See how one test’s confidence interval is calculated and the formal interval and decision rules.
A single-check approach waited for all 1,000 questions. A five-check approach examined cumulative results after 200, 400, 600, 800 and 1,000 questions, retaining the first conclusive decision. Each check includes more questions. These checks do not generate new attempts at previous questions.
Adjusted checks required stronger evidence to account for the five decision opportunities. Unadjusted checks reused the single-check standard each time. The adjustment method is explained in the appendix. A separate record check required an outcome for both versions on every question expected at that checkpoint; a missing grade is not a recorded failure.
The three scenarios we examine
A small gain, complete resultsB is +2 points better; switching requires more than +2. Can the analysis avoid an unjustified switch?
A worse version, missing gradesB’s true gain is −4 points; its unsuccessful outcomes are more likely to disappear. Can the analysis resist the misleading results?
A large gain, complete resultsB is +10 points better. Can the analysis confirm a worthwhile improvement?
All three include related questions. The figures show how often each approach recommends switching across 2,000 simulated tests per scenario. They do not show answer accuracy.
1. Grouping and repeated checking need separate safeguards
B’s true advantage is exactly +2 points, so it does not exceed our switching requirement. Records are complete.
- Approach 1 → Approach 3: accounting for related questions, while keeping one final check, reduced mistaken switches from 6.15% to 2.55% of the 2,000 tests.
- Approach 3 → Approach 4: checking five times without an adjustment raised mistaken switches to 7.20%, despite accounting for related questions.
- Approach 4 → Approach 5: adjusting those five checks reduced mistaken switches from 7.20% to 1.75%.
Two safeguards, two sources of false confidence
True gain: +2 points · Complete records · Lower is better: switching is not justified under the greater-than-2 rule.
| Approach | Tests recommending a switch |
|---|---|
| Approach 1: One checkTreat questions as independent | 6.15% |
| Approach 2: Five checksTreat questions as independent; no adjustment | 17.05%Most unjustified switches |
| Approach 3: Related questions, one checkAccount for groups; decide at the end | 2.55% |
| Approach 4: Related questions, five checksAccount for groups; no repeated-check adjustment | 7.20% |
| Approach 5: Related questions, five adjusted checksAccount for groups; stronger evidence at each check | 1.75%Fewest unjustified switches · tied |
| Approach 6: Adjusted checks plus record checksAlso check for duplicate, missing or invalid records | 1.75%Fewest unjustified switches · tied |
Approach 2 had the most unjustified switches (17.05%); Approaches 5 and 6 tied for the fewest (1.75%). When checking repeatedly, accounting for related questions alone was not enough; the extra decision opportunities also needed an adjustment. Uncertainty in these comparisons →
How much does relatedness matter? Across our weaker-to-stronger relatedness settings, Approach 2’s mistaken-switch rate rose from 8.30% to 25.50%. Approach 3 stayed around 2–3%, and Approach 5 around 1–2%. The small fluctuations in those latter rates do not establish meaningful differences.
In these tests, treating related questions as independent produced more mistaken switches as their results became more related. Accounting for groups remained useful across the range we tested. Relatedness settings and all six approaches →
2. Missing failures can reverse the conclusion
Now B is truly four points worse. The question is whether missing grades can make it look better.
When successful and unsuccessful answers each had a 5% chance of losing their grades, Approach 1 still estimated an average four-point disadvantage for B. It made no mistaken switches in these 2,000 tests. Missing data reduced the evidence available, but this random-loss example did not systematically favor B.
We then made failed answers more likely to lose their grades, keeping the loss rate for successful answers at 5%. The table shows how often each approach recommended switching to the worse version.
The risk changes with which grades go missing
B is truly four points worse. Successful grades have a 5% chance of going missing; the rows vary that probability for failures. Switching requires more than a two-point gain.
| Failure grades missing | Approach 1One check; ignore groups | Approach 3One check; account for groups | Approach 5Groups; five adjusted checks |
|---|---|---|---|
| 5%Equal loss for failures and successes | 0.00% | 0.00% | 0.00% |
| 20% | 1.65% | 0.45% | 0.55% |
| 30% | 15.10% | 2.80% | 2.10% |
| 40% | 55.05% | 17.90% | 11.65% |
At the severe 40%/5% setting, Approach 1 estimated an average +7.34-point advantage for B, reversing the true −4-point difference. Approach 5 still made mistaken switches in 11.65% of tests. Stronger evidence requirements reduced the number of switches; they did not restore the missing information.
These probabilities are stress-test assumptions, not measured product failure rates. With 10% of successful grades missing instead of 5%, the rates changed again. There is no universal “safe” missingness percentage in these results. All missingness settings and uncertainty →
How the six approaches handle the severe missing-data case
True gain: −4 points · Failed grades missing: 40%; successful grades missing: 5% · Every switch is incorrect.
| Approach | Tests recommending a switch |
|---|---|
| Approach 1: One checkTreat questions as independent | 55.05% |
| Approach 2: Five checksTreat questions as independent; no adjustment | 66.05%Most unjustified switches |
| Approach 3: Related questions, one checkAccount for groups; decide at the end | 17.90% |
| Approach 4: Related questions, five checksAccount for groups; no repeated-check adjustment | 28.90% |
| Approach 5: Related questions, five adjusted checksAccount for groups; stronger evidence at each check | 11.65%Fewest false switches among Approaches 1–5 |
| Approach 6: Adjusted checks plus record checksAlso check for duplicate, missing or invalid records | 0.00%Zero switches · all tests unresolved |
Approach 6 leaves all 2,000 tests unresolved. It does not identify which version is better.
Approach 6 left every test unresolved in all the settings with missing grades, including the equal 5%/5% case. It avoided mistaken switches by withholding every decision. That does not make its complete-record rule suitable for every incomplete evaluation.
Investigate why results are missing, not just how many. Selective loss can make a worse version look better, and a more cautious calculation cannot repair that evidence.
3. Stronger safeguards can delay genuine improvements
With complete records and a true +10-point gain, we want the analysis to recognize that B clears the switching requirement.
Approach 2 detected this large improvement most often: it recommended switching in 93.45% of the 2,000 tests. But in the earlier +2-point scenario, where the gain did not exceed our requirement, it also recommended switching in 17.05% of tests. A high detection rate alone does not establish a better decision method.
Approaches 4 and 5 both account for related questions. Adding the repeated-check adjustment in Approach 5 reduced confirmed switches from 83.65% to 63.00%. Unresolved tests increased from 16.25% to 36.95%.
The adjustment reduced mistaken switches in the first scenario, but confirmed fewer real improvements here at the same maximum test size. Neither result alone is enough to choose an approach.
Greater caution leaves more improvements unconfirmed
True gain: +10 points · Complete records · Higher is better: switching is justified if the test establishes the gain.
| Approach | Tests recommending a switch |
|---|---|
| Approach 1: One checkTreat questions as independent | 89.95% |
| Approach 2: Five checksTreat questions as independent; no adjustment | 93.45%Most genuine improvements detected |
| Approach 3: Related questions, one checkAccount for groups; decide at the end | 78.30% |
| Approach 4: Related questions, five checksAccount for groups; no repeated-check adjustment | 83.65% |
| Approach 5: Related questions, five adjusted checksAccount for groups; stronger evidence at each check | 63.00%Fewest genuine improvements detected · tied |
| Approach 6: Adjusted checks plus record checksAlso check for duplicate, missing or invalid records | 63.00%Fewest genuine improvements detected · tied |
Approach 2 detected the most genuine improvements (93.45%); Approaches 5 and 6 tied for the fewest (63.00%). This reverses their ranking in the first scenario. Choosing an approach requires considering both mistaken switches and missed improvements. Additional estimates →
We also varied the improvement required to switch. With the same genuine ten-point gain and the same test results, Approach 5 confirmed a switch in 74.60% of tests when the requirement was more than one point, 63.00% at more than two, and 26.30% at more than five.
Requiring stronger statistical evidence and requiring a larger business improvement are different choices. Raising the improvement requirement asks the test to establish more; it does not make the statistical method better.
A genuine two-point gain makes this distinction clear: it is worthwhile under a one-point requirement, exactly at the boundary under two, and below the requirement under five. Set that requirement before interpreting which decisions count as correct. All three switching requirements →
Which approach should you use?
In a real evaluation, you do not know the true improvement. Choose the approach before seeing the results, based on how the test is designed and what a mistaken or delayed switch would cost.
- Can you wait until the test ends? For related questions like those studied here, Approach 3 is a reasonable starting point: account for the groups and make one planned decision.
- Do you need early decisions? Use an approach designed for repeated checking. Approach 5 is our tested example, not a proven best option; we did not compare it with other sequential methods.
- Are the records trustworthy? Reconcile the intended questions with the recorded grades. Investigate missing results and duplicates before comparing scores. Check whether failures could disappear more often than successes; the overall missing percentage alone cannot answer that question. Approach 6’s strict refusal rule is not a substitute for investigating the cause.
- What would a wrong decision or a delay cost? Choose a common time horizon, such as the next month. Estimate the loss from an unjustified switch and the value forgone by delaying a useful update. Set the minimum worthwhile improvement, any non-negotiable risk limits and the testing budget upfront. If the evidence is still insufficient when that budget is exhausted, report the uncertainty. Do not extend the test just to obtain a favorable result.
You may accept more decision risk for a cheap, reversible switch than for a costly deployment. The analysis should still account for related questions and report uncertainty accurately.
We have not established an overall winner. That would require the costs of mistaken switches, delayed improvements and further testing, plus how often different situations occur. Our simulation did not estimate those inputs.
Worked example: how different decision costs change the choice between Approaches 3 and 5 →
Scope and limitations
- Conditional results: Relatedness, missing grades and duplicates were simulated, not calibrated to production evaluations. Varying the assumptions shows where the conclusions are sensitive; it does not establish real-world error rates or deployment value.
- Safeguards have limits: Intervals are approximate; the complete-record rule is deliberately strict. No approach met our error-and-decision targets across every applicable baseline setting. See the baseline results and assumption checks.
- Applied demonstration, not a new method: We stress-tested established statistics, not AI products. Related reading: Adding Error Bars to Evals.
The bottom line
The assistant that answered 75 questions correctly instead of 70 may be better. To decide whether to switch, we need to know how those questions were selected, whether the records are complete, and whether the estimated gain is large enough to matter.
Choose the approach before looking at the results. Account for related questions, plan when decisions can be made, and investigate missing records. Consider both mistaken switches and useful improvements left unconfirmed. If the test cannot resolve the choice within the agreed budget, report “not enough evidence yet.”
Methods and supporting material
Appendix
The methods and tables retain all six approaches: the 15 baseline scenarios, plus the comparisons that vary missingness, relatedness and the switching requirement. Expand only the detail you need.
How the findings change with the assumptions
Each setting contains 2,000 evaluations. Missingness comparisons reuse the same underlying outcomes and logging-loss draws; threshold comparisons reuse saved baseline outcomes; relatedness comparisons reuse the same random draws under different dependence settings. These are paired comparisons, not 46,000 independent new experiments.
- Missingness: failed-grade loss of 5%, 10%, 20%, 30% or 40%, crossed with successful-grade loss of 5% or 10%, plus a complete-record reference. B is truly four points worse. The equal-loss cases are controls; 5% failure loss with 10% success loss reverses the selection direction.
- Switching requirement: greater than one, two or five points in the three scenarios discussed above. We recomputed the first conclusive decision across every eligible check, then classified it against the new requirement.
- Relatedness: h = 0.15, 0.30 or 0.45 with a true +2-point gain and complete records. Within-group correlation of paired differences is h² / (0.5 − Δ²): approximately 4.5%, 18.0% and 40.5%. This sweep does not include independence.
For Approach 1, the first tested failed-grade loss above a 10% mistaken-switch rate was 30% when successful-grade loss was 5%, but 40% when it was 10%. The preceding tested settings had rates of 1.65% and 6.95%, respectively. This brackets crossings on the tested grid; it does not identify a precise or universal safe cutoff.
Intervals below describe simulation precision, not production risk, and are pointwise rather than simultaneous. “Correct evidence-hold” at the exact +2-point boundary retains the baseline operational convention; it does not prove the effect is strictly below the requirement. The strict record check can decide on a clean earlier prefix, although all tested nonzero-missingness settings here remained unresolved.
Download the complete sensitivity table data · Code and reproduction instructions · Design and verification scope
Missing grades: 11 settings
The new version is truly four points worse. Failure-grade loss varies from 5% to 40%, and success-grade loss from 5% to 10%. The 0%/0% setting is a paired complete-record reference; equal-loss and reversed-selection controls are retained.
Missing grades: 0% of failures · 0% of successes
True gain: -4 points. Switching requires more than 2 points, so a switch is incorrect here. Approach 1’s average final estimate: -4.01 points; bias: -0.01 points.
| Approach | Incorrect switch | Correct evidence-hold | Incorrect evidence-hold | Inconclusive | Mean groups used |
|---|---|---|---|---|---|
| Approach 1: One check | 0.00%[0.00%, 0.19%] | 71.75%[69.74%, 73.68%] | 0.00%[0.00%, 0.19%] | 28.25%[26.32%, 30.26%] | 200.00 |
| Approach 2: Five checks | 1.05%[0.69%, 1.60%] | 79.45%[77.62%, 81.16%] | 0.00%[0.00%, 0.19%] | 19.50%[17.82%, 21.29%] | 115.02 |
| Approach 3: Related questions, one check | 0.00%[0.00%, 0.19%] | 53.90%[51.71%, 56.08%] | 0.00%[0.00%, 0.19%] | 46.10%[43.92%, 48.29%] | 200.00 |
| Approach 4: Related questions, five checks | 0.25%[0.11%, 0.58%] | 63.50%[61.37%, 65.58%] | 0.00%[0.00%, 0.19%] | 36.25%[34.17%, 38.38%] | 143.46 |
| Approach 5: Related questions, five adjusted checks | 0.00%[0.00%, 0.19%] | 36.65%[34.57%, 38.79%] | 0.00%[0.00%, 0.19%] | 63.35%[61.21%, 65.43%] | 172.52 |
| Approach 6: Adjusted checks plus record checks | 0.00%[0.00%, 0.19%] | 36.65%[34.57%, 38.79%] | 0.00%[0.00%, 0.19%] | 63.35%[61.21%, 65.43%] | 172.52 |
Missing grades: 5% of failures · 5% of successes
True gain: -4 points. Switching requires more than 2 points, so a switch is incorrect here. Approach 1’s average final estimate: -4 points; bias: 0 points.
| Approach | Incorrect switch | Correct evidence-hold | Incorrect evidence-hold | Inconclusive | Mean groups used |
|---|---|---|---|---|---|
| Approach 1: One check | 0.00%[0.00%, 0.19%] | 70.15%[68.11%, 72.12%] | 0.00%[0.00%, 0.19%] | 29.85%[27.88%, 31.89%] | 200.00 |
| Approach 2: Five checks | 1.05%[0.69%, 1.60%] | 77.75%[75.87%, 79.52%] | 0.00%[0.00%, 0.19%] | 21.20%[19.46%, 23.05%] | 117.68 |
| Approach 3: Related questions, one check | 0.00%[0.00%, 0.19%] | 52.00%[49.81%, 54.18%] | 0.00%[0.00%, 0.19%] | 48.00%[45.82%, 50.19%] | 200.00 |
| Approach 4: Related questions, five checks | 0.40%[0.20%, 0.79%] | 61.40%[59.25%, 63.51%] | 0.00%[0.00%, 0.19%] | 38.20%[36.10%, 40.35%] | 144.58 |
| Approach 5: Related questions, five adjusted checks | 0.00%[0.00%, 0.19%] | 35.70%[33.63%, 37.83%] | 0.00%[0.00%, 0.19%] | 64.30%[62.17%, 66.37%] | 173.68 |
| Approach 6: Adjusted checks plus record checks | 0.00%[0.00%, 0.19%] | 0.00%[0.00%, 0.19%] | 0.00%[0.00%, 0.19%] | 100.00%[99.81%, 100.00%] | 200.00 |
Missing grades: 5% of failures · 10% of successes
True gain: -4 points. Switching requires more than 2 points, so a switch is incorrect here. Approach 1’s average final estimate: -5.34 points; bias: -1.34 points.
| Approach | Incorrect switch | Correct evidence-hold | Incorrect evidence-hold | Inconclusive | Mean groups used |
|---|---|---|---|---|---|
| Approach 1: One check | 0.00%[0.00%, 0.19%] | 82.55%[80.82%, 84.15%] | 0.00%[0.00%, 0.19%] | 17.45%[15.85%, 19.18%] | 200.00 |
| Approach 2: Five checks | 0.65%[0.38%, 1.11%] | 87.45%[85.93%, 88.83%] | 0.00%[0.00%, 0.19%] | 11.90%[10.55%, 13.39%] | 102.80 |
| Approach 3: Related questions, one check | 0.00%[0.00%, 0.19%] | 66.40%[64.30%, 68.44%] | 0.00%[0.00%, 0.19%] | 33.60%[31.56%, 35.70%] | 200.00 |
| Approach 4: Related questions, five checks | 0.30%[0.14%, 0.65%] | 73.40%[71.42%, 75.29%] | 0.00%[0.00%, 0.19%] | 26.30%[24.42%, 28.27%] | 131.26 |
| Approach 5: Related questions, five adjusted checks | 0.00%[0.00%, 0.19%] | 48.85%[46.66%, 51.04%] | 0.00%[0.00%, 0.19%] | 51.15%[48.96%, 53.34%] | 164.12 |
| Approach 6: Adjusted checks plus record checks | 0.00%[0.00%, 0.19%] | 0.00%[0.00%, 0.19%] | 0.00%[0.00%, 0.19%] | 100.00%[99.81%, 100.00%] | 200.00 |
Missing grades: 10% of failures · 5% of successes
True gain: -4 points. Switching requires more than 2 points, so a switch is incorrect here. Approach 1’s average final estimate: -2.67 points; bias: +1.33 points.
| Approach | Incorrect switch | Correct evidence-hold | Incorrect evidence-hold | Inconclusive | Mean groups used |
|---|---|---|---|---|---|
| Approach 1: One check | 0.10%[0.03%, 0.36%] | 51.55%[49.36%, 53.74%] | 0.00%[0.00%, 0.19%] | 48.35%[46.17%, 50.54%] | 200.00 |
| Approach 2: Five checks | 2.05%[1.51%, 2.77%] | 63.60%[61.47%, 65.68%] | 0.00%[0.00%, 0.19%] | 34.35%[32.30%, 36.46%] | 133.78 |
| Approach 3: Related questions, one check | 0.00%[0.00%, 0.19%] | 36.95%[34.86%, 39.09%] | 0.00%[0.00%, 0.19%] | 63.05%[60.91%, 65.14%] | 200.00 |
| Approach 4: Related questions, five checks | 0.70%[0.42%, 1.17%] | 47.25%[45.07%, 49.44%] | 0.00%[0.00%, 0.19%] | 52.05%[49.86%, 54.23%] | 157.26 |
| Approach 5: Related questions, five adjusted checks | 0.10%[0.03%, 0.36%] | 23.65%[21.84%, 25.56%] | 0.00%[0.00%, 0.19%] | 76.25%[74.34%, 78.06%] | 182.48 |
| Approach 6: Adjusted checks plus record checks | 0.00%[0.00%, 0.19%] | 0.00%[0.00%, 0.19%] | 0.00%[0.00%, 0.19%] | 100.00%[99.81%, 100.00%] | 200.00 |
Missing grades: 10% of failures · 10% of successes
True gain: -4 points. Switching requires more than 2 points, so a switch is incorrect here. Approach 1’s average final estimate: -4.01 points; bias: -0.01 points.
| Approach | Incorrect switch | Correct evidence-hold | Incorrect evidence-hold | Inconclusive | Mean groups used |
|---|---|---|---|---|---|
| Approach 1: One check | 0.05%[0.01%, 0.28%] | 68.40%[66.33%, 70.40%] | 0.00%[0.00%, 0.19%] | 31.55%[29.55%, 33.62%] | 200.00 |
| Approach 2: Five checks | 1.15%[0.77%, 1.72%] | 76.20%[74.28%, 78.02%] | 0.00%[0.00%, 0.19%] | 22.65%[20.87%, 24.54%] | 119.02 |
| Approach 3: Related questions, one check | 0.00%[0.00%, 0.19%] | 50.75%[48.56%, 52.94%] | 0.00%[0.00%, 0.19%] | 49.25%[47.06%, 51.44%] | 200.00 |
| Approach 4: Related questions, five checks | 0.45%[0.24%, 0.85%] | 60.30%[58.14%, 62.42%] | 0.00%[0.00%, 0.19%] | 39.25%[37.13%, 41.41%] | 145.78 |
| Approach 5: Related questions, five adjusted checks | 0.00%[0.00%, 0.19%] | 34.40%[32.35%, 36.51%] | 0.00%[0.00%, 0.19%] | 65.60%[63.49%, 67.65%] | 174.52 |
| Approach 6: Adjusted checks plus record checks | 0.00%[0.00%, 0.19%] | 0.00%[0.00%, 0.19%] | 0.00%[0.00%, 0.19%] | 100.00%[99.81%, 100.00%] | 200.00 |
Missing grades: 20% of failures · 5% of successes
True gain: -4 points. Switching requires more than 2 points, so a switch is incorrect here. Approach 1’s average final estimate: +0.27 points; bias: +4.27 points.
| Approach | Incorrect switch | Correct evidence-hold | Incorrect evidence-hold | Inconclusive | Mean groups used |
|---|---|---|---|---|---|
| Approach 1: One check | 1.65%[1.18%, 2.31%] | 16.10%[14.55%, 17.78%] | 0.00%[0.00%, 0.19%] | 82.25%[80.51%, 83.86%] | 200.00 |
| Approach 2: Five checks | 8.50%[7.36%, 9.80%] | 29.30%[27.35%, 31.33%] | 0.00%[0.00%, 0.19%] | 62.20%[60.05%, 64.30%] | 160.26 |
| Approach 3: Related questions, one check | 0.45%[0.24%, 0.85%] | 12.45%[11.07%, 13.97%] | 0.00%[0.00%, 0.19%] | 87.10%[85.56%, 88.50%] | 200.00 |
| Approach 4: Related questions, five checks | 2.40%[1.81%, 3.17%] | 21.30%[19.56%, 23.15%] | 0.00%[0.00%, 0.19%] | 76.30%[74.39%, 78.11%] | 177.34 |
| Approach 5: Related questions, five adjusted checks | 0.55%[0.31%, 0.98%] | 7.80%[6.70%, 9.06%] | 0.00%[0.00%, 0.19%] | 91.65%[90.36%, 92.78%] | 193.50 |
| Approach 6: Adjusted checks plus record checks | 0.00%[0.00%, 0.19%] | 0.00%[0.00%, 0.19%] | 0.00%[0.00%, 0.19%] | 100.00%[99.81%, 100.00%] | 200.00 |
Missing grades: 20% of failures · 10% of successes
True gain: -4 points. Switching requires more than 2 points, so a switch is incorrect here. Approach 1’s average final estimate: -1.06 points; bias: +2.94 points.
| Approach | Incorrect switch | Correct evidence-hold | Incorrect evidence-hold | Inconclusive | Mean groups used |
|---|---|---|---|---|---|
| Approach 1: One check | 0.70%[0.42%, 1.17%] | 28.75%[26.81%, 30.77%] | 0.00%[0.00%, 0.19%] | 70.55%[68.51%, 72.51%] | 200.00 |
| Approach 2: Five checks | 4.70%[3.86%, 5.72%] | 42.20%[40.05%, 44.38%] | 0.00%[0.00%, 0.19%] | 53.10%[50.91%, 55.28%] | 153.08 |
| Approach 3: Related questions, one check | 0.10%[0.03%, 0.36%] | 19.70%[18.02%, 21.50%] | 0.00%[0.00%, 0.19%] | 80.20%[78.40%, 81.89%] | 200.00 |
| Approach 4: Related questions, five checks | 1.60%[1.14%, 2.25%] | 30.00%[28.03%, 32.05%] | 0.00%[0.00%, 0.19%] | 68.40%[66.33%, 70.40%] | 171.12 |
| Approach 5: Related questions, five adjusted checks | 0.30%[0.14%, 0.65%] | 12.80%[11.41%, 14.34%] | 0.00%[0.00%, 0.19%] | 86.90%[85.35%, 88.31%] | 189.92 |
| Approach 6: Adjusted checks plus record checks | 0.00%[0.00%, 0.19%] | 0.00%[0.00%, 0.19%] | 0.00%[0.00%, 0.19%] | 100.00%[99.81%, 100.00%] | 200.00 |
Missing grades: 30% of failures · 5% of successes
True gain: -4 points. Switching requires more than 2 points, so a switch is incorrect here. Approach 1’s average final estimate: +3.59 points; bias: +7.59 points.
| Approach | Incorrect switch | Correct evidence-hold | Incorrect evidence-hold | Inconclusive | Mean groups used |
|---|---|---|---|---|---|
| Approach 1: One check | 15.10%[13.60%, 16.74%] | 2.70%[2.08%, 3.51%] | 0.00%[0.00%, 0.19%] | 82.20%[80.46%, 83.81%] | 200.00 |
| Approach 2: Five checks | 28.50%[26.56%, 30.52%] | 8.20%[7.08%, 9.48%] | 0.00%[0.00%, 0.19%] | 63.30%[61.16%, 65.38%] | 160.58 |
| Approach 3: Related questions, one check | 2.80%[2.16%, 3.62%] | 2.20%[1.64%, 2.94%] | 0.00%[0.00%, 0.19%] | 95.00%[93.96%, 95.87%] | 200.00 |
| Approach 4: Related questions, five checks | 8.95%[7.78%, 10.28%] | 6.80%[5.78%, 7.99%] | 0.00%[0.00%, 0.19%] | 84.25%[82.59%, 85.78%] | 183.56 |
| Approach 5: Related questions, five adjusted checks | 2.10%[1.56%, 2.83%] | 1.30%[0.89%, 1.90%] | 0.00%[0.00%, 0.19%] | 96.60%[95.71%, 97.31%] | 196.76 |
| Approach 6: Adjusted checks plus record checks | 0.00%[0.00%, 0.19%] | 0.00%[0.00%, 0.19%] | 0.00%[0.00%, 0.19%] | 100.00%[99.81%, 100.00%] | 200.00 |
Missing grades: 30% of failures · 10% of successes
True gain: -4 points. Switching requires more than 2 points, so a switch is incorrect here. Approach 1’s average final estimate: +2.27 points; bias: +6.27 points.
| Approach | Incorrect switch | Correct evidence-hold | Incorrect evidence-hold | Inconclusive | Mean groups used |
|---|---|---|---|---|---|
| Approach 1: One check | 6.95%[5.92%, 8.15%] | 4.60%[3.77%, 5.61%] | 0.00%[0.00%, 0.19%] | 88.45%[86.97%, 89.78%] | 200.00 |
| Approach 2: Five checks | 17.50%[15.90%, 19.23%] | 13.75%[12.31%, 15.33%] | 0.00%[0.00%, 0.19%] | 68.75%[66.68%, 70.74%] | 165.82 |
| Approach 3: Related questions, one check | 1.25%[0.85%, 1.84%] | 5.10%[4.22%, 6.15%] | 0.00%[0.00%, 0.19%] | 93.65%[92.50%, 94.64%] | 200.00 |
| Approach 4: Related questions, five checks | 5.35%[4.45%, 6.42%] | 11.20%[9.89%, 12.66%] | 0.00%[0.00%, 0.19%] | 83.45%[81.76%, 85.01%] | 183.52 |
| Approach 5: Related questions, five adjusted checks | 1.45%[1.01%, 2.07%] | 3.20%[2.51%, 4.07%] | 0.00%[0.00%, 0.19%] | 95.35%[94.34%, 96.19%] | 195.94 |
| Approach 6: Adjusted checks plus record checks | 0.00%[0.00%, 0.19%] | 0.00%[0.00%, 0.19%] | 0.00%[0.00%, 0.19%] | 100.00%[99.81%, 100.00%] | 200.00 |
Missing grades: 40% of failures · 5% of successes
True gain: -4 points. Switching requires more than 2 points, so a switch is incorrect here. Approach 1’s average final estimate: +7.34 points; bias: +11.34 points.
| Approach | Incorrect switch | Correct evidence-hold | Incorrect evidence-hold | Inconclusive | Mean groups used |
|---|---|---|---|---|---|
| Approach 1: One check | 55.05%[52.86%, 57.22%] | 0.05%[0.01%, 0.28%] | 0.00%[0.00%, 0.19%] | 44.90%[42.73%, 47.09%] | 200.00 |
| Approach 2: Five checks | 66.05%[63.95%, 68.09%] | 1.10%[0.73%, 1.66%] | 0.00%[0.00%, 0.19%] | 32.85%[30.83%, 34.94%] | 130.78 |
| Approach 3: Related questions, one check | 17.90%[16.28%, 19.64%] | 0.25%[0.11%, 0.58%] | 0.00%[0.00%, 0.19%] | 81.85%[80.10%, 83.48%] | 200.00 |
| Approach 4: Related questions, five checks | 28.90%[26.96%, 30.93%] | 1.40%[0.97%, 2.02%] | 0.00%[0.00%, 0.19%] | 69.70%[67.65%, 71.67%] | 169.94 |
| Approach 5: Related questions, five adjusted checks | 11.65%[10.32%, 13.13%] | 0.20%[0.08%, 0.51%] | 0.00%[0.00%, 0.19%] | 88.15%[86.66%, 89.49%] | 189.92 |
| Approach 6: Adjusted checks plus record checks | 0.00%[0.00%, 0.19%] | 0.00%[0.00%, 0.19%] | 0.00%[0.00%, 0.19%] | 100.00%[99.81%, 100.00%] | 200.00 |
Missing grades: 40% of failures · 10% of successes
True gain: -4 points. Switching requires more than 2 points, so a switch is incorrect here. Approach 1’s average final estimate: +6.05 points; bias: +10.05 points.
| Approach | Incorrect switch | Correct evidence-hold | Incorrect evidence-hold | Inconclusive | Mean groups used |
|---|---|---|---|---|---|
| Approach 1: One check | 36.85%[34.76%, 38.99%] | 0.30%[0.14%, 0.65%] | 0.00%[0.00%, 0.19%] | 62.85%[60.71%, 64.94%] | 200.00 |
| Approach 2: Five checks | 51.20%[49.01%, 53.39%] | 2.65%[2.03%, 3.45%] | 0.00%[0.00%, 0.19%] | 46.15%[43.97%, 48.34%] | 145.10 |
| Approach 3: Related questions, one check | 11.15%[9.84%, 12.60%] | 0.55%[0.31%, 0.98%] | 0.00%[0.00%, 0.19%] | 88.30%[86.82%, 89.64%] | 200.00 |
| Approach 4: Related questions, five checks | 20.20%[18.50%, 22.02%] | 2.75%[2.12%, 3.56%] | 0.00%[0.00%, 0.19%] | 77.05%[75.16%, 78.84%] | 176.86 |
| Approach 5: Related questions, five adjusted checks | 7.00%[5.96%, 8.20%] | 0.45%[0.24%, 0.85%] | 0.00%[0.00%, 0.19%] | 92.55%[91.32%, 93.62%] | 193.22 |
| Approach 6: Adjusted checks plus record checks | 0.00%[0.00%, 0.19%] | 0.00%[0.00%, 0.19%] | 0.00%[0.00%, 0.19%] | 100.00%[99.81%, 100.00%] | 200.00 |
Switching requirement: 9 settings
The same three scenarios are evaluated at requirements greater than 1, 2 and 5 points. Every check and first conclusive decision is recalculated. A switch can change from correct to incorrect when the requirement changes.
Require more than 1 points · true gain +2 points · complete records
True gain: +2 points. Switching requires more than 1 points, so a switch is correct here. Approach 1’s average final estimate: +2.01 points; bias: +0.01 points.
| Approach | Correct switch | Correct evidence-hold | Incorrect evidence-hold | Inconclusive | Mean groups used |
|---|---|---|---|---|---|
| Approach 1: One check | 12.35%[10.98%, 13.86%] | 0.00%[0.00%, 0.19%] | 2.90%[2.25%, 3.73%] | 84.75%[83.11%, 86.26%] | 200.00 |
| Approach 2: Five checks | 24.15%[22.32%, 26.07%] | 0.00%[0.00%, 0.19%] | 9.90%[8.67%, 11.29%] | 65.95%[63.84%, 67.99%] | 163.02 |
| Approach 3: Related questions, one check | 4.90%[4.04%, 5.94%] | 0.00%[0.00%, 0.19%] | 1.05%[0.69%, 1.60%] | 94.05%[92.93%, 95.00%] | 200.00 |
| Approach 4: Related questions, five checks | 12.55%[11.17%, 14.07%] | 0.00%[0.00%, 0.19%] | 4.30%[3.50%, 5.28%] | 83.15%[81.45%, 84.73%] | 183.08 |
| Approach 5: Related questions, five adjusted checks | 3.15%[2.47%, 4.01%] | 0.00%[0.00%, 0.19%] | 1.00%[0.65%, 1.54%] | 95.85%[94.88%, 96.64%] | 195.88 |
| Approach 6: Adjusted checks plus record checks | 3.15%[2.47%, 4.01%] | 0.00%[0.00%, 0.19%] | 1.00%[0.65%, 1.54%] | 95.85%[94.88%, 96.64%] | 195.88 |
Require more than 2 points · true gain +2 points · complete records
True gain: +2 points. Switching requires more than 2 points, so a switch is incorrect here. Approach 1’s average final estimate: +2.01 points; bias: +0.01 points.
| Approach | Incorrect switch | Correct evidence-hold | Incorrect evidence-hold | Inconclusive | Mean groups used |
|---|---|---|---|---|---|
| Approach 1: One check | 6.15%[5.18%, 7.29%] | 6.30%[5.32%, 7.45%] | 0.00%[0.00%, 0.19%] | 87.55%[86.03%, 88.93%] | 200.00 |
| Approach 2: Five checks | 17.05%[15.47%, 18.76%] | 16.05%[14.51%, 17.72%] | 0.00%[0.00%, 0.19%] | 66.90%[64.81%, 68.93%] | 164.16 |
| Approach 3: Related questions, one check | 2.55%[1.94%, 3.34%] | 2.50%[1.90%, 3.28%] | 0.00%[0.00%, 0.19%] | 94.95%[93.90%, 95.83%] | 200.00 |
| Approach 4: Related questions, five checks | 7.20%[6.15%, 8.42%] | 6.95%[5.92%, 8.15%] | 0.00%[0.00%, 0.19%] | 85.85%[84.25%, 87.31%] | 185.18 |
| Approach 5: Related questions, five adjusted checks | 1.75%[1.26%, 2.42%] | 1.80%[1.30%, 2.48%] | 0.00%[0.00%, 0.19%] | 96.45%[95.55%, 97.18%] | 196.26 |
| Approach 6: Adjusted checks plus record checks | 1.75%[1.26%, 2.42%] | 1.80%[1.30%, 2.48%] | 0.00%[0.00%, 0.19%] | 96.45%[95.55%, 97.18%] | 196.26 |
Require more than 5 points · true gain +2 points · complete records
True gain: +2 points. Switching requires more than 5 points, so a switch is incorrect here. Approach 1’s average final estimate: +2.01 points; bias: +0.01 points.
| Approach | Incorrect switch | Correct evidence-hold | Incorrect evidence-hold | Inconclusive | Mean groups used |
|---|---|---|---|---|---|
| Approach 1: One check | 0.35%[0.17%, 0.72%] | 31.20%[29.21%, 33.26%] | 0.00%[0.00%, 0.19%] | 68.45%[66.38%, 70.45%] | 200.00 |
| Approach 2: Five checks | 3.95%[3.18%, 4.90%] | 46.20%[44.02%, 48.39%] | 0.00%[0.00%, 0.19%] | 49.85%[47.66%, 52.04%] | 149.80 |
| Approach 3: Related questions, one check | 0.15%[0.05%, 0.44%] | 16.80%[15.23%, 18.50%] | 0.00%[0.00%, 0.19%] | 83.05%[81.34%, 84.63%] | 200.00 |
| Approach 4: Related questions, five checks | 1.35%[0.93%, 1.96%] | 26.85%[24.95%, 28.84%] | 0.00%[0.00%, 0.19%] | 71.80%[69.79%, 73.73%] | 173.94 |
| Approach 5: Related questions, five adjusted checks | 0.20%[0.08%, 0.51%] | 9.75%[8.53%, 11.13%] | 0.00%[0.00%, 0.19%] | 90.05%[88.66%, 91.29%] | 192.00 |
| Approach 6: Adjusted checks plus record checks | 0.20%[0.08%, 0.51%] | 9.75%[8.53%, 11.13%] | 0.00%[0.00%, 0.19%] | 90.05%[88.66%, 91.29%] | 192.00 |
Require more than 1 points · true gain -4 points · missing grades
True gain: -4 points. Switching requires more than 1 points, so a switch is incorrect here. Approach 1’s average final estimate: +7.34 points; bias: +11.34 points.
| Approach | Incorrect switch | Correct evidence-hold | Incorrect evidence-hold | Inconclusive | Mean groups used |
|---|---|---|---|---|---|
| Approach 1: One check | 67.75%[65.67%, 69.76%] | 0.05%[0.01%, 0.28%] | 0.00%[0.00%, 0.19%] | 32.20%[30.19%, 34.28%] | 200.00 |
| Approach 2: Five checks | 76.65%[74.75%, 78.45%] | 0.65%[0.38%, 1.11%] | 0.00%[0.00%, 0.19%] | 22.70%[20.92%, 24.59%] | 119.28 |
| Approach 3: Related questions, one check | 27.05%[25.15%, 29.04%] | 0.05%[0.01%, 0.28%] | 0.00%[0.00%, 0.19%] | 72.90%[70.91%, 74.80%] | 200.00 |
| Approach 4: Related questions, five checks | 38.65%[36.54%, 40.80%] | 0.90%[0.57%, 1.42%] | 0.00%[0.00%, 0.19%] | 60.45%[58.29%, 62.57%] | 161.92 |
| Approach 5: Related questions, five adjusted checks | 18.05%[16.43%, 19.80%] | 0.05%[0.01%, 0.28%] | 0.00%[0.00%, 0.19%] | 81.90%[80.15%, 83.53%] | 185.26 |
| Approach 6: Adjusted checks plus record checks | 0.00%[0.00%, 0.19%] | 0.00%[0.00%, 0.19%] | 0.00%[0.00%, 0.19%] | 100.00%[99.81%, 100.00%] | 200.00 |
Require more than 2 points · true gain -4 points · missing grades
True gain: -4 points. Switching requires more than 2 points, so a switch is incorrect here. Approach 1’s average final estimate: +7.34 points; bias: +11.34 points.
| Approach | Incorrect switch | Correct evidence-hold | Incorrect evidence-hold | Inconclusive | Mean groups used |
|---|---|---|---|---|---|
| Approach 1: One check | 55.05%[52.86%, 57.22%] | 0.05%[0.01%, 0.28%] | 0.00%[0.00%, 0.19%] | 44.90%[42.73%, 47.09%] | 200.00 |
| Approach 2: Five checks | 66.05%[63.95%, 68.09%] | 1.10%[0.73%, 1.66%] | 0.00%[0.00%, 0.19%] | 32.85%[30.83%, 34.94%] | 130.78 |
| Approach 3: Related questions, one check | 17.90%[16.28%, 19.64%] | 0.25%[0.11%, 0.58%] | 0.00%[0.00%, 0.19%] | 81.85%[80.10%, 83.48%] | 200.00 |
| Approach 4: Related questions, five checks | 28.90%[26.96%, 30.93%] | 1.40%[0.97%, 2.02%] | 0.00%[0.00%, 0.19%] | 69.70%[67.65%, 71.67%] | 169.94 |
| Approach 5: Related questions, five adjusted checks | 11.65%[10.32%, 13.13%] | 0.20%[0.08%, 0.51%] | 0.00%[0.00%, 0.19%] | 88.15%[86.66%, 89.49%] | 189.92 |
| Approach 6: Adjusted checks plus record checks | 0.00%[0.00%, 0.19%] | 0.00%[0.00%, 0.19%] | 0.00%[0.00%, 0.19%] | 100.00%[99.81%, 100.00%] | 200.00 |
Require more than 5 points · true gain -4 points · missing grades
True gain: -4 points. Switching requires more than 5 points, so a switch is incorrect here. Approach 1’s average final estimate: +7.34 points; bias: +11.34 points.
| Approach | Incorrect switch | Correct evidence-hold | Incorrect evidence-hold | Inconclusive | Mean groups used |
|---|---|---|---|---|---|
| Approach 1: One check | 21.10%[19.37%, 22.94%] | 1.10%[0.73%, 1.66%] | 0.00%[0.00%, 0.19%] | 77.80%[75.93%, 79.57%] | 200.00 |
| Approach 2: Five checks | 34.80%[32.74%, 36.91%] | 5.45%[4.54%, 6.53%] | 0.00%[0.00%, 0.19%] | 59.75%[57.58%, 61.88%] | 158.04 |
| Approach 3: Related questions, one check | 3.65%[2.91%, 4.56%] | 1.45%[1.01%, 2.07%] | 0.00%[0.00%, 0.19%] | 94.90%[93.85%, 95.78%] | 200.00 |
| Approach 4: Related questions, five checks | 9.45%[8.24%, 10.81%] | 5.35%[4.45%, 6.42%] | 0.00%[0.00%, 0.19%] | 85.20%[83.58%, 86.69%] | 184.06 |
| Approach 5: Related questions, five adjusted checks | 2.85%[2.21%, 3.67%] | 1.50%[1.05%, 2.13%] | 0.00%[0.00%, 0.19%] | 95.65%[94.67%, 96.46%] | 196.02 |
| Approach 6: Adjusted checks plus record checks | 0.00%[0.00%, 0.19%] | 0.00%[0.00%, 0.19%] | 0.00%[0.00%, 0.19%] | 100.00%[99.81%, 100.00%] | 200.00 |
Require more than 1 points · true gain +10 points · complete records
True gain: +10 points. Switching requires more than 1 points, so a switch is correct here. Approach 1’s average final estimate: +10 points; bias: +0 points.
| Approach | Correct switch | Correct evidence-hold | Incorrect evidence-hold | Inconclusive | Mean groups used |
|---|---|---|---|---|---|
| Approach 1: One check | 94.80%[93.74%, 95.69%] | 0.00%[0.00%, 0.19%] | 0.00%[0.00%, 0.19%] | 5.20%[4.31%, 6.26%] | 200.00 |
| Approach 2: Five checks | 96.70%[95.82%, 97.40%] | 0.00%[0.00%, 0.19%] | 0.20%[0.08%, 0.51%] | 3.10%[2.43%, 3.95%] | 80.16 |
| Approach 3: Related questions, one check | 87.20%[85.66%, 88.59%] | 0.00%[0.00%, 0.19%] | 0.00%[0.00%, 0.19%] | 12.80%[11.41%, 14.34%] | 200.00 |
| Approach 4: Related questions, five checks | 90.95%[89.61%, 92.13%] | 0.00%[0.00%, 0.19%] | 0.10%[0.03%, 0.36%] | 8.95%[7.78%, 10.28%] | 104.30 |
| Approach 5: Related questions, five adjusted checks | 74.60%[72.65%, 76.46%] | 0.00%[0.00%, 0.19%] | 0.00%[0.00%, 0.19%] | 25.40%[23.54%, 27.35%] | 138.70 |
| Approach 6: Adjusted checks plus record checks | 74.60%[72.65%, 76.46%] | 0.00%[0.00%, 0.19%] | 0.00%[0.00%, 0.19%] | 25.40%[23.54%, 27.35%] | 138.70 |
Require more than 2 points · true gain +10 points · complete records
True gain: +10 points. Switching requires more than 2 points, so a switch is correct here. Approach 1’s average final estimate: +10 points; bias: +0 points.
| Approach | Correct switch | Correct evidence-hold | Incorrect evidence-hold | Inconclusive | Mean groups used |
|---|---|---|---|---|---|
| Approach 1: One check | 89.95%[88.55%, 91.19%] | 0.00%[0.00%, 0.19%] | 0.00%[0.00%, 0.19%] | 10.05%[8.81%, 11.45%] | 200.00 |
| Approach 2: Five checks | 93.45%[92.28%, 94.45%] | 0.00%[0.00%, 0.19%] | 0.45%[0.24%, 0.85%] | 6.10%[5.13%, 7.24%] | 90.24 |
| Approach 3: Related questions, one check | 78.30%[76.44%, 80.05%] | 0.00%[0.00%, 0.19%] | 0.00%[0.00%, 0.19%] | 21.70%[19.95%, 23.56%] | 200.00 |
| Approach 4: Related questions, five checks | 83.65%[81.96%, 85.21%] | 0.00%[0.00%, 0.19%] | 0.10%[0.03%, 0.36%] | 16.25%[14.70%, 17.93%] | 116.18 |
| Approach 5: Related questions, five adjusted checks | 63.00%[60.86%, 65.09%] | 0.00%[0.00%, 0.19%] | 0.05%[0.01%, 0.28%] | 36.95%[34.86%, 39.09%] | 150.98 |
| Approach 6: Adjusted checks plus record checks | 63.00%[60.86%, 65.09%] | 0.00%[0.00%, 0.19%] | 0.05%[0.01%, 0.28%] | 36.95%[34.86%, 39.09%] | 150.98 |
Require more than 5 points · true gain +10 points · complete records
True gain: +10 points. Switching requires more than 5 points, so a switch is correct here. Approach 1’s average final estimate: +10 points; bias: +0 points.
| Approach | Correct switch | Correct evidence-hold | Incorrect evidence-hold | Inconclusive | Mean groups used |
|---|---|---|---|---|---|
| Approach 1: One check | 59.20%[57.03%, 61.33%] | 0.00%[0.00%, 0.19%] | 0.05%[0.01%, 0.28%] | 40.75%[38.62%, 42.92%] | 200.00 |
| Approach 2: Five checks | 70.55%[68.51%, 72.51%] | 0.00%[0.00%, 0.19%] | 1.35%[0.93%, 1.96%] | 28.10%[26.17%, 30.11%] | 124.96 |
| Approach 3: Related questions, one check | 40.15%[38.02%, 42.32%] | 0.00%[0.00%, 0.19%] | 0.00%[0.00%, 0.19%] | 59.85%[57.68%, 61.98%] | 200.00 |
| Approach 4: Related questions, five checks | 50.20%[48.01%, 52.39%] | 0.00%[0.00%, 0.19%] | 0.35%[0.17%, 0.72%] | 49.45%[47.26%, 51.64%] | 155.04 |
| Approach 5: Related questions, five adjusted checks | 26.30%[24.42%, 28.27%] | 0.00%[0.00%, 0.19%] | 0.10%[0.03%, 0.36%] | 73.60%[71.62%, 75.49%] | 181.76 |
| Approach 6: Adjusted checks plus record checks | 26.30%[24.42%, 28.27%] | 0.00%[0.00%, 0.19%] | 0.10%[0.03%, 0.36%] | 73.60%[71.62%, 75.49%] | 181.76 |
Settings reuse outcomes or random draws and are not independent replications. At the +2-point boundary, “correct evidence-hold” follows the stated operational classification; it does not establish that the true gain is strictly below the requirement. Refusing to decide is not counted as a correct conclusion.
Worked example: choosing an approach when mistakes and delays have different costs
Suppose a team is choosing between Approach 3 (one final check) and Approach 5 (five adjusted checks). Both account for related questions. This example combines our simulated decision rates with invented business assumptions; it is not a measured financial result or a comparison of every approach.
1. Define the decision and the costs
Use a one-month horizon. Keep the current version whenever the test does not recommend switching. Assume half of proposed updates have a true +2-point gain and half have a +10-point gain, with complete records. We chose this 50/50 mix for the example; the study did not estimate it.
- Unjustified switch: $10,000. Assume switching when the gain does not clear the more-than-2-point requirement incurs this net loss. A +2-point gain is not a quality regression; the penalty represents a rollout that fails the team’s worthwhile-improvement rule.
- Delayed useful update: $1,000. Assume keeping the current version for the month when the true gain is +10 points forgoes this much net value.
In practice, estimate these amounts from rollout and rollback effort, user exposure and the value of improved answers over the same period. Use ranges where evidence is weak. Set safety or contractual limits separately; a low average cost does not override them.
2. Combine those assumptions with the decision rates
In our complete-record simulations, Approach 3 recommended an unjustified switch in 2.55% of +2-point tests; Approach 5 did so in 1.75%. At a true +10-point gain, they failed to recommend switching in 21.70% and 37.00%, respectively. The latter includes both unresolved tests and conclusions below the requirement, so it differs from the 36.95% unresolved share.
Expected loss per proposed update = chance of an update that does not clear the requirement × false-switch rate × switch loss + chance of a worthwhile update × no-switch rate × delay loss.
- Approach 3: 0.50 × 0.0255 × $10,000 + 0.50 × 0.2170 × $1,000 = $236.00.
- Approach 5: 0.50 × 0.0175 × $10,000 + 0.50 × 0.3700 × $1,000 = $272.50.
Under these assumptions, Approach 3 has the lower expected loss. It allows more unjustified switches, but misses fewer useful updates. These are averages across hypothetical future updates, not the cost of one particular decision.
3. Check the choice at a higher cost
Now raise the assumed loss from an unjustified switch to $100,000, leaving everything else unchanged. Expected loss becomes $1,383.50 for Approach 3 and $1,060.00 for Approach 5. Approach 5 is now preferable between these two: preventing an unjustified switch matters more.
Before choosing in practice, vary both the costs and the assumed frequency of worthwhile updates. Use comparable historical evaluations where available; otherwise examine a range of plausible mixes. If small changes in these assumptions change the preferred approach, the simulation does not support a firm recommendation.
4. Account for what this example leaves out
This simplified example omits evaluation costs, the value of earlier decisions within the month, other effect sizes and missing-record problems. It uses estimated simulation rates without propagating their uncertainty. A deployment analysis should include those factors and estimate the rates under conditions relevant to the product.
This example also does not calculate whether more testing is worth the cost. That requires estimating how additional evidence would change decisions and comparing the expected reduction in loss with testing and delay costs. Our study did not estimate that value.
An approach with fewer mistaken switches can still have a higher expected cost if it delays enough useful updates. The choice in this example depends on the assumed costs and frequency of each kind of update.
Worked example: one test’s confidence interval
This is an illustrative calculation, not a selected simulation result. Suppose one test contains 1,000 questions in 200 independent groups of five. A gets 700 correct and B gets 750 correct: an estimated improvement of 5 percentage points.
- Calculate the difference within each group. If A gets 3 of 5 correct and B gets 4 of 5, B’s advantage in that group is 20 percentage points. Average the differences across all 200 groups.
- Measure how much those differences vary. Suppose their sample standard deviation is 20 percentage points. This must come from the group-level results; the totals of 700 and 750 cannot determine it.
- Calculate the standard error. 20 ÷ √200 ≈ 1.414 percentage points. We use 200 independent groups, not 1,000 independent questions.
- Calculate the approximate 95% interval. 5 ± 1.96 × 1.414 gives approximately +2.23 to +7.77 points. Its lower end exceeds +2, so the unadjusted analysis recommends switching.
- Adjust for five planned checks. 5 ± 2.576 × 1.414 gives approximately +1.36 to +8.64 points. This wider interval includes gains below +2, so the adjusted analysis remains inconclusive.
The outcomes and estimated gain are unchanged. The evidence requirement changes because there are five opportunities to decide. Each simulated test gets its own intervals; the 2,000 repetitions are used to evaluate decision frequencies, not to calculate this interval.
Which method handles repeated checking?
For five planned checks, we used a Bonferroni adjustment: a 99% confidence interval at each check rather than 95%. The Bonferroni principle divides the nominal error allowance across those decision opportunities, requiring stronger evidence each time.
This simple adjustment is not necessarily the most efficient. Our intervals remain approximations; adjustment does not repair missing records or account for related questions by itself. For continuous monitoring, always-valid confidence intervals are another option when their assumptions fit. We did not compare them here.
A. Outcome generation and data conditions
Independent groups contain five paired binary outcomes. Let Δ be the population mean of B − A and d a paired difference. Each group draws C uniformly from −1 and +1 and a common Bernoulli(0.5) value Z. With q = 0.5, the probabilities of d = −1, 0, +1 are (q − Δ − hC)/2, 1 − q, and (q + Δ + hC)/2.
Differences −1 and +1 map to (A,B) = (1,0) and (0,1); a zero difference maps to (Z,Z). The mean difference is Δ and within-group difference covariance is h². The baseline grid uses h = 0 for independent paired differences and h = 0.30 for grouped scenarios. The relatedness comparison varies h as specified in the assumption-check tables. Even in the independent-difference condition, raw A/B outcomes can share difficulty; independence concerns the paired differences being analyzed.
- Complete logs: one B record per intended question.
- Random loss: each B grade is missing with probability 20%.
- Selective loss: B failures lose grades with probability 40%; B successes, 5%.
- Duplication: every B success produces two identical records with the same question identity.
All A outcomes and intended identifiers remain available. Naive analyses use observed joined pairs, including duplicate rows. The integrity workflow collapses identical duplicates and declines a decision at a look if that prefix is incomplete or invalid. A later defect does not overturn an earlier valid decision. No workflow sees hidden grades or the population truth.
B. Decision intervals, monitoring and uncertainty
Question-level methods analyze observed paired differences. Group-level methods analyze the mean observed difference within each nonempty group, giving groups equal weight. Under selective loss, these available-data means need not estimate the intended complete-data target.
Intervals take the form mean ± z × sample standard deviation / √n. Final-look and unadjusted five-look workflows use z ≈ 1.96. The five-look adjustment uses z ≈ 2.576, allocating a total nominal two-sided error of 0.05 across five planned looks. These remain normal approximations, not finite-sample guarantees.
Looks occur after 40, 80, 120, 160 and 200 intended groups. The first conclusive decision is final. The result is inconclusive if there are fewer than two usable observations or the estimated variance is zero. We do not treat zero estimated variance as certainty.
At the exact +2-point truth, switching violates the strict greater-than-2 requirement. An evidence-hold is categorized as a correct operational direction in the archived table because B does not exceed the switching requirement. It does not prove that the true effect is strictly below it: the boundary case is used to assess false switches, not power to establish inferiority.
Across 2,000 evaluations per scenario, rates have pointwise 95% Wilson Monte Carlo intervals. Paired contrasts use evaluation-level indicator differences and an approximate interval; a conservative Hoeffding fallback is used for zero-variance contrasts. These intervals quantify simulation precision, not external validity, and are not simultaneous guarantees across the complete table.
The descriptive nonboundary screen required no more than 5% false decisions and at least 70% correct decisions at the available horizon. Meeting either point-estimate threshold is not proof of its population value. No single workflow met both screens in every nonboundary scenario.
C. Duplicate records and other scenarios
In the deliberately extreme duplicate-success scenario with a true −4-point difference, adding identity-based repair and integrity checks to the adjusted workflow reduced false switches from 26.55% to 0.10%. But only 38.30% of decisions were correct and 61.60% remained inconclusive. This repair uses retained identity information; it does not show that lost outcomes can be recovered or that arbitrary duplication is harmless.
With complete records, independent question-level differences and a true +10-point gain, adjusted monitoring correctly recommended switching in 86.45% of evaluations. It therefore confirmed more improvements in this setting than in the related-question setting. Conversely, a biased analysis can appear successful when its upward bias happens to favor a genuinely positive effect. Results across all scenarios are needed to interpret a method.
Separately recoding the final question-level method’s inconclusive outcomes as an operational hold misses 10.05% of worthwhile changes in the complete grouped +10-point scenario. That is a choice to defer, not a false statistical claim that the improvement is absent. This secondary recoding is not included among the six primary workflows below.
D. Reproducibility and evidence limits
The baseline protocol specified 15 scenarios before their results were observed, with 2,000 evaluations in each. The reporting run was followed by an exact same-seed replay; four scientific output files matched byte for byte. The replay checks reproducibility, not another independent sample.
Verification included 32 deterministic checks, separate calculations for the first two evaluations in each scenario, and an audit of saved outputs. It was not a second independent implementation of all 30,000 evaluations. The public reproduction package includes the simulation code, tests, frozen outcomes, seeds and instructions for checking these results. The provenance notes distinguish the original study from publication checks.
The 23 assumption-check settings are exploratory: their grid was chosen after the baseline findings were known, then frozen before execution. They reuse outcomes or random draws and do not constitute an additional independent sample. All five baseline anchors matched exactly. Verification included nine deterministic tests, checks of every threshold trajectory, separate regeneration of the first two evaluations per setting, and audits of saved counts, intervals and paired comparisons. It was not an independent regeneration of every missingness and relatedness evaluation. The assumption-check tables and downloadable data retain every setting rather than only those highlighted in the article.
No model or provider calls were made by the simulation. We did not measure researcher time, the cost of agent assistance, user impact or the prevalence of logging faults. The absence of model calls in the simulation does not mean the research had no cost.
Additional estimates behind the main findings
In the complete grouped +2-point scenario, adjusting the five checks reduced false switches by 5.45 percentage points, with a paired 95% Monte Carlo interval of 4.45 to 6.45 points. In the independent-difference +2-point scenario, unadjusted repeated checking raised false switches from 2.05% to 7.60%.
In the complete grouped +10-point scenario, the adjustment reduced correct switches by 20.65 percentage points, with a paired 95% Monte Carlo interval of 18.88 to 22.42 points. Mean groups used increased from 116.18 to 150.98, including unresolved tests at the full horizon. These are evidence counts, not measured token or dollar costs. The strict missingness check left all tests unresolved in all six baseline missing-grade scenarios, and in every nonzero-missingness setting in the assumption checks.
Complete baseline results: all 15 scenarios
“Correct decision” combines correct switches and correct evidence-holds under the specified threshold rule. At a true +2-point gain, read the boundary caveat. Expand a scenario; wide tables scroll horizontally.
Independent questions, complete data · true gain -4 points
| Approach | False switch | False evidence-hold | Correct decision | Inconclusive | Mean groups used |
|---|---|---|---|---|---|
| Approach 1: One check | 0.00%[0.00, 0.19] | 0.00% | 77.00%[75.10, 78.79] | 23.00% | 200.00 |
| Approach 2: Five checks | 0.15%[0.05, 0.44] | 0.00% | 82.90%[81.19, 84.49] | 16.95% | 119.30 |
| Approach 3: Related questions, one check | 0.00%[0.00, 0.19] | 0.00% | 76.70%[74.80, 78.50] | 23.30% | 200.00 |
| Approach 4: Related questions, five checks | 0.15%[0.05, 0.44] | 0.00% | 82.65%[80.93, 84.25] | 17.20% | 118.28 |
| Approach 5: Related questions, five adjusted checks | 0.00%[0.00, 0.19] | 0.00% | 62.00%[59.85, 64.10] | 38.00% | 152.68 |
| Approach 6: Adjusted checks plus record checks | 0.00%[0.00, 0.19] | 0.00% | 62.00%[59.85, 64.10] | 38.00% | 152.68 |
Independent questions, complete data · true gain +2 points
| Approach | False switch | False evidence-hold | Correct decision | Inconclusive | Mean groups used |
|---|---|---|---|---|---|
| Approach 1: One check | 2.05%[1.51, 2.77] | 0.00% | 3.10%[2.43, 3.95] | 94.85% | 200.00 |
| Approach 2: Five checks | 7.60%[6.52, 8.84] | 0.00% | 8.30%[7.17, 9.59] | 84.10% | 183.08 |
| Approach 3: Related questions, one check | 2.25%[1.69, 3.00] | 0.00% | 2.90%[2.25, 3.73] | 94.85% | 200.00 |
| Approach 4: Related questions, five checks | 7.95%[6.84, 9.22] | 0.00% | 8.45%[7.31, 9.75] | 83.60% | 182.12 |
| Approach 5: Related questions, five adjusted checks | 2.55%[1.94, 3.34] | 0.00% | 2.35%[1.77, 3.11] | 95.10% | 195.02 |
| Approach 6: Adjusted checks plus record checks | 2.55%[1.94, 3.34] | 0.00% | 2.35%[1.77, 3.11] | 95.10% | 195.02 |
Independent questions, complete data · true gain +10 points
| Approach | False switch | False evidence-hold | Correct decision | Inconclusive | Mean groups used |
|---|---|---|---|---|---|
| Approach 1: One check | 0.00%[0.00, 0.19] | 0.00% | 95.05%[94.01, 95.92] | 4.95% | 200.00 |
| Approach 2: Five checks | 0.00%[0.00, 0.19] | 0.00% | 96.70%[95.82, 97.40] | 3.30% | 90.30 |
| Approach 3: Related questions, one check | 0.00%[0.00, 0.19] | 0.00% | 95.05%[94.01, 95.92] | 4.95% | 200.00 |
| Approach 4: Related questions, five checks | 0.00%[0.00, 0.19] | 0.00% | 96.85%[95.99, 97.53] | 3.15% | 89.72 |
| Approach 5: Related questions, five adjusted checks | 0.00%[0.00, 0.19] | 0.00% | 86.45%[84.88, 87.88] | 13.55% | 121.62 |
| Approach 6: Adjusted checks plus record checks | 0.00%[0.00, 0.19] | 0.00% | 86.45%[84.88, 87.88] | 13.55% | 121.62 |
Shared contexts, complete data · true gain -4 points
| Approach | False switch | False evidence-hold | Correct decision | Inconclusive | Mean groups used |
|---|---|---|---|---|---|
| Approach 1: One check | 0.00%[0.00, 0.19] | 0.00% | 70.30%[68.26, 72.26] | 29.70% | 200.00 |
| Approach 2: Five checks | 1.00%[0.65, 1.54] | 0.00% | 78.20%[76.34, 79.95] | 20.80% | 114.44 |
| Approach 3: Related questions, one check | 0.00%[0.00, 0.19] | 0.00% | 53.65%[51.46, 55.83] | 46.35% | 200.00 |
| Approach 4: Related questions, five checks | 0.30%[0.14, 0.65] | 0.00% | 61.75%[59.60, 63.86] | 37.95% | 142.90 |
| Approach 5: Related questions, five adjusted checks | 0.05%[0.01, 0.28] | 0.00% | 38.50%[36.39, 40.65] | 61.45% | 171.32 |
| Approach 6: Adjusted checks plus record checks | 0.05%[0.01, 0.28] | 0.00% | 38.50%[36.39, 40.65] | 61.45% | 171.32 |
Shared contexts, complete data · true gain +2 points
| Approach | False switch | False evidence-hold | Correct decision | Inconclusive | Mean groups used |
|---|---|---|---|---|---|
| Approach 1: One check | 6.15%[5.18, 7.29] | 0.00% | 6.30%[5.32, 7.45] | 87.55% | 200.00 |
| Approach 2: Five checks | 17.05%[15.47, 18.76] | 0.00% | 16.05%[14.51, 17.72] | 66.90% | 164.16 |
| Approach 3: Related questions, one check | 2.55%[1.94, 3.34] | 0.00% | 2.50%[1.90, 3.28] | 94.95% | 200.00 |
| Approach 4: Related questions, five checks | 7.20%[6.15, 8.42] | 0.00% | 6.95%[5.92, 8.15] | 85.85% | 185.18 |
| Approach 5: Related questions, five adjusted checks | 1.75%[1.26, 2.42] | 0.00% | 1.80%[1.30, 2.48] | 96.45% | 196.26 |
| Approach 6: Adjusted checks plus record checks | 1.75%[1.26, 2.42] | 0.00% | 1.80%[1.30, 2.48] | 96.45% | 196.26 |
Shared contexts, complete data · true gain +10 points
| Approach | False switch | False evidence-hold | Correct decision | Inconclusive | Mean groups used |
|---|---|---|---|---|---|
| Approach 1: One check | 0.00%[0.00, 0.19] | 0.00% | 89.95%[88.55, 91.19] | 10.05% | 200.00 |
| Approach 2: Five checks | 0.00%[0.00, 0.19] | 0.45% | 93.45%[92.28, 94.45] | 6.10% | 90.24 |
| Approach 3: Related questions, one check | 0.00%[0.00, 0.19] | 0.00% | 78.30%[76.44, 80.05] | 21.70% | 200.00 |
| Approach 4: Related questions, five checks | 0.00%[0.00, 0.19] | 0.10% | 83.65%[81.96, 85.21] | 16.25% | 116.18 |
| Approach 5: Related questions, five adjusted checks | 0.00%[0.00, 0.19] | 0.05% | 63.00%[60.86, 65.09] | 36.95% | 150.98 |
| Approach 6: Adjusted checks plus record checks | 0.00%[0.00, 0.19] | 0.05% | 63.00%[60.86, 65.09] | 36.95% | 150.98 |
Shared contexts, random missing outcomes · true gain -4 points
| Approach | False switch | False evidence-hold | Correct decision | Inconclusive | Mean groups used |
|---|---|---|---|---|---|
| Approach 1: One check | 0.10%[0.03, 0.36] | 0.00% | 64.10%[61.97, 66.17] | 35.80% | 200.00 |
| Approach 2: Five checks | 1.25%[0.85, 1.84] | 0.00% | 73.30%[71.32, 75.19] | 25.45% | 124.08 |
| Approach 3: Related questions, one check | 0.00%[0.00, 0.19] | 0.00% | 46.85%[44.67, 49.04] | 53.15% | 200.00 |
| Approach 4: Related questions, five checks | 0.55%[0.31, 0.98] | 0.00% | 56.75%[54.57, 58.91] | 42.70% | 147.28 |
| Approach 5: Related questions, five adjusted checks | 0.05%[0.01, 0.28] | 0.00% | 32.60%[30.58, 34.69] | 67.35% | 175.08 |
| Approach 6: Adjusted checks plus record checks | 0.00%[0.00, 0.19] | 0.00% | 0.00%[0.00, 0.19] | 100.00% | 200.00 |
Shared contexts, random missing outcomes · true gain +2 points
| Approach | False switch | False evidence-hold | Correct decision | Inconclusive | Mean groups used |
|---|---|---|---|---|---|
| Approach 1: One check | 6.10%[5.13, 7.24] | 0.00% | 5.80%[4.86, 6.91] | 88.10% | 200.00 |
| Approach 2: Five checks | 15.00%[13.50, 16.63] | 0.00% | 15.15%[13.65, 16.79] | 69.85% | 166.76 |
| Approach 3: Related questions, one check | 2.70%[2.08, 3.51] | 0.00% | 2.60%[1.99, 3.39] | 94.70% | 200.00 |
| Approach 4: Related questions, five checks | 7.40%[6.33, 8.63] | 0.00% | 7.30%[6.24, 8.52] | 85.30% | 184.02 |
| Approach 5: Related questions, five adjusted checks | 2.40%[1.81, 3.17] | 0.00% | 2.10%[1.56, 2.83] | 95.50% | 195.46 |
| Approach 6: Adjusted checks plus record checks | 0.00%[0.00, 0.19] | 0.00% | 0.00%[0.00, 0.19] | 100.00% | 200.00 |
Shared contexts, random missing outcomes · true gain +10 points
| Approach | False switch | False evidence-hold | Correct decision | Inconclusive | Mean groups used |
|---|---|---|---|---|---|
| Approach 1: One check | 0.00%[0.00, 0.19] | 0.00% | 84.55%[82.90, 86.07] | 15.45% | 200.00 |
| Approach 2: Five checks | 0.00%[0.00, 0.19] | 0.40% | 89.55%[88.13, 90.82] | 10.05% | 100.36 |
| Approach 3: Related questions, one check | 0.00%[0.00, 0.19] | 0.00% | 71.75%[69.74, 73.68] | 28.25% | 200.00 |
| Approach 4: Related questions, five checks | 0.00%[0.00, 0.19] | 0.25% | 78.10%[76.23, 79.86] | 21.65% | 124.14 |
| Approach 5: Related questions, five adjusted checks | 0.00%[0.00, 0.19] | 0.05% | 54.90%[52.71, 57.07] | 45.05% | 156.64 |
| Approach 6: Adjusted checks plus record checks | 0.00%[0.00, 0.19] | 0.00% | 0.00%[0.00, 0.19] | 100.00% | 200.00 |
Shared contexts, selectively missing outcomes · true gain -4 points
| Approach | False switch | False evidence-hold | Correct decision | Inconclusive | Mean groups used |
|---|---|---|---|---|---|
| Approach 1: One check | 55.05%[52.86, 57.22] | 0.00% | 0.05%[0.01, 0.28] | 44.90% | 200.00 |
| Approach 2: Five checks | 66.05%[63.95, 68.09] | 0.00% | 1.10%[0.73, 1.66] | 32.85% | 130.78 |
| Approach 3: Related questions, one check | 17.90%[16.28, 19.64] | 0.00% | 0.25%[0.11, 0.58] | 81.85% | 200.00 |
| Approach 4: Related questions, five checks | 28.90%[26.96, 30.93] | 0.00% | 1.40%[0.97, 2.02] | 69.70% | 169.94 |
| Approach 5: Related questions, five adjusted checks | 11.65%[10.32, 13.13] | 0.00% | 0.20%[0.08, 0.51] | 88.15% | 189.92 |
| Approach 6: Adjusted checks plus record checks | 0.00%[0.00, 0.19] | 0.00% | 0.00%[0.00, 0.19] | 100.00% | 200.00 |
Shared contexts, selectively missing outcomes · true gain +2 points
| Approach | False switch | False evidence-hold | Correct decision | Inconclusive | Mean groups used |
|---|---|---|---|---|---|
| Approach 1: One check | 97.85%[97.12, 98.40] | 0.00% | 0.00%[0.00, 0.19] | 2.15% | 200.00 |
| Approach 2: Five checks | 98.45%[97.81, 98.91] | 0.00% | 0.05%[0.01, 0.28] | 1.50% | 73.24 |
| Approach 3: Related questions, one check | 83.05%[81.34, 84.63] | 0.00% | 0.00%[0.00, 0.19] | 16.95% | 200.00 |
| Approach 4: Related questions, five checks | 87.85%[86.35, 89.21] | 0.00% | 0.00%[0.00, 0.19] | 12.15% | 109.62 |
| Approach 5: Related questions, five adjusted checks | 70.35%[68.31, 72.31] | 0.00% | 0.00%[0.00, 0.19] | 29.65% | 143.18 |
| Approach 6: Adjusted checks plus record checks | 0.00%[0.00, 0.19] | 0.00% | 0.00%[0.00, 0.19] | 100.00% | 200.00 |
Shared contexts, selectively missing outcomes · true gain +10 points
| Approach | False switch | False evidence-hold | Correct decision | Inconclusive | Mean groups used |
|---|---|---|---|---|---|
| Approach 1: One check | 0.00%[0.00, 0.19] | 0.00% | 100.00%[99.81, 100.00] | 0.00% | 200.00 |
| Approach 2: Five checks | 0.00%[0.00, 0.19] | 0.00% | 100.00%[99.81, 100.00] | 0.00% | 45.26 |
| Approach 3: Related questions, one check | 0.00%[0.00, 0.19] | 0.00% | 100.00%[99.81, 100.00] | 0.00% | 200.00 |
| Approach 4: Related questions, five checks | 0.00%[0.00, 0.19] | 0.00% | 100.00%[99.81, 100.00] | 0.00% | 56.50 |
| Approach 5: Related questions, five adjusted checks | 0.00%[0.00, 0.19] | 0.00% | 99.75%[99.42, 99.89] | 0.25% | 71.98 |
| Approach 6: Adjusted checks plus record checks | 0.00%[0.00, 0.19] | 0.00% | 0.00%[0.00, 0.19] | 100.00% | 200.00 |
Shared contexts, duplicated successful outcomes · true gain -4 points
| Approach | False switch | False evidence-hold | Correct decision | Inconclusive | Mean groups used |
|---|---|---|---|---|---|
| Approach 1: One check | 99.25%[98.77, 99.54] | 0.00% | 0.00%[0.00, 0.19] | 0.75% | 200.00 |
| Approach 2: Five checks | 99.20%[98.70, 99.51] | 0.00% | 0.35%[0.17, 0.72] | 0.45% | 59.40 |
| Approach 3: Related questions, one check | 39.30%[37.18, 41.46] | 0.00% | 0.00%[0.00, 0.19] | 60.70% | 200.00 |
| Approach 4: Related questions, five checks | 50.00%[47.81, 52.19] | 0.00% | 0.55%[0.31, 0.98] | 49.45% | 154.16 |
| Approach 5: Related questions, five adjusted checks | 26.55%[24.66, 28.53] | 0.00% | 0.15%[0.05, 0.44] | 73.30% | 179.68 |
| Approach 6: Adjusted checks plus record checks | 0.10%[0.03, 0.36] | 0.00% | 38.30%[36.19, 40.45] | 61.60% | 172.18 |
Shared contexts, duplicated successful outcomes · true gain +2 points
| Approach | False switch | False evidence-hold | Correct decision | Inconclusive | Mean groups used |
|---|---|---|---|---|---|
| Approach 1: One check | 100.00%[99.81, 100.00] | 0.00% | 0.00%[0.00, 0.19] | 0.00% | 200.00 |
| Approach 2: Five checks | 100.00%[99.81, 100.00] | 0.00% | 0.00%[0.00, 0.19] | 0.00% | 43.88 |
| Approach 3: Related questions, one check | 96.70%[95.82, 97.40] | 0.00% | 0.00%[0.00, 0.19] | 3.30% | 200.00 |
| Approach 4: Related questions, five checks | 97.55%[96.78, 98.14] | 0.00% | 0.05%[0.01, 0.28] | 2.40% | 85.12 |
| Approach 5: Related questions, five adjusted checks | 90.45%[89.08, 91.66] | 0.00% | 0.00%[0.00, 0.19] | 9.55% | 114.72 |
| Approach 6: Adjusted checks plus record checks | 1.95%[1.43, 2.65] | 0.00% | 1.65%[1.18, 2.31] | 96.40% | 196.60 |
Shared contexts, duplicated successful outcomes · true gain +10 points
| Approach | False switch | False evidence-hold | Correct decision | Inconclusive | Mean groups used |
|---|---|---|---|---|---|
| Approach 1: One check | 0.00%[0.00, 0.19] | 0.00% | 100.00%[99.81, 100.00] | 0.00% | 200.00 |
| Approach 2: Five checks | 0.00%[0.00, 0.19] | 0.00% | 100.00%[99.81, 100.00] | 0.00% | 40.12 |
| Approach 3: Related questions, one check | 0.00%[0.00, 0.19] | 0.00% | 100.00%[99.81, 100.00] | 0.00% | 200.00 |
| Approach 4: Related questions, five checks | 0.00%[0.00, 0.19] | 0.00% | 100.00%[99.81, 100.00] | 0.00% | 48.20 |
| Approach 5: Related questions, five adjusted checks | 0.00%[0.00, 0.19] | 0.00% | 100.00%[99.81, 100.00] | 0.00% | 57.12 |
| Approach 6: Adjusted checks plus record checks | 0.00%[0.00, 0.19] | 0.00% | 63.40%[61.27, 65.48] | 36.60% | 149.46 |
References
- Miller, E. (2024). Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. Pairing, shared contexts and evaluation uncertainty.
- Johari, R., Koomen, P., Pekelis, L., and Walsh, D. (2021). Always Valid Inference: Continuous Monitoring of A/B Tests. Background on monitoring; our finite-look adjustment is not their always-valid procedure.
- Fabijan, A., et al. (2019). Diagnosing Sample Ratio Mismatch in Online Controlled Experiments. Motivation for investigating measurement pipelines, not evidence for our assumed loss rates.
- NIST/SEMATECH. Bonferroni’s method. The principle behind allocating error across a fixed set of checks.