AI evaluation · Economics · Agent behavior
I measure cost, success and decision quality, so a launch, review budget or model switch can be judged on evidence, not a headline score.
Independent experiments, simulations and data analysis by Mudit Gurnani.
Start with the result, then see whether it travelled.
Trust and review · Two-part study
A review rule that works on one model may not travel. Test the checker, not just the rule.
Part 1: at a 20% review budget, verification selected 143 answer-key mismatches vs 103 for confidence. Part 2: on Sol / FinQA, the checker flagged only 2 of 73 mismatches. The same rule stopped working when the checker stopped seeing them.
Cost and quality
What extra reasoning buys in correct answers, and how much the improvement costs.
Read the study →Featured exploration · Agent decisions
A two-part exploration of how instructions, starting roles and economic opportunities shaped simulated negotiations.
Start with Part 1 →New · · Agent decisions
Does the buyer’s lead survive when the opportunities are reversed? We tested both starting roles across original and mirrored situations.
Read the latest →Explore the studies, their methods, results and research code.
All 8 published studies
Agent decisions
Does the buyer’s lead survive when the opportunities are reversed? We tested both starting roles across original and mirrored situations.
Trust and review
Across 720 outcomes, a warning did little to improve results. A fresh test helped most when the first report falsely claimed success.
Trust and review
When you can check only some answers, how should you choose? Verification caught more numerical mistakes than AI confidence in this setup, but still missed many.
Cost and quality
More reasoning can improve an answer, but the payoff depends on the task and model. We compare what each upgrade costs with the extra correct answers it buys.
Agent decisions
Across 2,160 simulated negotiations, changes to buyer instructions had modest effects. The study examines the outcomes and the open question of why buyers appeared to lead.
Trust and review
Does a useful review rule work with a different model? Across four AI configurations and two datasets, the better method changed. Review policies need testing too.
Trust and review
A higher score can point to the wrong decision. Simulations show how repeated comparisons, related questions and missing results affect whether we should switch versions.
Trust and review
Test reports helped AI reviewers reject faulty fixes, but reviewers still differed in which passing fixes they accepted. We examine how evidence changes their judgments.
Behind the research
I’m Mudit Gurnani. My background is in finance and data science. I’m interested in the decisions behind AI adoption: what an improvement buys, whether the evidence holds up and what we should do differently because of it.
My focus is measuring what an AI system delivers per dollar, and whether the evidence is strong enough to support a launch, allocate a review budget or justify switching models.
Each study starts with a question, tests it with code and data, and explains both the findings and their limits. The interactive articles let you explore the results; the research code lets you inspect how they were produced.