Technical Report · October 2026
A short video introduction to Thinking-in-Bets.
Language model agents often abandon correct work when a user challenges it, yet simply telling them to stand firm makes them ignore real mistakes. Once an agent reads a complaint, it tends to discount its own earlier checks in order to agree with the user. We introduce Thinking-in-Bets, a training-free method in which an agent writes a short private record of its main claims, the checks supporting them, and its confidence before reading user feedback. When challenged, the agent consults this Claim Checklist to keep verified work intact while investigating uncertain or newly challenged parts.
We evaluate five Gemini models across eight benchmarks spanning software engineering, professional reports, academic reasoning, financial analysis, and command-line tasks. Recording claims before criticism nearly eliminates false reversals in patch review and protects passing report drafts under mistaken pushback, whereas writing the same record after reading the complaint recovers very little of the gain. In multi-turn coding, pre-reply claim records substantially reduce unnecessary file edits after mistaken complaints, and gating the record so it is written only when the agent is uncertain lowers token overhead while fixing more real bugs than the baseline. Even without user feedback, listing the key claims in a draft and re-checking the weakest one before submission improves both answer accuracy and confidence estimates.
The work was right, the user's criticism was mistaken, and the agent breaks it anyway to agree. This is sycophancy in agent form.
The user points out a real defect, and the agent leaves it unfixed. This is what blanket "stand firm" prompts tend to cause.
A useful agent must avoid both. We argue the root cause is timing: a standard agent judges its own work only after reading the complaint, so it reads its earlier checks through the lens of the objection. People have the same weakness, known as hindsight and outcome bias. Annie Duke's remedy in Thinking in Bets is to write down your beliefs and confidence before the outcome arrives. For an agent, the user's reply is the outcome.
At the end of each turn the agent writes a private record of its main claims. Each entry is a bet: a checkable claim, the agent's confidence that it holds, and the evidence behind it (a passing test, a file and line, a passage in the prompt, or a note that it has not been checked yet). The checklist also lists rejected options, so the agent does not cycle back to approaches it already ruled out.
# Claim Checklist from Figure 1 (written before the user's reply)
C1 cookies persist p = 0.98 evidence: cookie tests pass
C2 153/153 tests pass p = 0.96 evidence: pytest run
Alt drop cookie jar p = 0.04 rejected: breaks C1When the reply arrives, the agent matches it to the claims it targets and leaves the rest untouched. A fixed threshold (τ = 0.5) on the recorded bet, together with how specific the reply is, chooses one of four actions:
Record before every reply. Best when reviewing finished work: a submitted patch, report, or answer.
Run a quick confidence check first; write the full checklist only when it is below 0.9. Best mid-session, before code is tested.
No user at all: draft, list the claims the draft depends on, re-check the weakest one, then submit.
No training, no access to model weights, no extra models. Only prompting and one extra call per turn (which On-Demand mostly removes).
Five Gemini models (3.5 Flash-Lite, 3.5 Flash, 3.6 Flash, 3.8 Flash, 4.0 Argon) on eight public benchmarks: SWE-Together, SWE-Interact, DeepSWE, Agents' Last Exam, Humanity's Last Exam, SycEval, Vals Finance, and Terminal-Bench.
A bare complaint with no failing case lowers a standard agent's confidence on passing and failing requirements alike, so it flips about one in eight correct requirements. With the Claim Checklist written before the complaint, that falls to almost zero on every model.
The same checklist and the same revision rule, applied after reading the complaint, recover less than a quarter of the gain. Once the agent has read the objection, it writes a weaker record that agrees with the critic. Meanwhile, Thinking-in-Bets still fixes 92% of the reports that really were flawed.
Stand-firm rules and stubbornness prompts resist wrong hints by also rejecting right ones. Because Thinking-in-Bets attaches confidence to each step, it holds firm on what it checked and accepts hints on what it marked uncertain. It matches or beats the strongest stubbornness prompt on every model.
Standard agents rewrite source files after most mistaken complaints, which wastes turns and breaks working code. With claims recorded before the reply, the agent asks for a concrete failing case instead. On 3.8 Flash this also raises the share of tasks finished from 55% to 70%.
While code is still being written, a checklist on every turn can lock in an unfinished draft. On-Demand Betting writes it only when the agent is unsure. It fixes far more real bugs than Standard and uses 85% fewer checklist tokens than Always-On, at the cost of slightly more broken work.
Without any feedback, Draft-First Betting lists the claims a draft depends on and re-checks the least certain one before submitting. Accuracy rises on every model and stated confidence moves much closer to reality, with no labeled calibration data.
@techreport{bui2026thinkinginbets,
title = {Thinking-in-Bets at Test Time Reduces Sycophantic Revisions in LLM Agents},
author = {Bui, Nghi D. Q.},
institution = {Google},
year = {2026},
month = {October}
}