Pausing to verify: process-guided search and receding-horizon control
By Psysig Research
In queue
A model that writes step by step has no pause button: one early slip compounds through the whole answer, and rewriting everything breaks parts that were already right. This study tests checking each step as it forms — and redoing only the stretch that failed.
The problem: no pause button
When a model works through something hard — auditing a boundary, writing multi-step code, weighing competing policies — it produces the answer one step at a time, left to right. A subtle slip in step two becomes the foundation for every step after it, so the final conclusion can be confident and wrong.
Two common fixes try to catch mistakes after the fact. Both have structural problems:
1. The grader's blind spots
Generate fifty candidate answers and ask a scoring model to pick the best — the field calls this Best-of-N. It works at first, then backfires. The scorer only sees finished text and has its own quirks, so the more attempts you generate, the more the top pick tends to be an answer that games the scorer rather than the right one. True quality peaks early — at a budget researchers call N* — then slides while the score keeps climbing. That is Goodhart's Law: when a measure becomes a target, it stops being a good measure.
Figure 3.1 · Why more attempts stop helping
The grader has blind spots — and greedy picking finds them
Generate fifty candidate answers and let a scoring model pick the best: its score keeps climbing, but real quality peaks early at N* and then slides — the scorer starts rewarding its own quirks instead of the right answer.
BEFORE N*
More attempts genuinely help — the best candidate usually is the best answer.
AFTER N*
The picks keep pleasing the scorer while drifting from what you actually asked for. More compute buys worse answers.
THE FIX: CHECK EACH STEP, NOT JUST THE FINISH
2. Rewrites that break good work
The other fix asks the model to critique its draft and rewrite the whole answer. But regenerating six hundred words to fix one bad step tends to quietly alter the five hundred that were fine — a flaw in paragraph four gets repaired while paragraphs one through three drift.
The method: check each step, lock what passes
The protocol gives the system a real checkpoint by combining three mechanisms:
1. A checker for every step
Instead of scoring only the finished answer, a verifier model (a Process Reward Model, or PRM) reviews each step as it is proposed. When the search explores several possible next steps, the verifier prunes branches that introduce a contradiction or a rule violation before compute is wasted on them — a circuit breaker, not a postmortem.
2. Lock the step, then move on
Once the checker confirms a step holds up, it is locked into a running buffer of approved waypoints — the method is called Textual Model Predictive Control (TMPC). If a later step fails, generation backtracks to the last locked waypoint and redoes only the unresolved stretch. Confirmed work is never rewritten.
Figure 3.2 · Fixing only what failed
Lock the steps that check out — redo only the ones that don't
Rewriting a whole answer to fix one bad step usually breaks two good ones. This protocol verifies each step as it forms, locks it, and regenerates only the stretch that failed.
THE OLD WAY · REWRITE THE WHOLE ANSWER
One flaw at step 4 → the full draft regenerates → three previously-correct steps silently change.
THE STUDY · LOCK CHECKED STEPS (TMPC)
Steps 1–4 verified and locked → the flaw at step 5 retries only steps 5–6 → confirmed work is never rewritten.
3. Choose calmly between good options
When several candidate steps all pass review, the system samples among them with a measured acceptance rule (Metropolis–Hastings, from the QALIGN work) instead of always grabbing the top-scorer — which keeps it from snapping onto the scorer's blind spots.
When people fix a careful argument, they don't burn the pages that already hold up. They mark the last step they're sure of and work forward from there.
How we test it
Exploring several next steps at once can multiply memory demand fifty-fold if every branch duplicates the whole conversation. The protocol uses copy-on-write memory: every branch reads the same locked pages, and only the still-undecided stretch gets fresh memory. We test whether that makes stepwise verification affordable on both rigs:
Figure 3.3 · Making the slow lane affordable
Share the checked pages; copy only what is still being decided
Exploring several next steps at once can multiply memory fifty-fold if every branch duplicates the whole conversation. Copy-on-write memory lets every branch read the same locked pages and only allocates new pages for what is still open.
NAIVE · COPY EVERYTHING PER BRANCH
- Memory at 8 branches
- ≈ 8× THE PROMPT
- On a desktop rig
- RUNS OUT FAST
THE STUDY · SHARE LOCKED PAGES (COW)
- Memory at 8 branches
- ≈ 2× THE PROMPT
- Measured reduction
- ≥ 60%
Rig A — the office workstation
- Hardware: Apple M4 Max workstation with 64 GB shared memory.
- Model stack: An untouched 8B model paired with a compact 1.5B step-checker.
- Search setup: Short lookahead (3 steps) with 2–4 candidate branches, using shared memory pages.
- What we watch for: Whether a small local model that checks its steps can match a much larger unguided one — inside desktop memory and thermal limits.
Rig B — the server GPU
- Hardware: Single NVIDIA H100 with 80 GB high-speed memory.
- Model stack: Untouched 14B and 32B models with a 7B step-checker.
- Search setup: Branching width up to 16 with copy-on-write paged memory — all branches share the prompt and locked steps; new pages go only to the open stretch.
- What we watch for: Total memory amplification, page-sharing rates, and end-to-end latency as the attempt budget grows from 4 to 256.
What we watch
Four observables cover both the reasoning quality and the systems cost:
- Score versus truth: Across attempt budgets from 4 to 256, we plot the checker's score against independently verified correctness — watching the Goodhart gap open under greedy picking and close under stepwise verification.
- Locked-work survival: The share of previously confirmed steps that still hold after later revisions — comparing waypoint locking against whole-answer rewrites.
- Memory amplification: Peak memory for a branching search relative to a single straight-through answer, tracking how well shared pages keep the tree cheap.
- Small-model payoff: Total compute and wall-clock time for an 8B model doing careful stepwise search, versus simply throwing a 70B model at the task once.
What success looks like
Four quantitative expectations:
- No Goodhart collapse: Where greedy picking starts degrading around 32 attempts, stepwise verification keeps improving (or holds) out to 128 attempts with no score-gaming artifacts.
- Confirmed work survives: Waypoint locking preserves at least 98% of verified steps across revision passes — versus the 18–30% of good work lost in unconstrained rewrites.
- Memory stays flat: At 8 branches, shared pages cut memory by at least 60% versus naive duplication — keeping total amplification under 2.4× a single answer.
- A small model punches up: On multi-step math and policy benchmarks, the 8B model checking its steps on the desk rig matches or beats a single pass from an unguided 70B.