Tiered inference governance: routing compute by consequence

Why rewriting the model costs too much

Most AI systems are made safer before anyone uses them. The model is retrained — a process the field calls preference training or post-training — so its default answers stay polite and inside policy. That works, but it charges every request the same price: the capability that was edited out to cover the worst case is missing on the easiest tasks too.

In our companion piece we called that price the alignment tax. It shows up as three bills: softer answers on every prompt, a separately retrained copy of the model for every client rulebook, and rules that still live inside the conversation — where the next incoming document can argue them away.

This study tests a different deal. Keep one capable model completely untouched, put a small gate in front of it, and let the gate decide how carefully each request should be handled — the way a good employee answers a calendar question differently than a contract clause.

The method: three lanes behind one gate

Before the model answers, a small classifier — a tiny model, not the expensive one — reads the request and scores three things: how complex it is, whether it touches a policy boundary, and what happens if the answer is wrong. Then it picks a lane.

Lane 1: Routine work runs straight through (~65% of traffic)

Formatting, summaries, internal notes — these go directly to the untouched model with a light settings overlay for tone and scope (a steering nudge applied while it runs). Nothing is retrained, and nothing is added to response time.

Lane 2: Guided work gets a small checker (~25% of traffic)

Customer replies and policy-sensitive drafts get a companion: a small checker model trained once on the house rules. For each step it proposes five candidate next words, and the frozen main model verifies all five in a single pass — accepting whichever it would have said anyway. The field calls this Reward-Shifted Speculative Sampling (RSSS); in plain terms it is draft-and-verify. Only the small checker is ever retrained — the capable model stays whole — and checking five words at once often makes this lane faster than the plain model alone.

Lane 3: Consequential work pauses to verify (~10% of traffic)

Multi-step logic, tool actions, regulated advice, and anything that smells like an attempt to talk the model out of its rules take the slow lane. Each step is checked before it is written down, approved steps are locked in, and only the stretch that failed gets redone (process-guided search with waypoint locking — Study 3 covers the mechanics).

A greeting should not cost the same governance as a liability clause — and staying in bounds should not mean dulling the model.

How we test it

The same three-lane protocol runs on two rigs a real team could own — a workstation that fits under a desk, and a datacenter GPU for multi-tenant serving:

Rig A — the office workstation

• Hardware: Apple M4 Max workstation with 64 GB of shared memory — the kind of machine a small team could sit next to.
• Model stack: Untouched 8B and 32B open models, a compact 1.5B checker, and a router small enough to be nearly free (110M parameters).
• What we watch for: Memory pressure while the checker drafts, heat under sustained Lane 3 verification, and whether the system still feels instant without a cloud connection.

Rig B — the server GPU

• Hardware: A single NVIDIA H100 datacenter card with 80 GB of high-speed memory.
• Model stack: Untouched 32B and 70B models with a 3B checker, served through vLLM — the standard high-traffic engine.
• What we watch for: Whether dozens of tenants can share the rig at once, and whether response-time promises hold under a realistic mix of all three lanes.

What we watch

We do not grade only the finished answer — the harness records four live signals while the system generates:

1. Routing mistakes: Does the gate ever send a risky request down the fast lane? We measure how quickly it classifies (milliseconds) and how many high-risk prompts it catches, against an adversarial test suite.
2. Draft acceptance: What fraction of the checker's proposed words the main model accepts. A sudden drop is an early warning that checker and model are disagreeing about this request.
3. Split latency: Time to the first word versus time per word after that — so we can tell the gate's overhead apart from actual writing speed, at the 50th, 95th, and 99th percentiles.
4. Capability checks: Standard reasoning, math, and coding benchmarks run through the governed pipeline, compared point-for-point against the untouched model.

What success looks like

The protocol passes if it meets four falsifiable targets:

• The gate is invisible: It sorts a request in under 8 milliseconds on the desk rig (under 2 ms on the server) and catches at least 98.5% of risky prompts — so consequential work never slips into the fast lane unverified.
• The checker lane is a net win: The main model accepts at least 68% of proposed words, making the lane at least 1.25× faster than writing alone — while matching the preference scores of a fully retrained model (≥ 92%).
• No alignment tax: On reasoning benchmarks the governed model stays within half a point of the untouched one's score — avoiding the 4–9% capability drop typical of heavy retraining.
• Fits real hardware: The full traffic mix (65/25/10 across lanes) stays inside the desk rig's 64 GB, and the server rig keeps at least 85% memory efficiency under shared load.

Key references