How small can a verifier be?

What the verifier sees

Each verifier reads a problem and a proposed answer and returns a single verdict. Here are some examples to illustrate each task and how the verifiers judge the proposed solution. Hover above the samples to avoid transitioning to other examples.

How it's measured

The Pipeline

Every task has the same shape: a problem P, a candidate answer A, and a ground-truth verdict T. The verifier needs to check whether A is a correct answer to P. How we obtain the triples (P, A, T) depends on the task (see the pipeline below), but they are always used the same way: a task-specific verifier is fine-tuned to read the pair (P, A) and reproduce T. At test time the verifier sees only (P, A) and emits its own verdict v̂, which we compare against the held-out T.

A verifier's score on a task is plain accuracy over its frozen N = 1,200-example test set:

What differs between tasks is where the label T comes from. For Countdown and Maze supervision is free: a generator proposes candidate answers and an exact checker supplies T. For faithfulness there is no checker, so T is HaluEval's human judgement, and zero-shot Gemini 2.5 Flash is scored on the same test set as a baseline.

Note that we sample 6 answers per problem (at temperature 0.9), which yields a natural mix of correct and incorrect answers to the same problem (and the wrong ones are realistic, model-made mistakes, exactly the near-misses a verifier must catch). It also forces the verifier to judge the answer rather than the problem: when one problem appears with some answers labelled Yes and others No, it can't shortcut on problem features alone.

Finally, it turns 5,000 problems into ~30k labelled examples, and two build-time additions make that set both balanced and difficult for models. Gold solutions are each problem's known-correct answer (for Countdown, an expression using every number once that reaches the target; for Maze, the true shortest-path length), injected as a guaranteed positive. Without them, problems the generator never solves would contribute only wrong answers, so the "correct" class would collapse onto the easy problems and a verifier could score well just by learning "correct looks easy." Hard negatives go the other way: take a correct answer and perturb it just enough to break it (swap one operator, nudge one number), keeping it only if the checker now says it's wrong. Because these are one edit away from correct, catching them requires actually doing the computation, not pattern-matching the surface. This whole set is built once, then deduplicated, balanced to exactly 50 % correct / 50 % wrong, and split, so nothing is repeated or reweighted when scoring. The accuracy above is simply one verdict per item over the frozen 1,200.

Data per task

Each task is built into the same balanced 12,000-example set on an 80 / 10 / 10 split, and the test slice is frozen and shared across every model so the curves are apples-to-apples.

The models

The same 19-model ladder is fine-tuned on every task with the same data, spanning almost five orders of magnitude from 0.07M-parameter models I pretrained from scratch up to an off-the-shelf 7B. The generators, checkers and reference judge are the ones shown in the pipeline and table above.

Pretrained from scratch

my models: 11

Llama-style models copying Monad's architecture (8,192-token vocabulary, tied embeddings) and tokenizer that I pretrained on PleIAs/SYNTH for this project to reach far below any released model.

Nano sweepfrom scratch: sub-1M

probing below 1M, where verification starts to break

70K150K230K340K630K

Tiny sweepfrom scratch

explore parameter counts that cannot be found online

1M2M3M5M7M10M

Taken off the shelf

released: 8

Existing pretrained models, fine-tuned as verifiers with no architecture changes, extending to 7B.

MonadPleias

Pleias' smallest reasoning model: its architecture & tokenizer seed the from-scratch models above

56M

SmolLM2Hugging Face

instruct models

135M360M1.7B

Qwen2.5Alibaba

instruct models

0.5B1.5B3B7B

Fine-tuning the verifiers (SFT)

All SFT ran on a single H100. Full fine-tune and LoRA produce nearly identical accuracy wherever both were run, so the size trend does not depend on how the verifier is adapted.

Pretraining the tiny models from scratch

Architecture is Llama-style with tied embeddings, head dim 64, and a 3× MLP — identical in kind to Monad, only smaller. Pretraining used a cosine schedule at LR 6e-4 (bf16, from random init); the ~10M model trains in ~10 min on one H100. To locate the cliff I also pretrained a fixed-width (hidden 128) depth sweep at ~2M, 3M, 5M and 7M, plus five nano models below 1M (0.63M down to 0.07M). Token budgets and wall-clock times above are approximate (a representative subset of the eleven models is shown).

The Frontier

This section presents the evaluation results of all models presented above on the three chosen tasks. The pattern is the same everywhere: verification is flat, then on. Below a task-specific size the verifier sits at chance; then it switches on abruptly and plateaus, barely moving for the next three orders of magnitude. Countdown switches on at 0.63M, Maze at 1M, and faithfulness latest of all (it sits at chance below 1M and only climbs from ~1-2M) but each then stays essentially flat up to 7B. Surprisingly, the supposedly hardest to verify task faithfulness climbs highest of all above its switch-on, and even a 2M verifier beats zero-shot Gemini 2.5 Flash (dashed line). "Harder to solve" is not the same as "harder to check." Because the evaluation set is balanced, just guessing randomly gives you a 50% chance of getting the right answer, so this is our performance floor.

How to read it. Each point is plain accuracy over that frozen 1,200-example test set, and the shaded band is its Wilson 95% CI. Use the switch above the chart to flip to AUROC, a ranking metric (the chance a random correct answer is scored above a random wrong one) read from the model's confidence rather than its written verdict. AUROC stays informative even where the verdict text won't parse, which is how the sub-1M models still show a signal there (see The low end). Full per-rung numbers for every task are in the appendix.

Why is the plateau flat?

Across the plateau, accuracy barely moves as the verifier grows. Two things could cause that: every model could be failing a different slice (idiosyncratic errors that happen to sum to the same rate), or they could all be failing the same examples (a shared difficulty structure). To tell them apart, we took the same 6 plateau models on each task and asked, for each of the 1200 test examples, how many get it right.

Judging Faithfulness

Per-rung faithfulness numbers, including the zero-shot Gemini reference, are in the appendix.

Limitations and what's next

• Exploitable shortcuts. In Maze's for instance model-made wrong path-lengths are longer (correct ≈ 6.6 steps, wrong ≈ 15.2), so a rule as dumb as "answer ≤ 10 → correct" already scores well.
• Proxies over reasoning. More generally, a verifier may not reason about the task at all but latch onto proxies that happen to predict the verdict. The Maze length cue and HaluEval's stylistic tells are two we caught, but how often this happens, and how to detect it, would take further work to quantify.
• In-distribution classifiers. The faithfulness verifiers learn HaluEval's hallucination signatures, so their high scores measure in-distribution fine-tuning rather than general faithfulness judging.
• Next: a reward-hacking stress test. Let a model optimize against a small verifier and see whether it learns to fool it, and whether smaller verifiers get fooled sooner.
• Next: use small verifiers at inference time. Try to use fast, small verifiers to test and accept/reject for instance separate reasoning traces based on their correctness. This can help discarding erroneous answers early on at inference time.
• Next: small verifier factory. Expand the synthetic pipeline for fast adaptation to new tasks. This ideally includes an automatic characterisation of the task, generation of synthetic data for the new distribution and validation of different models to find the smallest one with good performance.

References

The models, datasets, and prior work this project builds on.