Scaling LLM test-time compute optimally can be more effective than scaling model parameters

Charlie Snell (UC Berkeley; work done at Google DeepMind), Jaehoon Lee, Kelvin Xu, Aviral Kumar (Google DeepMind). arXiv:2408.03314v1 [cs.LG], 6 August 2024 (later ICLR 2025; the arXiv v1 was read). Read by researcher R6 (capability), 6 October 2026. Provenance: papers/capability/snell2024_test_time_compute.provenance.json.

What was read

• Read in full: every line of the pdftotext conversion (2,339 lines, 37 pages): sections 1–8, references, and Appendices A–M (related work, revision and majority results, unsupervised difficulty bins, PRM training and aggregation, PRM against ORM, prompts, revision-model training, selection and verifier, ReST, example captions).
• Not read: the figures as images, and the example generations in Appendices L–M (images; captions read). The per-cell relative-improvement numbers in Figure 1 could not be mapped reliably to conditions from the text layer and are not used.

What they did

• Question: if a model may spend a fixed, non-trivial amount of compute at inference, how much can it improve on a hard prompt? And when is that better than a bigger model?
• Setting: MATH (12k train, 500 test), with PaLM 2-S* (Codey) fine-tuned for two mechanisms:
  1. Search against a process reward model (PRM): a verifier that scores each step, trained without human labels from Monte Carlo rollouts. Methods: best-of-N weighted, beam search, lookahead search.
  2. Refining the proposal distribution: a model fine-tuned to revise its own earlier attempts sequentially, combined with parallel sampling.
• Question difficulty is defined relative to the base model: five quantile bins of its pass@1, estimated from 2,048 samples per question. An "oracle" version uses ground truth; a "predicted" version uses verifier scores.
• "Compute-optimal" strategy: pick the best method and its settings per difficulty bin and budget.
• FLOPs-matched comparison against a ~14× larger pretrained model (same data, more parameters).
  • Exchange rate: pretraining X = 6·N·D_pretrain, inference Y = 2·N·D_inference.
  • Evaluated at three inference loads, R = D_inference / D_pretrain = 0.16, 0.79 and 22.

Main results (verified)

• Compute-optimal allocation is about 4× more efficient than best-of-N. With both revisions and PRM search, it matches or beats best-of-N "using up to 4x less test-time compute" (for example 64 against 256 samples). Overall the authors give a gain of 2–4×.
• Which method helps depends on difficulty:
  • easy questions: sequential revision is best, and strong search over-optimises the verifier (beam search degrades with budget);
  • medium questions: beam search beats best-of-N, with a mix of sequential and parallel compute;
  • the hardest bin: "no method makes much meaningful progress".
• Revision has costs: about 38% of correct answers were turned incorrect by naive further revision, so selection by verifier or majority is needed.
• Test-time compute against parameters:
  • "On easy and medium questions, which are within a model's capabilities, or in settings with small inference requirement, test-time compute can easily cover up for additional pretraining." There, a small model with optimal test-time compute outperforms a ~14× larger model at equal FLOPs.
  • "However, on challenging questions which are outside a given base model's capabilities or under higher inference requirement, pretraining is likely more effective."
  • "Test-time and pretraining compute are not 1-to-1 exchangeable."
• The premise, stated by the authors: test-time compute helps most "when models already have all the basic 'knowledge' needed to answer a question, and instead the primary challenge is about drawing (complex) inferences from this knowledge".
• Smaller findings:
  • Using only the PRM's last-step score was the best aggregation, better than min or product.
  • The PRM beat an outcome reward model (ORM) by a growing margin as samples increased.
  • Optimising the revision model with ReST hurt sequential revision.

Limits

• One base model family and one domain (competition math with checkable answers). Capability-specific fine-tuning (verifier, reviser) was required. Untuned models do not revise well.
• Difficulty estimation itself costs 2,048 samples per question, which the analysis does not charge.
• The bigger model is greedy-decoded, not itself given test-time compute. The comparison scales parameters with data fixed, not compute-optimally.

What it means for Kurisutina (inference)

• "Thinking time substitutes for size" holds only inside the base's competence.
  • Extra deliberation lifts problems the model already sometimes solves. It does not reach problems outside its capability; there, a bigger or better-trained base wins.
  • A replica of a person whose skill exceeds the base on hard problems cannot think its way up to that person. This bounds the working answer's "thinking time substitutes for size to a degree": the degree is "easy and medium problems for this base".
• Thinking allocated by difficulty is itself a person trait. Humans spend more time on harder problems; the compute-optimal policy does the same. For fidelity the replica should reproduce the person's allocation and its errors (how long they think, when they stop), not the compute-optimal one. Over-thinking relative to the person is overshoot (compare Maia, where search made predictions less human).
• Over-optimisation creates non-human errors. Strong search against an imperfect verifier produced degenerate, repetitive or too-short solutions on easy problems. A replica that "thinks harder" with the wrong critic will make errors no person makes.