Scaling up test-time compute with latent reasoning: a recurrent depth approach ("Huginn")

ELLIS Institute Tübingen and Max Planck Institute for Intelligent Systems, University of Maryland, Lawrence Livermore National Laboratory. arXiv:2502.05171v2 [cs.LG], 17 February 2025 (preprint). Model: huggingface.co/tomg-group-umd/huginn-0125; code and data: github.com/seal-rg/recurrent-pretraining (neither inspected). Read by researcher R6 (capability), 6 October 2026, at the main session's request, as the recurrent-depth model pretrained from scratch. Provenance: papers/capability/geiping2025_recurrent_depth.provenance.json.

What was read

• Read in full: the pdftotext conversion (5,908 lines, 37 pages): sections 1–10, references, Appendix A (implications, arithmetic study, MT-Bench Table 6, implementation), Appendix B (latent-space figures) and Appendix C (data Tables 7–8).
• Not read: Figures 1, 3, 5–14 and 15–25 as images. Captions and text were read. In Appendix B, lines holding only PCA axis ticks and token labels were filtered out; every caption was read.

What they did

• Architecture. A prelude P (2 layers) embeds tokens. A core block R (4 layers) is iterated r times, and a coda C (2 layers) decodes.
  • Each iteration takes the previous state s and the embedded input e: s_i = R(e, s_{i−1}). The input is re-injected by concatenation and a learned adapter.
  • The start state s_0 is random noise, for path independence.
  • Sandwich RMSNorm; width h = 5280; vocabulary 65,536.
  • Parameters: about 1.5B in prelude and coda, 1.5B in the core, and 0.5B in the tied embedding: 3.5B in all. At r = 32 the effective depth is 132 layers.
• Training (pretrained from scratch).
  • The iteration count is drawn per micro-batch from a log-normal Poisson distribution (mean 33, median 29, heavy tail). All workers use the same r ("locked-step").
  • Truncated backpropagation: gradients flow through only the last k = 8 iterations, "so maximum activation memory and backward compute is now independent of r". The prelude still learns every step, because e is injected every step.
  • Gradient checkpointing per iteration; data parallelism only. 4,096 AMD MI250X GPUs, 16M-token batches.
  • About 800B tokens (795B scheduled), weighted towards code, math and science, with instruction data mixed in. Constant learning rate 4×10⁻⁵, never cooled down; weights averaged afterwards.
• A same-parameter, non-recurrent twin was trained in the same setting for 180B tokens: one pass through the core.

Main results (verified)

• Training at scale was fragile.

  • "At small scales all normalization strategies worked … The same was not true at scale."
  • Failed run 1: hidden states collapsed (token correlation → 1.0).
  • Failed run 2: the model "learned early to ignore the incoming state". Validation perplexity was the same at 1 or 32 iterations.
  • The final run needed sandwich norms, a learned adapter, a scaled embedding and a peak learning rate of 4×10⁻⁵.

• Compute. It "chews through raw FLOPs close to what a 32B parameter transformer would consume during pretraining". At test time it improves "up to a computation load equivalent to 50 billion parameters".

• Against the same-parameter twin at 180B tokens:

  Model	ARC-C	HellaSwag	SciQ	GSM8K CoT
  Fixed-depth twin	26.96	37.34	73.20	1.82 / 2.20
  Recurrent, r = 32	29.18	48.80	80.60	9.02 / 10.24
  Recurrent, r = 1	23.72	29.19	54.10	0.00 / 0.15

  • The authors say the advantage is "especially pronounced" on harder tasks such as ARC-C, and models are "more similar" on fact recall (SciQ).
  • Their own table does not bear that out: ARC-C +2.2 points, SciQ +7.4. The clear gap is GSM8K with chain of thought (about 5×).
  • Evaluated at r = 1, the 800B model "effectively stops improving" after 180B tokens. The gains live "entirely in the iterations of the recurrent block".

• At 800B tokens, r = 32:

  • ARC-E 69.91, ARC-C 38.23, HellaSwag 65.21, MMLU 31.38, SciQ 93.50. That is "roughly comparable" to OLMo-7B (2.5T tokens) and behind later OLMo models.
  • GSM8K CoT 34.80/42.08, and 47.23% flexible after weight averaging at r = 64. Better than every comparison model except OLMo-2-7B (66.72).
  • HumanEval 23.17.
  • Scores rise with r: ARC-E is 49.07 at r = 4, 65.11 at r = 8 and 69.91 at r = 32.

• Facts against reasoning. On OpenBookQA, "closed" (no fact given) against "open" (a relevant fact supplied): 38.2 → 49.2 (+11.0), almost closing the gap to OLMo-2 (46.2 → 53.4).

  • The authors: "the recurrent model has less capacity to memorize facts but more capacity to reason about its context".

• Saturation depends on the task and context. HellaSwag needs about 8 iterations; GSM8K uses more.

  • ARC-C saturates at 8–12 iterations zero-shot, about 20 with one example, and about 32 with 25–50 examples.

• Abilities with no extra training:

  • Per-token adaptive exit: stop when the KL divergence between successive steps falls below 5×10⁻⁴. Mean steps differ by category: 12.7 for high-school mathematics against 16.2 for moral scenarios.
  • KV-cache sharing: a fixed cache of k slots per token, with iteration i using slot i mod k. The authors explain why it works: all entries come from the same K/V projections of successive states, so they "match".
    • MT-Bench, first turn: cache of 4 slots 5.856, 8 slots 5.631, 16 slots 5.687; full cache at 32 iterations 5.662; KL exit 5.562. Standard errors are about 0.39, and "differences … are not stat. significant".
  • "Continuous chain of thought": start each token from the previous token's final state, which cuts mean steps by 1–2.
  • Self-speculative decoding, drafting with fewer iterations.

• Latent dynamics. States converge to fixed points on easy tokens and "orbit" on numerical or deliberative ones; some states drift in one direction ("sliders"). From different random starts, trajectories converge to the same behaviour (path independence).

• Coconut, as the authors relate it (Hao et al. 2024, not read): Coconut fine-tunes fixed-depth models on reasoning chains to accept their last hidden state as input. "The main distinction between both approaches is whether to pretrain from scratch for recurrence, or whether to finetune existing fixed-depth models … and whether Chain-of-Thought data is required." Recurrent depth needs no CoT data.

• Hypothesis, not tested: recurrent depth suits reasoning while mixture-of-experts suits "storing and retrieving complex information". A future architecture could combine both.

• Arithmetic (Appendix A.1): reliable two-operand addition up to 4–5 digits at r = 32. With 2–3-digit numbers it needs more than 16 iterations; it rarely adds 4–5 single digits correctly.

Limits

• A single proof-of-concept run: an untested data mix, no learning-rate cooldown, 47,000 steps.
• The only controlled baseline is the same-parameter twin, at 180B tokens. There is no same-compute dense twin (about 32B-equivalent), so efficiency per FLOP is not shown. Comparisons with OLMo and Pythia confound data, tokens and tokenizer.
• The cache-sharing evidence is one MT-Bench comparison with standard errors of about 0.39, so it can only rule out large losses. MELT later found that untrained cache sharing fails on Ouro's long reasoning, though MELT did not test Huginn.
• Latent reasoning is less overseeable than verbalised chain of thought. The authors raise this themselves.
• Implementation quirk: under mixed precision with truncated backpropagation, "gradients are only correct while executing the compiled model".

What it means for Kurisutina (inference)

• A looped model can be pretrained from scratch, at scale, with constant training memory in the loop count. Truncated backpropagation through the last k iterations, plus per-iteration checkpointing, decouples activation memory from depth. This is the closest published answer to "can a MELT-like model be trained from scratch, and on a 12 GB GPU". The recipe is for an ordinary per-loop cache, not MELT's gated shared cache.
• Iteration-count randomisation is what made the depth a usable dial:
  • adaptive exit, cache sharing and extrapolation came with no extra training;
  • Ouro (fixed T = 4 at the end) degrades beyond its trained depth;
  • MELT needed conversion training for a shared cache.
  • For a per-person "deliberation steps" setting, training with random depth is the design that makes the setting meaningful at inference.
• The same split, weaker here.
  • Depth raised multi-step reasoning most (GSM8K about 5×).
  • The fact-recall evidence is mixed: SciQ also rose 7 points. Open-book OpenBookQA gained 11 points against 5–13 for other models, which suggests fewer memorised facts.
  • Saunshi's and Ouro's controlled tests are the stronger evidence.
• Small-scale caveat: small models trained stably under every norm choice, and failures appeared only at 3.5B. Small-scale success does not certify a scaled run.