diff --git "a/app/src/index.html" "b/app/src/index.html"
--- "a/app/src/index.html"
+++ "b/app/src/index.html"
@@ -8,6 +8,7 @@
Scaling test-time compute for open models: How we implemented DeepMind’s compute-optimal recipe to solve hard math problems like OpenAI’s o1
+
@@ -55,7 +56,7 @@
Strategies for test-time compute scaling
There are two main strategies for scaling test-time compute:
Self-Refinement: Models iteratively refine their own outputs or “thoughts” by identifying and correcting errors in subsequent iterations. While effective on some tasks, this strategy usually requires models to have built-in mechanisms for self-refinement, which can limit its applicability.
Search Against a Verifier: This approach focuses on generating multiple candidate answers and using verifier to select the best one. A verifier can be anything from a hard-coded heuristic to a learned reward model, but for the purposes of this blog post we will focus on learned verifiers. It includes techniques such as Best-of-N sampling and tree search. Search strategies are more flexible and can adapt to the difficulty of the problem, although their performance is constrained by the quality of the verifier.
In this blog post, we’ll concentrate on search-based methods as they represent a practical and scalable solution for test-time compute optimization. In particular, we’ll examine the three strategies illustrated below:
Best-of-N: Generate multiple responses per problem and assign scores to each candidate answer, typically using a reward model. Then select the answer with the highest reward (or a weighted variant discussed later). This approach emphasizes answer quality over frequency.
Beam search: A systematic search method that explores the solution space, often combined with a process reward model (PRM) to optimise both the sampling and evaluation of intermediate steps in problem-solving. Unlike conventional reward models that produce a single score on the final answer, PRMs provide a sequence of scores, one for each step of the reasoning process. This ability to provide fine-grained feedback makes PRMs a natural fit for search methods with LLMs.
Diverse verifier tree search (DVTS): An extension of beam search we developed that splits the initial beams into independent subtrees, which are then expanded greedily using a PRM. This method improves solution diversity and overall performance, particularly with larger test-time compute budgets.
With an understanding of the key search strategies, let’s move on to how we evaluated them in practice.
-
Experimental setup
As illustrated in the diagram above, our experimental setup involves a pipeline with the following steps:
We begin by feeding a math problem to an LLM, which generates Npartial solutions, e.g. an intermediate step in a derivation.
Each step is scored by a PRM, which estimates the probability of each step to eventually reach the correct final answer.
The steps and PRM scores are then used by a given search strategy to select which partial solutions should be further explored to generate the next round of intermediate steps.
Once the search strategy terminates, the final candidate solutions are ranked by the PRM to produce the final answer.
To compare various search strategies, we used the following open models and datasets:
Model: We used meta-llama/Llama-3.2-1B-Instruct as our primary model for scaling test-time compute. With 1B parameters, its lightweight nature enables fast iterations, and its unsaturated performance on math benchmarks makes it an ideal choice for highlighting the benefits of scaling.
Process reward model: To guide our search strategies, we used RLHFlow/Llama3.1-8B-PRM-Deepseek-Data, an 8B reward model that has been trained using process supervision. Process supervision is a training approach where models receive feedback on each step of their reasoning process, not just the final outcome. We picked this model since it belongs to the same model family as our policy and gave better results than other PRMs like Math-Shepherd we tested in this weight class.
Dataset: We evaluated on the MATH-500 subset of the MATH benchmark, a dataset released by OpenAI as part of their research on process supervision. These math problems span seven subjects and are challenging for both humans and most LLMs. Take a look at the dataset viewer below to get a taste for the problem difficulty!
+
Experimental setup
As illustrated in the diagram above, our experimental setup involves a pipeline with the following steps:
We begin by feeding a math problem to an LLM, which generates \(N\) partial solutions, e.g. an intermediate step in a derivation.
Each step is scored by a PRM, which estimates the probability of each step to eventually reach the correct final answer.
The steps and PRM scores are then used by a given search strategy to select which partial solutions should be further explored to generate the next round of intermediate steps.
Once the search strategy terminates, the final candidate solutions are ranked by the PRM to produce the final answer.
To compare various search strategies, we used the following open models and datasets:
Model: We used meta-llama/Llama-3.2-1B-Instruct as our primary model for scaling test-time compute. With 1B parameters, its lightweight nature enables fast iterations, and its unsaturated performance on math benchmarks makes it an ideal choice for highlighting the benefits of scaling.
Process reward model: To guide our search strategies, we used RLHFlow/Llama3.1-8B-PRM-Deepseek-Data, an 8B reward model that has been trained using process supervision. Process supervision is a training approach where models receive feedback on each step of their reasoning process, not just the final outcome. We picked this model since it belongs to the same model family as our policy and gave better results than other PRMs like Math-Shepherd we tested in this weight class.
Dataset: We evaluated on the MATH-500 subset of the MATH benchmark, a dataset released by OpenAI as part of their research on process supervision. These math problems span seven subjects and are challenging for both humans and most LLMs. Take a look at the dataset viewer below to get a taste for the problem difficulty!
