+ 
+
+
Following the strong community reception of our FineWeb English dataset
Over the last few years, the scaling of train-time compute has dominated the progress of large language models (LLMs). Although this paradigm has proven to be remarkably effective, the resources needed to pretrain ever larger models are becoming prohibitively expensive, with billion-dollar clusters already on the horizon. This trend has sparked significant interest in a complementary approach: test-time compute scaling. Rather than relying on ever-larger pretraining budgets, test-time methods use dynamic inference strategies that allow models to “think longer” on harder problems. A prominent example is OpenAI’s o1 model, which shows consistent improvement on difficult math problems as one increases the amount of test-time compute:
-However, we quickly encountered a significant challenge: how can one effectively evaluate models across different languages during training?
+
For English, it's straightforward: we can utilize well-established benchmarks like MMLU
Although we don’t know how o1 was trained, recent research from DeepMind shows that test-time compute can be scaled optimally through strategies like iterative self-refinement or using a reward model to perform search over the space of solutions. By adaptively allocating test-time compute per prompt, smaller models can rival—and sometimes even outperform—their larger, more resource-intensive counterparts. Scaling test-time compute is especially advantageous when memory is constrained and the available hardware is not sufficient to run a larger model. However, this promising approach was demonstrated with closed-source models, and no implementation details or code were released 😢.
Over the past months we’ve been diving deep in trying to reverse engineer and reproduce several of these results and are finally happy to share some of our knowledge. More precisely, in this blog post we’ll cover:
So how well does compute-optimal scaling work in practice? Check out this plot where the tiny 1B and 3B Llama Instruct models outperform their much larger 8B and 70B siblings on the challenging MATH-500 benchmark if you give them enough “time to think” 🤯:
In the rest of this blog post, we’ll dive deep into the ingredients behind results like this one and walk you through practical strategies for implementing test-time compute scaling.
-To address these challenges, we developed a scalable and data-driven framework for evaluation task selection, which allows anyone to choose strong model evaluations for their language from existing tasks! We then applied this framework to a set of 9 diverse languages, resulting in the creation of FineTasks - a comprehensive and diverse multilingual evaluation suite.
+There are two main strategies for scaling test-time compute:
In this blog post, we’ll concentrate on search-based methods as they represent a practical and scalable solution for test-time compute optimization. In particular, we’ll examine the three strategies illustrated below:
With an understanding of the key search strategies, let’s move on to how we evaluated them in practice.
-In this blog post, we discuss:
-
As illustrated in the diagram above, our experimental setup involves a pipeline with the following steps:
To compare various search strategies, we used the following open models and datasets:
meta-llama/Llama-3.2-1B-Instruct as our primary model for scaling test-time compute. With 1B parameters, its lightweight nature enables fast iterations, and its unsaturated performance on math benchmarks makes it an ideal choice for highlighting the benefits of scaling.RLHFlow/Llama3.1-8B-PRM-Deepseek-Data, an 8B reward model that has been trained using process supervision. Process supervision is a training approach where models receive feedback on each step of their reasoning process, not just the final outcome. We picked this model since it belongs to the same model family as our policy and gave better results than other PRMs like Math-Shepherd we tested in this weight class.To warmup, we’ll begin with a simple baseline and progressively incorporate additional techniques to improve performance.
+Majority voting—or self-consistency decoding if you want to be fancy—is the most straightforward method to aggregate an LLM’s outputs. As the name suggests, for a given math problem we generate candidate solutions and pick the most frequent answer. For all our experiments we sampled up to candidates with temperature and generated up to 2048 tokens per problem.
One quirk with the MATH benchmark is that answers must be formatted in a LaTeX box like \boxed{answer} . We initially tried the following simple system prompt for Llama 3.2 1B
Please think step by step and put your final answer within \boxed{}.but found the resulting accuracy with greedy decoding () to be far worse than the 30.6% that Meta reported in their release. Luckily, Meta also published the prompts they used for their evals and switching our system prompt to theirs made all the difference:
Solve the following math problem efficiently and clearly:
- What Makes a Task "Fine"?
- Covering all 7000+ languages spoken over the world would be monumental endeavor, so we settled on using 9 languages that offered diversity in script, language family and resource availability: Chinese, French, Arabic, Russian, Thai, Hindi, Turkish, Swahili, and Telugu.
- For these languages, we collected all available tasks that we could find, implementing a total of 185 tasks across languages in LightEval, HuggingFace's model evaluation library.
+- For simple problems (2 steps or fewer):
+Provide a concise solution with minimal explanation.
- Then, we began task selection with two primary goals: ensuring evaluation diversity, and making sure each task provided a reliable signal during pre-training.
+- For complex problems (3 steps or more):
+Use this step-by-step format:
- For evaluation diversity, we aimed to assess a broad range of model capabilities, including:
-
- - Reading comprehension (RC): Understanding provided context and answering questions based on it.
- - General knowledge (GK): Answering questions about facts from various fields without added context.
- - Natural Language Understanding (NLU): Comprehending the semantics of provided input.
- - Common-sense reasoning (RES): Demonstrating the ability to perform simple reasoning requiring embodied knowledge.
- - Generative tasks: Ability to generate text in the target language without the "help" of multiple choice options.
-
+## Step 1: [Concise description]
+[Brief explanation and calculations]
- We consider that tasks provide a reliable signal if they provide a dependable score. This means the score should be above the random baseline, increase as training progresses, show low variability across different seeds, and provide consistent model ranking at each training stepFor similar sized models trained with the same hyperparameters on the same amount of data. .
+## Step 2: [Concise description]
+[Brief explanation and calculations]
- Finding how much signal our tasks give during pre-training
- To thoroughly examine the signal our tasks provide, we trained many 1.5B parameter models for each language, using 30B tokens from subsets of the supported languages of the five largest openly available multilingual web datasets. These models were trained with the same hyperparameters and tokenizer. We then evaluated them at regular checkpoint intervals on the collected tasks (with no instruction and no system prompt in a 0-shot setting).
+...
- This process required multiple evaluation runs for each task due to iterations on its implementation, resulting in a total of 73 000 GPU hours consumed 🔥!
+Regardless of the approach, always conclude with:
- With 49 models trained we could finally define what a reliable signal means to us!
+Therefore, the final answer is: $\boxed{answer}$. I hope it is correct.
- Monotonicity
- One of our core requirements for a task is that it can be learned from training data and this learning can be gradually observed as the training progresses. Without this improvement through time, it's uncertain whether there will ever be an improvement in the future.
+Where [answer] is just the final number or expression that solves the problem.To measure this, we used the Spearman rank correlation to quantify the correlation between steps and score. Spearman rank correlation can capture monotonicity even when scores don't evolve linearly with the number of steps. We required each task to have at least an average correlation of 0.5 over all model training runs.
+One subtlety with evaluating answers to math problems is that strings like and are distinct, but represent mathematically equivalent answers. The standard way to handle this is to convert the a pair of answers to SymPy objects and then check whether subtracting the two objects and applying sympy.simplify gives zero.
While this approach works well when comparing a small number of candidate answers, we found it was terribly slow when comparing many pairs in a list of candidates; in some cases, slower than generating the candidates in the first place! To deal with this, we first reduced each answer to its canonical form and then computed the frequency of each form to determine the majority vote. Expand the detail below if you’re curious about how the code looks.
To obtain the canonical form of an algebraic expression, we first convert the LaTeX string to SymPy, apply sympy.simplify, and finally convert back to LaTeX:
from latex2sympy2 import latex2sympy
+from sympy import latex, simplify
+def get_canonical_form(expression: str) -> str:
+ parsed_expr = latex2sympy(expression)
+ simplified_expr = simplify(parsed_expr)
+ return latex(simplified_expr)With this function, we can then iterate over all candidate solutions in an list and keep track of how many times a canonical form has been seen before computing the final majority vote:
def find_majority_answer(answers: List[str]) -> str:
+ canonical_groups = defaultdict(int)
+ canonical_to_original = {}
-
-
-
-
-
- Low noise
- When comparing model performance on tasks, we need to consider whether differences are due to evaluation noise or genuine performance variations.
-
- Noise can arise from the stochastic processes involved in model training, such as random token sampling, data shuffling, or model initialization.
-
- For each task, we computed:
-
- - First, a standard deviation of model scores for every step (approximately every 1B tokens), which we call the per-step-std.
- - Then, to obtain a global variability measurement, we averaged all the per-step-std values to get the avg-std over the full training. We assume this value is an upper-bound across model architectures and training datasets (as it was approximated by models trained on a "dirtier" dataset, therefore with higher variability).
- - Finally, we computed the signal-to-noise ratio (SNR) as the main metric for task variability. We calculate SNR as the mean score at 30B tokens of all runs divided by the avg-std. This metric measures how significant the overall score is relative to the score variations (noise).
-
-
- We aimed for each task to have an SNR > 20. The only exception to this rule are generative tasks, which typically have relatively low SNR, but are still worth including as they provide insights into how the model behaves when prompted to generate unconstrained (without answer options). In a multilingual setting, this is particularly relevant as some models trained on multiple languages can exhibit high task scores but then suddenly reply in the wrong language for generative tasks!
-
-
-
-
-
-
-
- Non-Random Performance
- Many model capabilities are acquired later in training, thus many tasks (especially harder ones, such as math-related ones) show baseline-level performance for an extended period. While these tasks are useful, they're not ideal for early pre-training evaluation, and we did not want to keep them for this setting.
-
- We first computed the baseline random performance of the task (as the sum of 1/n_choices for all samples for multiple choice questions, and as zero for generative evaluations). Then we calculated the task's distance from the baseline as the maximum score across all models minus the baseline.
-
-
-
-
-
-
-
- Model Ordering Consistency
- Let's not forget that the main goal of these evaluations is to compare models and datasets!
-
- In the future, we want to use these evaluations to select the best datasets for full model pretraining. This means our tasks should rank datasets trained using very few tokens (we typically run data ablations on 30B tokens), in the same order as they would when trained for longer, after significantly more steps.
-
- In other words, we would like tasks to have predictive capability regarding future performance during pre-training: if pre-training dataset A outperforms pre-training dataset B at 30 billion tokens, we would like this trend to continue at 300 billion tokens.
-
- Proving this is inherently impossible, but there is a necessary preliminary condition that we can test for: for the results to be consistent at large scales, they must also first show consistency at smaller scales!
-
- To measure this consistency in task ordering, we computed the average Kendall's Tau of models ranking between every two consecutive steps. We only considered steps starting after 15B tokens of pre-training, as we found orderings before the range incredibly noisy. A high value of this metric indicates that the ordering remains consistent as training progresses.
-
-
-
-
-
-
-
-
-
- Important properties of evaluation impacting stability
- Now that we covered what we were looking for in our tasks, let's examine two important aspects that can affect the above properties: task formulations and metric choice.
-
-
-
- Task Formulations
- The way tasks are presented to the model is crucial, particularly for multiple-choice (MC) tasks. In these scenarios, we must carefully determine how the choices are displayed and what the model is expected to predict.
-
- There are two common approaches: Cloze Formulation (CF) and Multi-Choice Formulation (MCF). In CF, choices are not provided in context, allowing the model to predict each option directly. In contrast, MCF presents the choices in the prompt, using A/B/C/D prefixes, with the targets being those letter prefixes.
-
-
-
- It's important to know that:
-
- - The choice of formulation significantly impacts task scores
- - Both formulations behave very differently during training. As noted by both OLMES
-
-
- Therefore, we decided to utilize CF for task selection and MCF for later evaluation of major open source models, as they have generally undergone enough training for these evaluations to have a signal.
-
- Metrics
- As the targets in CF of multiple choice tasks are choices themselves, each target can have a different number of tokens, characters, and unconditional probability (probability of generating the choice without a context prefix).
-
-
- To account for this, we consider the following accuracy variations:
-
- - Accuracy :
- acc = \underset{i}{\arg\max}(ln(P (a_i|q)))
- - Accuracy normalized over character length :
- acc_char = \underset{i}{\arg\max}\frac{ln(P (a_i|q))}{num\_characters(a_i)}
- - Accuracy normalized over token length :
- acc_token = \underset{i}{\arg\max}\frac{ln(P (a_i|q))}{num\_tokens(a_i)}
- - PMI Accuracy :
- acc_pmi = \underset{i}{\arg\max}ln\frac{P (a_i|q)}{P (a_i|u)} , where u = ''Answer:''
-
-
- Where a_i is the answer choice i , q is a question prompt and P (a_i|q) is the probability of having a_i follow q . For more details see
-
-
-
- For our generative tasks on the other hand, we used the following metrics:
-
- prefix_match: Exact match where only the prefix of the answer must matchf1: F1 score computed over predicted/gold words extracted using a word tokenizer
-
- For both generative metrics, minor preprocessing is applied to remove articles and punctuation, and lowercase the text.
-
- The Fine selection
- With our goals and evaluation setup properly defined, we proceeded with task selection!
-
- We reviewed tasks one by one, choosing based on the quantified properties. For each language, we aimed to have at least one task for each of the four categories outlined above. Additionally we wanted to have at least 1 generative task for each language.
-
- In cases where multiple versions of a task existed (e.g., MMLU with different translation methods or native versions), we prioritized native versions as long as their metrics were reasonable, followed by human translations of English tasks. If no such version was available, we made our selection entirely based on metrics.
-
- Thus, after removing about half of the tasks, we arrived at 96 final ones, forming "FineTasks."
-
- Explore tasks
- Use the dropdowns below to navigate the list of tasks and how different metrics affect them.
-
-
- All tasks from the selection comply with the criteria outlined in previous sections, with the only exception being indicqa_tel, which we chose to include to ensure we had at least one generative task for Telugu. Overall we managed to cover all task categories for each language (the only exception being Thai Reasoning, where all tasks were unfortunately too noisy with low monotonicity to consider them).
-
- One of the biggest surprises was that some tasks, even when translated using the same method, were reliable in one language but not in others. This was evident with xWinograd, which worked quite well for Russian but did not meet our conditions for French. An even more extreme example was XNLI, which performed well for 6 out of 7 languages, failing to satisfy the reliability properties for Chinese. We had to test four different implementations before finding a reliable version, which, interestingly, was the only one that was created by native speakers and not machine translated.
-
- Feel free to use the dropdowns below to explore the evolution of scores over training for all tested tasks and metrics.
-
-
-
- Metrics recommendation
- Selecting the best evaluation metrics proved to be a challenging task. Not only is there no single metric that consistently outperforms the rest, but we often encountered situations where one metric had better monotonicity while another had a higher signal-to-noise ratio. In such cases, we typically made our decision based on the selected metric for tasks' implementation in a different language. We are aware that such hand-picking is often not possible and thus offer the following recommendations:
-
- Multichoice Tasks
-
- - We found base accuracy to perform well for tasks with answer options varying subtly (e.g. Yes/No/Also), particularly NLI tasks. In such cases, where the answer options are often each a single token, the base accuracy is advisable to use.
- - While OLMES
- - The metrics we found to be most reliable overall were length normalization metrics (token or character-based). However, the best choice was dependent on language, rather than being consistent for a given task. Due to that, we recommend using the maximum of acc_char and acc_token for the most reliable results.
Note that acc_token is heavily tokenizer dependent. On our ablations all models were trained using the same tokenizer.
-
-
- Generative Tasks
- For generative metrics, the choice is clearer: we suggest using the F1 score unless exact matching is required, as in math-related tasks. F1 is generally less noisy and more resilient to small changes in the generations.
-
-
- Open/Closed Source models tackle FineTasks
- Since we spent a lot of time and compute on task selection, we were interested in how well major open-source models would do on FineTasks. Given that our evaluation suite primarily targets pretrained models, we focused on these, with a few exceptions for models that don't offer a base (pretrained) version. These exceptions were included mainly out of curiosity, and their results should be interpreted with caution. Such models may significantly outperform other models due to the inclusion of supervised fine-tuning (SFT) data.
-
- To assess the multilingual performance disparity between open-source and closed-source models, we expanded our selection by adding a closed source model: gpt-4o-mini.
-
- As outlined in the task formulations, we are using MCF for this evaluation and employing a 5-shot approach, as recommended by OLMES
-
- Computing a global "multilingual" score
- In the previous sections, we treated each task independently. However, to determine an overall "multilingual" score of a model, we need to aggregate the results from these tasks. We begin by rescaling the individual task scores in line with the OpenLLM leaderboard We first average by task type to properly measure all model capabilities without letting a single category dominate.
-
- For the final global "multilingual" score we followed a different approach. Instead of averaging the language scores directly, we ranked the model's performance across languages in comparison to other models and then averaged those rank scores. This method ensures that the result reflects the overall model's performance across all languages, preventing an exceptionally high score in one language from skewing the final outcome.
-
- FineTasks Results
- After spending even more compute 🔥 on evaluating the selected models, we gathered the results in the following table. Here are our insights:
-
-
-
-
- Qwen family of models takes both top spots!
- The Qwen models perform exceptionally well, taking both first and second place with their 72B and 32B versions. Their key strength appears to be in handling high- and mid-resource languages (particularly Chinese), where they consistently ranked first. However, they struggled with lower-resource languages, especially Swahili and Telugu, where their performance lagged.
+ for answer in answers:
+ canonical_form = get_canonical_form(answer)
- General Knowledge: The curse of monolingual models
- The most surprising finding from our evaluation is how models explicitly trained to specialize in a narrow set of languages — like Sarvam-2B-v0.5 for Telugu, or Typhoon-v1.5-8B for Thai — tend to perform exceptionally well on generative tasks, while falling short when it comes to reasoning and general knowledge (GK) tasks, oftentimes getting close to random performance. We hypothesize two explanations: The models haven't undergone extensive enough training to be able to understand the MCF format or the higher exposure to various languages and especially English allows the non-specialized models to perform better at such GK/RES tasks. We note that good generative task performance reveals a good understanding of the target language.
+ # Increment count for the canonical form
+ canonical_groups[canonical_form] += 1
- The only exceptions to this rule are typhoon-v1.5-72b and Yi-1.5-34B, both tackling the RES/GK tasks well and managing to rank in the top 4 for their respective languages. We note that typhoon-v1.5-72b is based on Qwen models, and that Yi also included English in its training data.
+ # Track the original answer for this canonical form
+ if canonical_form not in canonical_to_original:
+ canonical_to_original[canonical_form] = answer
- A lower resource winner: Gemma-2
- Although it didn't take first place, Gemma2 performed really well in the multilingual domain, especially considering its size. It showed consistent results across all the languages we tested, excelling in low-resource languages like Telugu and Swahili. For anyone working with low-resource languages, we highly recommend Gemma-2 as a strong option.
+ # Find the canonical form with the largest count
+ max_count = max(canonical_groups.values())
+ for canonical_form, count in canonical_groups.items():
+ if count == max_count:
+ # Return the first occurring group in case of a tie
+ return canonical_to_original[canonical_form]This approach was significantly faster than checking each pair of solutions independently for equality.
Here’s how majority voting performs when applied to the generations from Llama 3.2 1B Instruct:
The results show that majority voting yields a significant improvement over the greedy decoding baseline, but its gains start to plateau after approximately generations. This limitation arises because majority voting struggles with problems that require nuanced reasoning or tasks where errors are consistent across generations. If you’re also wondering why the majority voting accuracy is worse than the 0-shot CoT baseline for N=1 and 2, that’s because we sample at T=0.8, which makes it less likely we produce the correct answer among a handful of candidates.
Building on the limitations of majority voting, let’s see how incorporating a reward model can enhance performance.
-As mentioned in the beginning, comparing closed-source models requires extra caution. These models often undergo extensive supervised fine-tuning (SFT), employ highly optimized prompting techniques, and may even generate multiple responses and select the best one. Despite these advantages, the o4-mini ranks only just above the medium-sized 27B Gemma-2. Based on this evidence, we believe that the gap between open-source and closed-source models is very narrow, if not entirely negligible.
+Best-of-N is a simple, but effective extension to majority voting that uses a reward model to determine the most plausible answer. This method comes in two main variants:
where is the reward model score of the -th solution solution to problem .
Typically, one usually uses an outcome reward model (ORM) to get a single, solution-level score. But to allow for fair comparison with the other search strategies discussed later, we will use the same PRM to score the solutions from Best-of-N. As illustrated below, PRMs produce a cumulative sequence of step-level scores per solution, so we need to perform a reduction over the steps to obtain a single solution-level score:
In the literature, the most common reductions are the following:
We experimented with each reduction and found—like DeepMind—that “last” performs best for our choice of task and PRM. We use this aggregation throughout all of our experiments and you can expand the detail below to see how we implemented it, along with the weighting procedure discussed above.
Here’s the results one gets from applying both variants of Best-of-N:
The results reveal a clear advantage: weighted Best-of-N consistently outperforms vanilla Best-of-N, especially with larger generation budgets. Its ability to aggregate scores across identical responses ensures that even less frequent but higher-quality answers are effectively prioritized.
However, despite these improvements, we’re still falling short of the performance achieved by the Llama 8B model and the Best-of-N approach is starting to plateau at generations. Can we push the boundaries further by supervising the search process step-by-step? Let’s find out 🚀!
-If you would like to evaluate your models on FineTasks and expand the above table we made it easy for you. Simply run the following command with your model of choice:
+Beam search is a structured search method that systematically explores the solution space, making it a powerful tool for improving model outputs at test-time. When combined with a PRM, beam search can optimize both the generation and evaluation of intermediate steps in problem-solving. The way it works is as follows:
\n or double new line \n\n.By allowing the PRM to evaluate the correctness of intermediate steps, beam search can identify and prioritize promising paths early in the process. This step-by-step evaluation is particularly beneficial for complex reasoning tasks like mathematics, where verifying partial solutions can significantly improve final outcomes.
When we implemented beam search with process supervision, we encountered two major footguns with the Llama 3 chat template that are worth mentioning:
\n or \n\n to terminate a step, these tokens are lost on subsequent steps and force the model to produce peculiar outputs.The solution is to overwrite the Llama 3 chat template to prevent trimming and exclude the BOS token prefix.
+
In our experiments, we followed DeepMind’s hyperparameter choices and ran beam search with the following:
As shown below, the results are striking: with a test-time budget of , beam search achieves the same accuracy as Best-of-N for , i.e. it is 4x more compute efficient! Moreover, beam search matches the performance of Llama 3.1 8B with just solutions per problem. The average performance on MATH by computer science PhD students is around 40%, so reaching nearly 55% isn’t too bad for a 1B model 💪!
Although in aggregate it is clear that beam search is a better search strategy than Best-of-N or majority voting, the DeepMind paper showed that each strategy has tradeoffs that depend on the problem difficulty and test-time compute budget.
To see which problems are best suited for which strategy, DeepMind computed a distribution over estimated problem difficulty, and then binned the results into quintiles. In other words, each problem is assigned one of 5 levels, where level 1 indicates easier problems and level 5 indicates the hardest ones. To estimate problem difficulty, DeepMind generated 2048 candidate solutions with standard sampling per problem and then proposed the following heuristics:
Here’s the breakdown of the various methods according to the pass@1 scores and across four test-time compute budgets of :
In this plot, each bar denotes a test-time compute budget, and within each bar we show the relative accuracy of each method. For example, in the group of four bars on difficulty level 2 we see that:
Although we see that beam search gives consistent gains in the medium and hard problems (levels 3-5), it tends to do worse than Best-of-N (and even majority voting!) on the simpler problems and especially at large compute budgets.
We realized from looking at the resulting trees produced by beam search, that if a single step is assigned high reward, then the whole tree collapses to that trace and thus diversity is impacted. This prompted us to explore an extension to beam search that maximises diversity - let’s take a look!
-lighteval accelerate\
- --model_args vllm,pretrained=model_name,pairwise_tokenization=True \
- --custom_task lighteval.tasks.multilingual.tasks \
- --tasks 'examples/tasks/finetasks/{cf,mcf}/{ara,fra,rus,tur,swa,hin,tel,tha,zho}' \
- --max_samples '1000'As we saw above beam search gives strong performance over Best-of-N, but tends to underperform on simpler problems and at large test-time compute budgets. To address this, we developed an extension we call Diverse Verifier Tree Search (DVTS) that is designed to maximise diversity at large .
DVTS works in a similar fashion as beam search, with the following modifications:
Here’s the results from applying DVTS to Llama 1B:
As we can see, DVTS provides a complementary strategy to beam search: at small beam search is more effective at finding correct solutions, but at large the diversity of DVTS candidates kicks in and we get better performance.
We can also see this manifested in the problem difficulty breakdown, where DVTS enhances performance on the easy / medium problems at large N, while beam search is best at small N across model problem difficulties:
FineTasks is just the beginning of our multilingual journey. As a first step in the creation of the future FineWeb multilingual release, we are using this evaluation setup to curate a high quality pretraining dataset covering a large number of languages. You can expect more news from us soon! We plan to also continue working to make evaluation in non-English domains as seamless as it is in English—and we need your help to achieve that!
+Armed with various search strategies, a natural question is which one is best? In the DeepMind paper, they proposed a compute-optimal scaling strategy where one selects the search method and hyperparameters that achieves the best performance for a given compute budget :
where is the ground-truth for question and denotes the compute-optimal scaling strategy. Since computing directly is somewhat tricky, DeepMind proposed an approximation based on the problem difficulty, i.e. allocate test-time compute according to which search strategy achieves best performance for a given difficulty level.
For example, on simpler problems and lower compute budgets, it is better to use strategies like Best-of-N, while on harder problems, beam search is the better choice. We can represent the compute-optimal strategy mathematically as follows:
[ADD DETAILS]
And voila, we now have our compute-optimal curve!
LightEval now supports over 550 tasks across various non-English languages, making it the evaluation framework with the best multilingual coverage available. However, there's still much more to do. For many languages, no tasks exist yet, despite our ongoing work. This is where we believe the strong Hugging Face community can make a difference.
+We also explored scaling up the compute-optimal recipe to Llama 3.2 3B Instruct to see at what point the benefits of the PRM fade in comparison to the policy’s own capacity. To our surprise, compute-optimal scaling works remarkably well, with the 3B model surpassing the performance of Llama 3.1 70B Instruct (22x it's size!):
We've made it incredibly easy to contribute new tasks, by developing a templating system which supports most of the popular task types, while maintaining authenticity of native language use, right down to correct punctuation. Even if you aren't able to contribute full evaluation tasks, you can still help. Many languages currently lack translations for anchor words used in evaluation, leaving hundreds of tasks unusable. You can help fill this gap by adding them following our mini guide.
+This exploration of test-time compute scaling has revealed both the potential and the challenges of leveraging search-based methods. As we look ahead, several exciting directions emerge:
Strong verifiers play a critical role in enhancing performance. However, their current limitations are apparent, as highlighted in benchmarks like ProcessBench. Improving the robustness and generalization of verifiers will be crucial for advancing these methods.
The ultimate goal—or "holy grail"—is achieving self-verification, where models can validate their own outputs autonomously. This approach appears to be what models like o1 are doing, but remains difficult to implement in practice. Unlike standard supervised fine-tuning (SFT), self-verification demands more nuanced strategies. The recent DeepMind paper on self-verification and Score sheds light on this challenge and offers a pathway for future research.
Incorporating explicit intermediate steps or “thoughts” during generation could further enhance reasoning and decision-making. By integrating structured reasoning into the search process, we may unlock better performance on complex tasks.
This method can also serve as a powerful data generation process, creating high-quality training datasets. For example, fine-tuning models like Llama 1B on correct traces produced by search could yield significant gains. This on-policy approach resembles techniques like ReST or V-StaR but with the added benefits of search, offering a promising direction for iterative improvement.
Open process reward models (PRMs) are relatively rare, limiting their broader application. Developing and sharing more PRMs for different domains is a critical area where the community can contribute significantly.
While current methods excel in domains like math and code, where solutions are inherently verifiable, extending these techniques to other areas remains a major challenge. How can we adapt these strategies for less structured or subjective tasks? This is a vital question for future exploration.
We are grateful to Aviral Kumar for many discussions about test-time compute scaling and for sharing implementation details from their work. We thank Chun Te Lee for designing the lovely banner and Thomas Wolf, Leandro von Werra, Colin Raffel, and Quentin Gallouédec for many helpful suggestions to improve the blog post. We also thank Hugo Larcher and Mathieu Morlon for continually optimising the Hugging Face Science Cluster to make the GPUs go brrr 🔥!
-We're looking forward to revisiting this analysis in the future, not with just 9 languages, but at least 50—thanks to community contributions! Let's level the playing field between English and other languages together! 🤗
For attribution in academic contexts, please cite this work as
-Kydlicek, et al., "FineTasks: Finding signal in a haystack of 200+ multilingual tasks", 2024.+
Beeching, Edward, and Tunstall, Lewis, and Rush, Sasha, "Scaling test-time compute with open models.", 2024.
BibTeX citation
-@misc{kydlicek2024finetasksmultilingualtasks,
- title={FineTasks: Finding signal in a haystack of 200+ multilingual tasks},
- author={Hynek Kydlíček and Guilherme Penedo and Clémentine Fourier and Nathan Habib and Thomas Wolf},
- url={https://huggingface.co/spaces/HuggingFaceFW/blogpost-fine-tasks},
+ @misc{beeching2024scalingtesttimecompute,
+ title={Scaling test-time compute with open models},
+ author={Edward Beeching and Lewis Tunstall and Sasha Rush},
+ url={https://huggingface.co/spaces/HuggingFaceH4/blogpost-scaling-test-time-compute},
}