Merge branch 'main' of https://huggingface.co/spaces/bigcode/multilingual-code-evals into main

app.py

@@ -221,7 +221,7 @@ with demo:
                         """
                     **Notes:**
                     - Win Rate represents how often a model outperforms other models in each language, averaged across all languages.
-                    - The scores of instruction-tuned models might be significantly higher on humaneval-python than other languages because we use the instruction prompt format of this benchmark.
+                    - The scores of instruction-tuned models might be significantly higher on humaneval-python than other languages. We use the instruction format of HumanEval. For other languages, we use base MultiPL-E prompts.
                     - For more details check the 📝 About section.
                     """,
                         elem_classes="markdown-text",

src/text_content.py

@@ -9,7 +9,10 @@ The growing number of code models released by the community necessitates a compr
 
 ### Benchamrks & Prompts
 - HumanEval-Python reports the pass@1 on HumanEval; the rest is from MultiPL-E benchmark.
-- For all languages, we use the original benchamrk prompts for all models except HumanEval-Python, where we separate base from instruction models. We use the original code completion prompts for HumanEval for all base models, but for Instruction models, we use the Instruction version of HumanEval in [HumanEvalSynthesize](https://huggingface.co/datasets/bigcode/humanevalpack) delimited by the tokens/text recommended by the authors of each model. Figure below shows the example of OctoCoder vs Base HumanEval prompt, you can find the other prompts [here](https://github.com/bigcode-project/bigcode-evaluation-harness/blob/1d5e773a65a764ce091dd3eded78005e9144935e/lm_eval/tasks/humanevalpack.py#L211).
+- For all languages, we use the original benchamrk prompts for all models except HumanEval-Python, where we separate base from instruction models. We use the original code completion prompts for HumanEval for all base models, but for Instruction models, we use the Instruction version of HumanEval in [HumanEvalSynthesize](https://huggingface.co/datasets/bigcode/humanevalpack) delimited by the tokens/text recommended by the authors of each model.
+An exception to this is the Phind models. They seem to follow to base prompts better than the instruction versions. Therefore, we use base HumanEval prompts, following the authors' recommendation."
+
+Figure below shows the example of OctoCoder vs Base HumanEval prompt, you can find the other prompts [here](https://github.com/bigcode-project/bigcode-evaluation-harness/blob/1d5e773a65a764ce091dd3eded78005e9144935e/lm_eval/tasks/humanevalpack.py#L211).
 
 <img src="https://huggingface.co/datasets/loubnabnl/repo-images/resolve/main/humaneval_instruct.png" alt="OctoCoder vs Base HumanEval prompt" width="800px">