update text on prompts
Browse files- src/text_content.py +2 -1
src/text_content.py
CHANGED
@@ -10,11 +10,12 @@ The growing number of code models released by the community necessitates a compr
|
|
10 |
### Benchamrks & Prompts
|
11 |
- HumanEval-Python reports the pass@1 on HumanEval; the rest is from MultiPL-E benchmark.
|
12 |
- For all languages, we use the original benchamrk prompts for all models except HumanEval-Python, where we separate base from instruction models. We use the original code completion prompts for HumanEval for all base models, but for Instruction models, we use the Instruction version of HumanEval in [HumanEvalSynthesize](https://huggingface.co/datasets/bigcode/humanevalpack) delimited by the tokens/text recommended by the authors of each model.
|
13 |
-
An exception to this is the Phind models. They seem to follow to base prompts better than the instruction versions. Therefore, we use base HumanEval prompts, following the authors' recommendation."
|
14 |
|
15 |
Figure below shows the example of OctoCoder vs Base HumanEval prompt, you can find the other prompts [here](https://github.com/bigcode-project/bigcode-evaluation-harness/blob/1d5e773a65a764ce091dd3eded78005e9144935e/lm_eval/tasks/humanevalpack.py#L211).
|
16 |
|
17 |
<img src="https://huggingface.co/datasets/loubnabnl/repo-images/resolve/main/humaneval_instruct.png" alt="OctoCoder vs Base HumanEval prompt" width="800px">
|
|
|
|
|
18 |
|
19 |
### Evaluation Parameters
|
20 |
- All models were evaluated with the [bigcode-evaluation-harness](https://github.com/bigcode-project/bigcode-evaluation-harness/tree/main) with top-p=0.95, temperature=0.2, max_length_generation 512, and n_samples=50.
|
|
|
10 |
### Benchamrks & Prompts
|
11 |
- HumanEval-Python reports the pass@1 on HumanEval; the rest is from MultiPL-E benchmark.
|
12 |
- For all languages, we use the original benchamrk prompts for all models except HumanEval-Python, where we separate base from instruction models. We use the original code completion prompts for HumanEval for all base models, but for Instruction models, we use the Instruction version of HumanEval in [HumanEvalSynthesize](https://huggingface.co/datasets/bigcode/humanevalpack) delimited by the tokens/text recommended by the authors of each model.
|
|
|
13 |
|
14 |
Figure below shows the example of OctoCoder vs Base HumanEval prompt, you can find the other prompts [here](https://github.com/bigcode-project/bigcode-evaluation-harness/blob/1d5e773a65a764ce091dd3eded78005e9144935e/lm_eval/tasks/humanevalpack.py#L211).
|
15 |
|
16 |
<img src="https://huggingface.co/datasets/loubnabnl/repo-images/resolve/main/humaneval_instruct.png" alt="OctoCoder vs Base HumanEval prompt" width="800px">
|
17 |
+
- An exception to this is the Phind models. They seem to follow to base prompts better than the instruction versions. Therefore, following the authors' recommendation we use base HumanEval prompts.
|
18 |
+
- Also note that for WizardCoder-Python-34B-V1.0 & WizardCoder-Python-13B-V1.0 (CodeLLaMa based), we use the HumanEval-Python instruction prompt that the original authors used with their postprocessing (instead of HumanEvalSynthesize), code is available [here](https://github.com/bigcode-project/bigcode-evaluation-harness/pull/133)).
|
19 |
|
20 |
### Evaluation Parameters
|
21 |
- All models were evaluated with the [bigcode-evaluation-harness](https://github.com/bigcode-project/bigcode-evaluation-harness/tree/main) with top-p=0.95, temperature=0.2, max_length_generation 512, and n_samples=50.
|