Update README.md
Browse files
README.md
CHANGED
|
@@ -23,16 +23,9 @@ The vision of OpenCSG is to empower every industry, every company, and every ind
|
|
| 23 |
## Model Description
|
| 24 |
|
| 25 |
The [StarCoder](https://huggingface.co/bigcode/starcoder) models are 15.5B parameter models trained on 80+ programming languages from [The Stack (v1.2)](https://huggingface.co/datasets/bigcode/the-stack), with opt-out requests excluded.
|
|
|
|
| 26 |
<br>
|
| 27 |
|
| 28 |
-
This is the repository for the base 7B version finetuned based on [CodeLlama-7b-hf](https://huggingface.co/codellama/CodeLlama-7b-hf).
|
| 29 |
-
|
| 30 |
-
| Model Size | Base Model |
|
| 31 |
-
| --- | ----------------------------------------------------------------------------- |
|
| 32 |
-
| 7B | [opencsg/Opencsg-CodeLlama-7b-v0.1](https://huggingface.co/opencsg/opencsg-CodeLlama-7b-v0.1) |
|
| 33 |
-
| 13B | [opencsg/Opencsg-CodeLlama-13b-v0.1](https://huggingface.co/opencsg/opencsg-CodeLlama-13b-v0.1) |
|
| 34 |
-
| 34B | [opencsg/Opencsg-CodeLlama-34b-v0.1](https://huggingface.co/opencsg/opencsg-CodeLlama-34b-v0.1) |
|
| 35 |
-
|
| 36 |
|
| 37 |
## Model Eval
|
| 38 |
|
|
@@ -43,18 +36,14 @@ It is impratical for us to manually set specific configuration for each fine-tun
|
|
| 43 |
Thus, OpenCSG strained our brains to provide a relatively fair method to compare the fine-tuned models on HumanEval benchmark.
|
| 44 |
To simplify the comparision, we chosed the Pass@1 metric on python language, but our finetuning dataset includes samples in multi language.
|
| 45 |
|
| 46 |
-
**For fair, we evaluated the fine-tuned and origin
|
| 47 |
|
| 48 |
**Otherwise, we use greedy decoding method for each model during the evaluation.**
|
| 49 |
|
| 50 |
| Model | HumanEval python pass@1 |
|
| 51 |
| --- |----------------------------------------------------------------------------- |
|
| 52 |
-
|
|
| 53 |
-
| opencsg-
|
| 54 |
-
| CodeLlama-13b-hf | 36.0%|
|
| 55 |
-
| opencsg-CodeLlama-13b-v0.1(4k) | **45.1%** |
|
| 56 |
-
| CodeLlama-34b-hf | 48.2%|
|
| 57 |
-
| opencsg-CodeLlama-34b-v0.1(4k)| **48.8%** |
|
| 58 |
|
| 59 |
**TODO**
|
| 60 |
- we will provide much more benchmark scores on fine-tuned models in future.
|
|
@@ -70,7 +59,7 @@ from transformers import AutoTokenizer
|
|
| 70 |
import transformers
|
| 71 |
import torch
|
| 72 |
|
| 73 |
-
model = "opencsg/opencsg-
|
| 74 |
|
| 75 |
tokenizer = AutoTokenizer.from_pretrained(model, trust_remote_code=True)
|
| 76 |
pipeline = transformers.pipeline(
|
|
@@ -107,14 +96,10 @@ for seq in sequences:
|
|
| 107 |
```
|
| 108 |
# Training
|
| 109 |
|
| 110 |
-
## Basic Model
|
| 111 |
-
|
| 112 |
-
[codellama-7b-hf](https://huggingface.co/codellama/CodeLlama-7b-hf)
|
| 113 |
-
|
| 114 |
## Hardware
|
| 115 |
|
| 116 |
- **GPUs:** 8 Tesla A800
|
| 117 |
-
- **Training time:**
|
| 118 |
|
| 119 |
## Software
|
| 120 |
|
|
|
|
| 23 |
## Model Description
|
| 24 |
|
| 25 |
The [StarCoder](https://huggingface.co/bigcode/starcoder) models are 15.5B parameter models trained on 80+ programming languages from [The Stack (v1.2)](https://huggingface.co/datasets/bigcode/the-stack), with opt-out requests excluded.
|
| 26 |
+
Based on StarCoder, opencsg-starcoder-v0.1 was fintuned by OpenCSG LLM Research Team througth full-paramters fine-tuning method.
|
| 27 |
<br>
|
| 28 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 29 |
|
| 30 |
## Model Eval
|
| 31 |
|
|
|
|
| 36 |
Thus, OpenCSG strained our brains to provide a relatively fair method to compare the fine-tuned models on HumanEval benchmark.
|
| 37 |
To simplify the comparision, we chosed the Pass@1 metric on python language, but our finetuning dataset includes samples in multi language.
|
| 38 |
|
| 39 |
+
**For fair, we evaluated the fine-tuned and origin starcoder models only with the original cases' prompts, not including any other instruction else.**
|
| 40 |
|
| 41 |
**Otherwise, we use greedy decoding method for each model during the evaluation.**
|
| 42 |
|
| 43 |
| Model | HumanEval python pass@1 |
|
| 44 |
| --- |----------------------------------------------------------------------------- |
|
| 45 |
+
| starcoder | 35.98%|
|
| 46 |
+
| opencsg-starcoder-v0.1| **39.02%** |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 47 |
|
| 48 |
**TODO**
|
| 49 |
- we will provide much more benchmark scores on fine-tuned models in future.
|
|
|
|
| 59 |
import transformers
|
| 60 |
import torch
|
| 61 |
|
| 62 |
+
model = "opencsg/opencsg-starcoder-v0.1"
|
| 63 |
|
| 64 |
tokenizer = AutoTokenizer.from_pretrained(model, trust_remote_code=True)
|
| 65 |
pipeline = transformers.pipeline(
|
|
|
|
| 96 |
```
|
| 97 |
# Training
|
| 98 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 99 |
## Hardware
|
| 100 |
|
| 101 |
- **GPUs:** 8 Tesla A800
|
| 102 |
+
- **Training time:** 7 hours
|
| 103 |
|
| 104 |
## Software
|
| 105 |
|