|
--- |
|
datasets: |
|
- ML4SE2023-G1-WizardCoder/EvolInstruct-SCoT-1k |
|
language: |
|
- en |
|
tags: |
|
- code |
|
--- |
|
|
|
# ML4SE23_G1_WizardCoder-SCoT-1B-V1.0 |
|
|
|
IN4334 ML4SE |
|
|
|
Group1 WizardCoder |
|
|
|
This model is the result of the fine-tunign of the WizardCoder-1B-V1.0 model using Structured Chain-of-Though (S-CoT) enhanced instructions. |
|
S-CoT is used to enhance a sample of about 1200 entries from the Evol-Instruct 80k dataset. |
|
The resulting dataset is then used for the training task. |
|
The current WizardCoder model and the new S-CoT fine-tuned one are compared on both versions of HumanEval and MBPP (S-CoT enhanced and not) on the pass@1 metric. |
|
The S-CoT enhancement of the evaluation datasets allows to study its effect when used just as a prompting technique, independently of the S-CoT fine-tuning of the model. |
|
|
|
## Fine-tuning Details |
|
|
|
| Hyperparameter | [WizardCoder-1B-V1.0](https://huggingface.co/WizardLM/WizardCoder-1B-V1.0) | |
|
|----------------|---------------------| |
|
| Batch size | 16 | |
|
| Learning rate | 2e-5 | |
|
| Epochs | 3 | |
|
| Max length | 2048 | |
|
| Warmup step | 30 | |
|
| LR scheduler | cosine | |
|
| Dataset | [ML4SE23_G1_EvolInstruct-SCoT-1k](https://huggingface.co/datasets/ML4SE2023-G1-WizardCoder/ML4SE23_G1_EvolInstruct-SCoT-1k) | |
|
|
|
The hardware consisted on a GPU instance rented from [DataCrunch](https://datacrunch.io/) with the following specifications: |
|
|
|
| NVidia RTX A6000 48GB 1A6000.10V | |
|
|----------------------------------| |
|
| 2 GPUs | |
|
| 48GB VRAM per GPU | |
|
| 60 GB RAM | |
|
| 10 CPUs | |
|
| 100GB SSD Storage | |
|
| Ubuntu 20.04 | |
|
| CUDA 11.6 | |
|
|
|
## Results |
|
|
|
Results of pass@1(%) on HumanEval and MBPP compared to HumanEval-SCoT and MBPP-SCoT using WizardCoder-1B, WizardCoder-SCoT-1B and WizardCoder-15B. |
|
|
|
| **Dataset** | **WizardCoder-1B-V1.0** | **WizardCoder-SCoT-1B-V1.0** | **WizardCoder-15B-V1.0** | |
|
|----------------|-------------------------|------------------------------|--------------------------| |
|
| HumanEval | 23.78 | **17.68** | 57.3 | |
|
| HumanEval-SCoT | **44.51** | **27.44** | **57.3** | |
|
| MBPP | 23.4 | **19.4** | 51.8 | |
|
| MBPP-SCoT | **40** | **28** | **45.6** | |
|
|
