File size: 3,344 Bytes
d7d0d9b 3069b27 d207946 3069b27 d7d0d9b a4451a4 d207946 d490b51 d207946 99ffd4c d207946 4a54bba d207946 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 |
---
license: apache-2.0
datasets:
- verifiers-for-code/CodeNet-16K
- verifiers-for-code/CodeNet-Planner
language:
- en
library_name: transformers
tags:
- code
---
# 🦙 Llama-3-LlamaPlanner
## Model Description
LlamaPlanner is a fine-tuned version of Meta's Llama-8B model which has been specifically designed for generating high-quality plans for code generation tasks. The model was trained on CodeNet-16k, a curated dataset of competitive programming problems, and their corresponding plans generated using Llama-3-70B. By leveraging the power of Parameter Efficient Fine-Tuning (PEFT), LlamaPlanner achieves performance comparable to much larger models in generating effective plans for code generation.
## Model Details
- **Base Model:** Llama-8B Instruct
- **Fine-Tuning Approach:** Parameter Efficient Fine-Tuning (PEFT) using Unsloth
- **Training Data:** CodeNet-16k, a filtered and deduplicated dataset of 16,500 competitive programming problems and their plans generated using Llama-3-70B
- **Training Infrastructure:** H100-SXM5 GPU
- **Evaluation Benchmarks:** HumanEval and EvalPlus
## How to Use
To use LlamaPlanner with the Hugging Face Transformers library, follow these steps:
```python
import transformers
import torch
model_id = "verifiers-for-code/Llama-3-LlamaPlanner"
pipeline = transformers.pipeline(
"text-generation", model=model_id, model_kwargs={"torch_dtype": torch.bfloat16}, device_map="auto"
)
prompt = "Generate a plan for a program that sorts an array of integers in ascending order."
pipeline(prompt)
```
## Training Details
LlamaPlanner was trained using the following steps:
1. Filtering and preprocessing the CodeNet dataset to create CodeNet-16k
2. Generating plans for each problem using Llama-3-70B
3. Formatting the problem description, input description, output description, and samples as input, and the generated plans as output
4. Performing PEFT on the Llama-8B Instruct base model using Unsloth with different ranks and alpha values
5. Training on an H100-SXM5 GPU for varying epochs
## Evaluation Results
LlamaPlanner was evaluated on the HumanEval and EvalPlus benchmarks using various methods, including zero-shot, self-planning, base planner model, and fine-tuned planner model. The results demonstrated that LlamaPlanner outperforms the base Llama-3-8B model by 14% on HumanEval and 11% on EvalPlus. Additionally, plans generated by LlamaPlanner helped boost the performance of Llama-3-70B on HumanEval.
## Citation
If you use LlamaPlanner in your research or applications, please cite the model using the following BibTeX entry:
```bibtex
@misc{llamaplanner,
title={LlamaPlanner: A Fine-Tuned Llama-8B Model for Effective Plan Generation in Code Generation Tasks},
author={Abhinav Chinta and Sumuk Shashidhar and Vaibhav Sahai},
year={2023},
howpublished={\url{https://huggingface.co/verifiers-for-code/LlamaPlanner}},
}
```
## License
LlamaPlanner is released under the Apache License 2.0.
## Acknowledgements
We would like to thank Meta for releasing the Llama model family and the open-source community for their contributions to the development of large language models and their applications in code generation tasks. |