File size: 1,340 Bytes
a864ceb d97087d c0a7d69 4fa0d0f 005c24f 4fa0d0f cbac22d 4fa0d0f |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
[CodeParrot](https://huggingface.co/lvwerra/codeparrot) uses GPT-2 architecture with BPE tokenizer trained on Python code from the training split of the data, and a context mength of 1024. We released this model as an educational tool for training large language models from scratch on code, with detailed tutorials and descriptions of the training process. It makes use of 🤗 [`accelerate`](https://huggingface.co/docs/accelerate/index) for distributed training and mixed precision. See this [blog](https://huggingface.co/blog/codeparrot) and [repo](https://github.com/huggingface/transformers/tree/main/examples/research_projects/codeparrot) for more details.
|Model | # parameters |
| - | - |
| GPT2 | 110M |
| GPT2 | 1.5B |
You can load the model and tokenizer directly from 🤗 [`transformers`](https://huggingface.co/docs/transformers/index):
```python
from transformers import AutoTokenizer, AutoModelWithLMHead
tokenizer = AutoTokenizer.from_pretrained("lvwerra/codeparrot")
model = AutoModelWithLMHead.from_pretrained("lvwerra/codeparrot")
inputs = tokenizer("def hello_world():", return_tensors="pt")
outputs = model(**inputs)
```
Or you can use a `pipeline`:
```python
from transformers import pipeline
pipe = pipeline("text-generation", model="lvwerra/codeparrot")
outputs = pipe("def hello_world():")
``` |