File size: 923 Bytes
db5608b 22c60ae db5608b 22c60ae db5608b |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
[CodeGen](https://huggingface.co/Salesforce/codegen-16B-mono) architecture follows a standard transformer decoder with left-to-right causal masking. With rotary position embedding for the positional encoding [(Su et al., 2021)](https://arxiv.org/abs/2104.09864), and a context length of 2048. CodeGen models are trained in various sizes.
<div align="center">
|Model | # parameters |
| - | - |
| Decoder | 350M |
| Decoder | 2.7B |
| Decoder | 6.1B |
| Decoder | 16.1B |
</div>
You can load the model and tokenizer directly from [`transformers`](https://huggingface.co/docs/transformers/index):
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained('Salesforce/codegen-16B-mono')
model = AutoModelForCausalLM.from_pretrained('Salesforce/codegen-16B-mono')
inputs = tokenizer("def hello_world():", return_tensors="pt")
outputs = model(**inputs)
``` |