| [CodeGen](https://huggingface.co/Salesforce/codegen-16B-mono) architecture follows a standard transformer decoder with left-to-right causal masking. With rotary position embedding for the positional encoding [(Su et al., 2021)](https://arxiv.org/abs/2104.09864), and a context length of 2048. CodeGen models are trained in various sizes. | |
| |Model | # parameters | | |
| | - | - | | |
| | Decoder | 350M | | |
| | Decoder | 2.7B | | |
| | Decoder | 6.1B | | |
| | Decoder | 16.1B | | |
| You can load the model and tokenizer directly from [`transformers`](https://huggingface.co/docs/transformers/index): | |
| ```python | |
| from transformers import AutoTokenizer, AutoModelForCausalLM | |
| tokenizer = AutoTokenizer.from_pretrained('Salesforce/codegen-16B-mono') | |
| model = AutoModelForCausalLM.from_pretrained('Salesforce/codegen-16B-mono') | |
| inputs = tokenizer("def hello_world():", return_tensors="pt") | |
| outputs = model(**inputs) | |
| ``` |