The CodeGen architecture follows a standard transformer decoder with left-to-right causal masking. With rotary position embedding for the positional encoding (Su et al., 2021), and a context length of 2048. CodeGen models are trained in various sizes.
| Model | # parameters |
|---|---|
| Salesforce/codegen-350m-mono | 350M |
| Salesforce/codegen-2B-mono | 2.7B |
| Salesforce/codegen-6B-mono | 6.1B |
| Salesforce/codegen-16B-mono | 16.1B |
You can load the model and tokenizer directly from 🤗 transformers:
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained('Salesforce/codegen-16B-mono')
model = AutoModelForCausalLM.from_pretrained('Salesforce/codegen-16B-mono')
inputs = tokenizer("def hello_world():", return_tensors="pt")
outputs = model(**inputs)