update architecture

architectures/codegen.txt

@@ -0,0 +1,20 @@
+[CodeGen](https://huggingface.co/Salesforce/codegen-16B-mono) architecture follows a standard transformer decoder with left-to-right causal masking. With rotary position embedding for the positional encoding [(Su et al., 2021)](https://arxiv.org/abs/2104.09864), and a context length of 2048. CodeGen models are trained in various sizes. 
+
+|Model | # parameters | 
+|   -   |   -  | 
+| Decoder | 350M |
+| Decoder | 2.7B |
+| Decoder | 6.1B |
+| Decoder | 16.1B |
+
+You can load the model and tokenizer directly from [`transformers`](https://huggingface.co/docs/transformers/index):
+
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM
+
+tokenizer = AutoTokenizer.from_pretrained('Salesforce/codegen-16B-mono')
+model = AutoModelForCausalLM.from_pretrained('Salesforce/codegen-16B-mono')
+
+inputs = tokenizer("def hello_world():", return_tensors="pt")
+outputs = model(**inputs)
+```

architectures/polycoder.txt

@@ -0,0 +1,9 @@
+[PolyCoder](https://github.com/VHellendoorn/Code-LMs) uses GPT2 architecture, with BPE tokenizer trained on a random 5% subset of the data (all languages), and a context mength of 2048. To study the effect of scaling of model size, the odel was trained in 3 different sizes.
+
+|Model | # parameters | 
+|   -   |   -  | 
+| GPT2 | 160M |
+| GPT2 | 400M |
+| GPT2 | 2.7B |
+
+PolyCoder is currently being integrated in `transformers`. Meanwhile it can be loaded following the instructions in the original Github [repo](https://github.com/vhellendoorn/code-lms#models).