update html
Browse files
architectures/polycoder.txt
CHANGED
|
@@ -1,9 +1,14 @@
|
|
| 1 |
[PolyCoder](https://github.com/VHellendoorn/Code-LMs) uses GPT2 architecture, with BPE tokenizer trained on a random 5% subset of the data (all languages), and a context mength of 2048. To study the effect of scaling of model size, the odel was trained in 3 different sizes.
|
| 2 |
|
|
|
|
|
|
|
| 3 |
|Model | # parameters |
|
| 4 |
| - | - |
|
| 5 |
| GPT2 | 160M |
|
| 6 |
| GPT2 | 400M |
|
| 7 |
| GPT2 | 2.7B |
|
| 8 |
|
|
|
|
|
|
|
|
|
|
| 9 |
PolyCoder is currently being integrated in `transformers`. Meanwhile it can be loaded following the instructions in the original Github [repo](https://github.com/vhellendoorn/code-lms#models).
|
|
|
|
| 1 |
[PolyCoder](https://github.com/VHellendoorn/Code-LMs) uses GPT2 architecture, with BPE tokenizer trained on a random 5% subset of the data (all languages), and a context mength of 2048. To study the effect of scaling of model size, the odel was trained in 3 different sizes.
|
| 2 |
|
| 3 |
+
<div align="center">
|
| 4 |
+
|
| 5 |
|Model | # parameters |
|
| 6 |
| - | - |
|
| 7 |
| GPT2 | 160M |
|
| 8 |
| GPT2 | 400M |
|
| 9 |
| GPT2 | 2.7B |
|
| 10 |
|
| 11 |
+
</div>
|
| 12 |
+
|
| 13 |
+
|
| 14 |
PolyCoder is currently being integrated in `transformers`. Meanwhile it can be loaded following the instructions in the original Github [repo](https://github.com/vhellendoorn/code-lms#models).
|