Commit
•
e8086d7
1
Parent(s):
151881c
Update README.md
Browse files
README.md
CHANGED
|
@@ -46,7 +46,7 @@ Supported languages (9 in total) are as follows:
|
|
| 46 |
|
| 47 |
## Training procedure
|
| 48 |
|
| 49 |
-
This checkpoint is initialized from off-the-shelf LLMs, i.e. its encoder is initialized from CodeGen-350M-mono and its decoder is initialized from CodeGen-16B-mono.
|
| 50 |
It is trained on the unimodal code data at the first-stage pretraining, which includes a diverse set of pretraining tasks including _span denoising_ and two variants of _causal language modeling_.
|
| 51 |
After that, it is further trained on the Python subset with the causal language modeling objective for another epoch to better adapt for Python code generation.
|
| 52 |
Please refer to the paper for more details.
|
|
|
|
| 46 |
|
| 47 |
## Training procedure
|
| 48 |
|
| 49 |
+
This checkpoint is initialized from off-the-shelf LLMs, i.e. its encoder is initialized from [CodeGen-350M-mono](https://huggingface.co/Salesforce/codegen-350M-mono) and its decoder is initialized from [CodeGen-16B-mono](https://huggingface.co/Salesforce/codegen-16B-mono).
|
| 50 |
It is trained on the unimodal code data at the first-stage pretraining, which includes a diverse set of pretraining tasks including _span denoising_ and two variants of _causal language modeling_.
|
| 51 |
After that, it is further trained on the Python subset with the causal language modeling objective for another epoch to better adapt for Python code generation.
|
| 52 |
Please refer to the paper for more details.
|