Update README.md
Browse files
README.md
CHANGED
@@ -13,12 +13,11 @@ datasets:
|
|
13 |
|
14 |
# MiniLMv2-L6-H384_R-fineweb-100k
|
15 |
|
16 |
-
This is a MiniLMv2 model
|
17 |
|
18 |
- activation updated to SiLU prior to further training
|
19 |
- MLM @ 40% mask ratio
|
20 |
|
21 |
-
-
|
22 |
## Model description
|
23 |
|
24 |
This model is a fine-tuned version of [nreimers/MiniLMv2-L6-H384-distilled-from-RoBERTa-Large](https://huggingface.co/nreimers/MiniLMv2-L6-H384-distilled-from-RoBERTa-Large) on the BEE-spoke-data/fineweb-100k_en-med dataset.
|
|
|
13 |
|
14 |
# MiniLMv2-L6-H384_R-fineweb-100k
|
15 |
|
16 |
+
This is a MiniLMv2 model continually pre-trained on an MLM task with the goal of improving downstream fine-tuning/performance:
|
17 |
|
18 |
- activation updated to SiLU prior to further training
|
19 |
- MLM @ 40% mask ratio
|
20 |
|
|
|
21 |
## Model description
|
22 |
|
23 |
This model is a fine-tuned version of [nreimers/MiniLMv2-L6-H384-distilled-from-RoBERTa-Large](https://huggingface.co/nreimers/MiniLMv2-L6-H384-distilled-from-RoBERTa-Large) on the BEE-spoke-data/fineweb-100k_en-med dataset.
|