Update README.md
Browse files
README.md
CHANGED
@@ -3,9 +3,14 @@ language:
|
|
3 |
- ru
|
4 |
---
|
5 |
|
6 |
-
This is a base
|
7 |
-
|
8 |
-
|
|
|
|
|
|
|
|
|
|
|
9 |
|
10 |
The model can be used as-is to produce text embeddings or it can be further fine-tuned for a specific downstream task.
|
11 |
|
|
|
3 |
- ru
|
4 |
---
|
5 |
|
6 |
+
This is a base Longformer model designed for Russian language.
|
7 |
+
It was initialized from [blinoff/roberta-base-russian-v0](https://huggingface.co/blinoff/roberta-base-russian-v0) weights and has been modified to support a context length of up to 4096 tokens.
|
8 |
+
We fine-tuned it on a dataset of Russian books. For a detailed information check out our post on Habr.
|
9 |
+
|
10 |
+
Model attributes:
|
11 |
+
* 12 attention heads
|
12 |
+
* 12 hidden layers
|
13 |
+
* 4096 tokens length of context
|
14 |
|
15 |
The model can be used as-is to produce text embeddings or it can be further fine-tuned for a specific downstream task.
|
16 |
|