|
--- |
|
language: |
|
- ru |
|
license: apache-2.0 |
|
pipeline_tag: text-generation |
|
--- |
|
|
|
# BulgakovLM 3B |
|
|
|
A language model trained on Russian. May be suitable for further tuning. The 100 gigabyte dataset consisted primarily of web pages, books, poems, and prose. The model was trained over 2 epochs. |
|
|
|
Uses GPT-J architecture with a context window of 4k tokens. |
|
|
|
Trained thanks to a TRC grant on TPU-VM v3-8 |
|
|
|
# Usage |
|
```python |
|
from transformers import AutoTokenizer, AutoModelForCausalLM |
|
import torch |
|
|
|
tokenizer = AutoTokenizer.from_pretrained("0x7o/BulgakovLM-3B") |
|
model = AutoModelForCausalLM.from_pretrained("0x7o/BulgakovLM-3B") |
|
|
|
input_ids = tokenizer("Искусственный интеллект - это", return_tensors='pt').to(model.device)["input_ids"] |
|
output = model.generate(input_ids, max_new_tokens=48, do_sample=True, temperature=0.7) |
|
print(tokenizer.decode(output[0])) |
|
``` |
|
Output: |
|
``` |
|
Искусственный интеллект - это всего-навсего программа, которая анализирует данные и решает, насколько тот или иной выбор может оказаться оптимальным. Как и во всех остальных сферах человеческой деятельности, в IT есть свои плюсы и минусы. И если в прошлом веке искусственный интеллект был чем |
|
``` |
|
|
|
# Evaluation |
|
The results are obtained through the Russian-language benchmark [MERA](https://mera.a-ai.ru/ru) |
|
|
|
Total score: 0.198 |
|
|
|
| Задача | Результат | Метрика | |
|
|--------------|---------------|--------------------| |
|
| BPS | 0.44 | Accuracy | |
|
| LCS | 0.118 | Accuracy | |
|
| RCB | 0.333 / 0.167 | Avg. F1 / Accuracy | |
|
| USE | 0 | Grade Norm | |
|
| RWSD | 0.523 | Accuracy | |
|
| PARus | 0.498 | Accuracy | |
|
| ruTiE | 0.5 | Accuracy | |
|
| MultiQ | 0.059 / 0.007 | F1-score/EM | |
|
| ruMMLU | 0.25 | Accuracy | |
|
| CheGeKa | 0.006 / 0 | F1 / EM | |
|
| ruModAr | 0.001 | Accuracy | |
|
| SimpleAr | 0.001 | Accuracy | |
|
| ruMultiAr | 0.011 | Accuracy | |
|
| MathLogicQA | 0.245 | Accuracy | |
|
| ruHumanEval | 0 / 0 / 0 | pass@k | |
|
| ruWorldTree | 0.265 / 0.246 | Avg. F1 / Accuracy | |
|
| ruOpenBookQA | 0.24 / 0.221 | Avg. F1 / Accuracy | |
|
|
