README.md

---
library_name: transformers
tags:
- nli
- bert
- natural-language-inference
language:
- ru
metrics:
- accuracy
- f1
- precision
- recall
base_model:
- cointegrated/rubert-tiny2
pipeline_tag: text-classification
model-index:
- name: rubert-tiny-nli-terra-v0
  results:
  - task:
      type: text-classification
      name: Text Classification
    dataset:
      name: TERRA
      type: NLI
      split: validation
    metrics:
    - type: accuracy
      value: 0.6677524429967426
      name: Accuracy
    - type: f1
      value: 0.6666666666666666
      name: F1
    - type: precision
      value: 0.6666666666666666
      name: Precision
    - type: recall
      value: 0.6666666666666666
      name: Recall
---

**⚠️ Disclaimer: This model is in the early stages of development and may produce low-quality predictions. For better results, consider using the recommended Russian natural language inference models available [here](https://huggingface.co/cointegrated).**

# RuBERT-tiny-nli v0

This model is an initial attempt to fine-tune the [RuBERT-tiny2](https://huggingface.co/cointegrated/rubert-tiny2) model for a two-way natural language inference task, utilizing the Russian [Textual Entailment Recognition](https://russiansuperglue.com/tasks/task_info/TERRa) dataset. While it aims to enhance understanding of Russian text, its performance is currently limited.


## Usage
How to run the model for NLI:

```python
# !pip install transformers sentencepiece --quiet
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_id = 'Marwolaeth/rubert-tiny-nli-terra-v0'
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)
if torch.cuda.is_available():
    model.cuda()

# An example from the base model card
premise1 = 'Сократ - человек, а все люди смертны.'
hypothesis1 = 'Сократ никогда не умрёт.'
with torch.inference_mode():
    prediction = model(
      **tokenizer(premise1, hypothesis1, return_tensors='pt').to(model.device)
    )
    p = torch.softmax(prediction.logits, -1).cpu().numpy()[0]
print({v: p[k] for k, v in model.config.id2label.items()})
# {'not_entailment': 0.7698182, 'entailment': 0.23018183}

# An example concerning sentiments
premise2 = 'Я ненавижу желтые занавески'
hypothesis2 = 'Мне нравятся желтые занавески'
with torch.inference_mode():
    prediction = model(
      **tokenizer(premise2, hypothesis2, return_tensors='pt').to(model.device)
    )
    p = torch.softmax(prediction.logits, -1).cpu().numpy()[0]
print({v: p[k] for k, v in model.config.id2label.items()})
# {'not_entailment': 0.60584205, 'entailment': 0.3941579}
```

## Model Performance Metrics

The following metrics summarize the performance of the model on the test dataset:

| Metric                           | Value                     |
|----------------------------------|---------------------------|
| **Validation Loss**              | 0.6261                    |
| **Validation Accuracy**          | 66.78%                    |
| **Validation F1 Score**          | 66.67%                    |
| **Validation Precision**         | 66.67%                    |
| **Validation Recall**            | 66.67%                    |
| **Validation Runtime***          | 0.7043 seconds            |
| **Samples per Second***          | 435.88                    |
| **Steps per Second***            | 14.20                     |

*Using T4 GPU with Google Colab