README.md

metadata

library_name: transformers
tags:
  - nli
  - bert
  - natural-language-inference
language:
  - ru
metrics:
  - accuracy
  - f1
  - precision
  - recall
base_model:
  - cointegrated/rubert-tiny2
pipeline_tag: text-classification
model-index:
  - name: rubert-tiny-nli-terra-v0
    results:
      - task:
          type: text-classification
          name: Text Classification
        dataset:
          name: TERRA
          type: NLI
          split: validation
        metrics:
          - type: accuracy
            value: 0.6677524429967426
            name: Accuracy
          - type: macro f1
            value: 0.6666666666666666
            name: Macro F1
          - type: macro precision
            value: 0.6666666666666666
            name: Macro Precision
          - type: macro recall
            value: 0.6666666666666666
            name: Macro Recall

⚠️ Disclaimer: This model is in the early stages of development and may produce low-quality predictions. For better results, consider using the recommended Russian natural language inference models available here.

RuBERT-tiny-nli v0

This model is an initial attempt to fine-tune the RuBERT-tiny2 model for a two-way natural language inference task, utilizing the Russian Textual Entailment Recognition dataset. While it aims to enhance understanding of Russian text, its performance is currently limited.

Usage

How to run the model for NLI:

# !pip install transformers sentencepiece --quiet
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_id = 'Marwolaeth/rubert-tiny-nli-terra-v0'
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)
if torch.cuda.is_available():
    model.cuda()

# An example from the base model card
premise1 = 'Сократ - человек, а все люди смертны.'
hypothesis1 = 'Сократ никогда не умрёт.'
with torch.inference_mode():
    prediction = model(
      **tokenizer(premise1, hypothesis1, return_tensors='pt').to(model.device)
    )
    p = torch.softmax(prediction.logits, -1).cpu().numpy()[0]
print({v: p[k] for k, v in model.config.id2label.items()})
# {'not_entailment': 0.7698182, 'entailment': 0.23018183}

# An example concerning sentiments
premise2 = 'Я ненавижу желтые занавески'
hypothesis2 = 'Мне нравятся желтые занавески'
with torch.inference_mode():
    prediction = model(
      **tokenizer(premise2, hypothesis2, return_tensors='pt').to(model.device)
    )
    p = torch.softmax(prediction.logits, -1).cpu().numpy()[0]
print({v: p[k] for k, v in model.config.id2label.items()})
# {'not_entailment': 0.60584205, 'entailment': 0.3941579}

Model Performance Metrics

The following metrics summarize the performance of the model on the test dataset:

Metric	Value
Validation Loss	0.6261
Validation Accuracy	66.78%
Validation F1 Score	66.67%
Validation Precision	66.67%
Validation Recall	66.67%
Validation Runtime*	0.7043 seconds
Samples per Second*	435.88
Steps per Second*	14.20

*Using T4 GPU with Google Colab