|
--- |
|
language: |
|
- ru |
|
library_name: sentence-transformers |
|
tags: |
|
- sentence-transformers |
|
- sentence-similarity |
|
- feature-extraction |
|
widget: [] |
|
pipeline_tag: sentence-similarity |
|
license: apache-2.0 |
|
datasets: |
|
- deepvk/ru-HNP |
|
- deepvk/ru-WANLI |
|
- Shitao/bge-m3-data |
|
- RussianNLP/russian_super_glue |
|
- reciTAL/mlsum |
|
- Milana/russian_keywords |
|
- IlyaGusev/gazeta |
|
- d0rj/gsm8k-ru |
|
- bragovo/dsum_ru |
|
- CarlBrendt/Summ_Dialog_News |
|
--- |
|
|
|
# USER-bge-m3 |
|
|
|
|
|
**U**niversal **S**entence **E**ncoder for **R**ussian (USER) is a [sentence-transformer](https://www.SBERT.net) model for extracting embeddings exclusively for Russian language. |
|
It maps sentences & paragraphs to a 1024 dimensional dense vector space and can be used for tasks like clustering or semantic search. |
|
|
|
|
|
This model is initialized from [`TatonkaHF/bge-m3_en_ru`](https://huggingface.co/TatonkaHF/bge-m3_en_ru) which is shrinked version of [`baai/bge-m3`](https://huggingface.co/BAAI/bge-m3) model and trained to work mainly with the Russian language. Its quality on other languages was not evaluated. |
|
|
|
|
|
## Usage |
|
|
|
|
|
Using this model becomes easy when you have [`sentence-transformers`](https://www.SBERT.net) installed: |
|
|
|
|
|
``` |
|
pip install -U sentence-transformers |
|
``` |
|
|
|
|
|
Then you can use the model like this: |
|
|
|
|
|
```python |
|
from sentence_transformers import SentenceTransformer |
|
|
|
|
|
input_texts = [ |
|
"Когда был спущен на воду первый миноносец «Спокойный»?", |
|
"Есть ли нефть в Удмуртии?", |
|
"Спокойный (эсминец)\nЗачислен в списки ВМФ СССР 19 августа 1952 года.", |
|
"Нефтепоисковые работы в Удмуртии были начаты сразу после Второй мировой войны в 1945 году и продолжаются по сей день. Добыча нефти началась в 1967 году." |
|
] |
|
|
|
|
|
model = SentenceTransformer("deepvk/USER-bge-m3") |
|
embeddings = model.encode(input_texts, normalize_embeddings=True) |
|
``` |
|
|
|
|
|
However, you can use model directly with [`transformers`](https://huggingface.co/docs/transformers/en/index) |
|
|
|
|
|
```python |
|
import torch.nn.functional as F |
|
from torch import Tensor, inference_mode |
|
from transformers import AutoTokenizer, AutoModel |
|
|
|
|
|
input_texts = [ |
|
"Когда был спущен на воду первый миноносец «Спокойный»?", |
|
"Есть ли нефть в Удмуртии?", |
|
"Спокойный (эсминец)\nЗачислен в списки ВМФ СССР 19 августа 1952 года.", |
|
"Нефтепоисковые работы в Удмуртии были начаты сразу после Второй мировой войны в 1945 году и продолжаются по сей день. Добыча нефти началась в 1967 году." |
|
] |
|
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained("deepvk/USER-bge-m3") |
|
model = AutoModel.from_pretrained("deepvk/USER-bge-m3") |
|
model.eval() |
|
|
|
|
|
encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt') |
|
with torch.no_grad(): |
|
model_output = model(**encoded_input) |
|
# Perform pooling. In this case, cls pooling. |
|
sentence_embeddings = model_output[0][:, 0] |
|
|
|
# normalize embeddings |
|
sentence_embeddings = torch.nn.functional.normalize(sentence_embeddings, p=2, dim=1) |
|
|
|
# [[0.5567, 0.3014], |
|
# [0.1701, 0.7122]] |
|
scores = (sentence_embeddings[:2] @ sentence_embeddings[2:].T) |
|
``` |
|
|
|
Also, you can use native [FlagEmbedding](https://github.com/FlagOpen/FlagEmbedding) library for evaluation. Usage is described in [`bge-m3` model card](https://huggingface.co/BAAI/bge-m3). |
|
|
|
|
|
# Training Details |
|
|
|
|
|
We follow the [`USER-base`](https://huggingface.co/deepvk/USER-base) model training algorithm, with several changes as we use different backbone. |
|
|
|
|
|
**Initialization:** [`TatonkaHF/bge-m3_en_ru`](https://huggingface.co/TatonkaHF/bge-m3_en_ru) – shrinked version of [`baai/bge-m3`](https://huggingface.co/BAAI/bge-m3) to support only Russian and English tokens. |
|
|
|
|
|
**Fine-tuning:** Supervised fine-tuning two different models based on data symmetry and then merging via [`LM-Cocktail`](https://arxiv.org/abs/2311.13534): |
|
|
|
|
|
1. Since we split the data, we could additionally apply the [AnglE loss](https://arxiv.org/abs/2309.12871) to the symmetric model, which enhances performance on symmetric tasks. |
|
|
|
|
|
2. Finally, we added the original `bge-m3` model to the two obtained models to prevent catastrophic forgetting, tuning the weights for the merger using `LM-Cocktail` to produce the final model, **USER-bge-m3**. |
|
|
|
|
|
### Dataset |
|
|
|
|
|
During model development, we additional collect 2 datasets: |
|
[`deepvk/ru-HNP`](https://huggingface.co/datasets/deepvk/ru-HNP) and |
|
[`deepvk/ru-WANLI`](https://huggingface.co/datasets/deepvk/ru-WANLI). |
|
|
|
|
|
| Symmetric Dataset | Size | Asymmetric Dataset | Size | |
|
|-------------------|-------|--------------------|------| |
|
| **AllNLI** | 282 644 | [**MIRACL**](https://huggingface.co/datasets/Shitao/bge-m3-data/tree/main) | 10 000 | |
|
| [MedNLI](https://github.com/jgc128/mednli) | 3 699 | [MLDR](https://huggingface.co/datasets/Shitao/bge-m3-data/tree/main) | 1 864 | |
|
| [RCB](https://huggingface.co/datasets/RussianNLP/russian_super_glue) | 392 | [Lenta](https://github.com/yutkin/Lenta.Ru-News-Dataset) | 185 972 | |
|
| [Terra](https://huggingface.co/datasets/RussianNLP/russian_super_glue) | 1 359 | [Mlsum](https://huggingface.co/datasets/reciTAL/mlsum) | 51 112 | |
|
| [Tapaco](https://huggingface.co/datasets/tapaco) | 91 240 | [Mr-TyDi](https://huggingface.co/datasets/Shitao/bge-m3-data/tree/main) | 536 600 | |
|
| [**deepvk/ru-WANLI**](https://huggingface.co/datasets/deepvk/ru-WANLI) | 35 455 | [Panorama](https://huggingface.co/datasets/its5Q/panorama) | 11 024 | |
|
| [**deepvk/ru-HNP**](https://huggingface.co/datasets/deepvk/ru-HNP) | 500 000 | [PravoIsrael](https://huggingface.co/datasets/TarasHu/pravoIsrael) | 26 364 | |
|
| | | [Xlsum](https://huggingface.co/datasets/csebuetnlp/xlsum) | 124 486 | |
|
| | | [Fialka-v1](https://huggingface.co/datasets/0x7o/fialka-v1) | 130 000 | |
|
| | | [RussianKeywords](https://huggingface.co/datasets/Milana/russian_keywords) | 16 461 | |
|
| | | [Gazeta](https://huggingface.co/datasets/IlyaGusev/gazeta) | 121 928 | |
|
| | | [Gsm8k-ru](https://huggingface.co/datasets/d0rj/gsm8k-ru) | 7 470 | |
|
| | | [DSumRu](https://huggingface.co/datasets/bragovo/dsum_ru) | 27 191 | |
|
| | | [SummDialogNews](https://huggingface.co/datasets/CarlBrendt/Summ_Dialog_News) | 75 700 | |
|
|
|
|
|
|
|
|
|
**Total positive pairs:** 2,240,961 |
|
**Total negative pairs:** 792,644 (negative pairs from AIINLI, MIRACL, deepvk/ru-WANLI, deepvk/ru-HNP) |
|
|
|
|
|
For all labeled datasets, we only use its training set for fine-tuning. |
|
For datasets Gazeta, Mlsum, Xlsum: pairs (title/text) and (title/summary) are combined and used as asymmetric data. |
|
|
|
|
|
`AllNLI` is an translated to Russian combination of SNLI, MNLI and ANLI. |
|
|
|
|
|
## Experiments |
|
|
|
|
|
We compare our mode with the basic [`baai/bge-m3`](https://huggingface.co/BAAI/bge-m3) on the [`encodechka`](https://github.com/avidale/encodechka) benchmark. |
|
In addition, we evaluate model on the russian subset of [`MTEB`](https://github.com/embeddings-benchmark/mteb) on Classification, Reranking, Multilabel Classification, STS, Retrieval, and PairClassification tasks. |
|
We use validation scripts from the official repositories for each of the tasks. |
|
|
|
|
|
Results on encodechka: |
|
| Model | Mean S | Mean S+W | STS | PI | NLI | SA | TI | IA | IC | ICX | NE1 | NE2 | |
|
|-------------|--------|----------|------|------|------|------|------|------|------|------|------|------| |
|
| [`baai/bge-m3`](https://huggingface.co/BAAI/bge-m3) | 0.787 | 0.696 | 0.86 | 0.75 | 0.51 | 0.82 | 0.97 | 0.79 | 0.81 | 0.78 | 0.24 | 0.42 | |
|
| `USER-bge-m3` | **0.799** | **0.709** | **0.87** | **0.76** | **0.58** | 0.82 | 0.97 | 0.79 | 0.81 | 0.78 | **0.28** | **0.43** | |
|
|
|
|
|
Results on MTEB: |
|
|
|
|
|
| Type | [`baai/bge-m3`](https://huggingface.co/BAAI/bge-m3) | `USER-bge-m3` | |
|
|---------------------------|--------|-------------| |
|
| Average (30 datasets) | 0.689 | **0.706** | |
|
| Classification Average (12 datasets) | 0.571 | **0.594** | |
|
| Reranking Average (2 datasets) | **0.698** | 0.688 | |
|
| MultilabelClassification (2 datasets) | 0.343 | **0.359** | |
|
| STS Average (4 datasets) | 0.735 | **0.753** | |
|
| Retrieval Average (6 datasets) | **0.945** | 0.934 | |
|
| PairClassification Average (4 datasets) | 0.784 | **0.833** | |
|
|
|
|
|
## Limitations |
|
|
|
|
|
We did not thoroughly evaluate the model's ability for sparse and multi-vec encoding. |
|
|
|
|
|
## Citations |
|
|
|
|
|
``` |
|
@misc{deepvk2024user, |
|
title={USER: Universal Sentence Encoder for Russian}, |
|
author={Malashenko, Boris and Zemerov, Anton and Spirin, Egor}, |
|
url={https://huggingface.co/datasets/deepvk/USER-base}, |
|
publisher={Hugging Face} |
|
year={2024}, |
|
} |
|
``` |
|
|
