|
--- |
|
pipeline_tag: sentence-similarity |
|
language: fr |
|
datasets: |
|
- stsb_multi_mt |
|
tags: |
|
- Text |
|
- Sentence Similarity |
|
- Sentence-Embedding |
|
- camembert-large |
|
license: apache-2.0 |
|
model-index: |
|
- name: sentence-camembert-large by Van Tuan DANG |
|
results: |
|
- task: |
|
name: Sentence-Embedding |
|
type: Text Similarity |
|
dataset: |
|
name: Text Similarity fr |
|
type: stsb_multi_mt |
|
args: fr |
|
metrics: |
|
- name: Test Pearson correlation coefficient |
|
type: Pearson_correlation_coefficient |
|
value: 88.63 |
|
--- |
|
## Description: |
|
This [**Sentence-CamemBERT-Large**](https://huggingface.co/Lajavaness/sentence-camembert-large) Model is an Embedding Model for French developed by [La Javaness](https://www.lajavaness.com/). The purpose of this embedding model is to represent the content and semantics of a French sentence as a mathematical vector, allowing it to understand the meaning of the text beyond individual words in queries and documents. It offers powerful semantic search capabilities. |
|
## Pre-trained sentence embedding models are state-of-the-art of Sentence Embeddings for French. |
|
|
|
The [Lajavaness/sentence-camembert-large](https://huggingface.co/Lajavaness/sentence-camembert-large) model is an improvement over the [dangvantuan/sentence-camembert-base](https://huggingface.co/dangvantuan/sentence-camembert-large) offering greater robustness and better performance on all STS benchmark datasets. It has been fine-tuned using the pre-trained [facebook/camembert-large](https://huggingface.co/camembert/camembert-large) and |
|
[Siamese BERT-Networks with 'sentences-transformers'](https://www.sbert.net/) on dataset [stsb](https://huggingface.co/datasets/stsb_multi_mt/viewer/fr/train). Additionally, it has been combined with [Augmented SBERT](https://aclanthology.org/2021.naacl-main.28.pdf) on dataset [stsb](https://huggingface.co/datasets/stsb_multi_mt/viewer/fr/train). The model benefits from Pair Sampling Strategies using two models: [CrossEncoder-camembert-large](https://huggingface.co/dangvantuan/CrossEncoder-camembert-large) and [dangvantuan/sentence-camembert-large](https://huggingface.co/dangvantuan/sentence-camembert-large) |
|
|
|
## Usage |
|
The model can be used directly (without a language model) as follows: |
|
|
|
```python |
|
from sentence_transformers import SentenceTransformer |
|
model = SentenceTransformer("Lajavaness/sentence-camembert-large") |
|
|
|
sentences = ["Un avion est en train de décoller.", |
|
"Un homme joue d'une grande flûte.", |
|
"Un homme étale du fromage râpé sur une pizza.", |
|
"Une personne jette un chat au plafond.", |
|
"Une personne est en train de plier un morceau de papier.", |
|
] |
|
|
|
embeddings = model.encode(sentences) |
|
``` |
|
|
|
## Evaluation |
|
The model can be evaluated as follows on the French test data of stsb. |
|
|
|
```python |
|
from sentence_transformers import SentenceTransformer |
|
from sentence_transformers.readers import InputExample |
|
from datasets import load_dataset |
|
def convert_dataset(dataset): |
|
dataset_samples=[] |
|
for df in dataset: |
|
score = float(df['similarity_score'])/5.0 # Normalize score to range 0 ... 1 |
|
inp_example = InputExample(texts=[df['sentence1'], |
|
df['sentence2']], label=score) |
|
dataset_samples.append(inp_example) |
|
return dataset_samples |
|
|
|
# Loading the dataset for evaluation |
|
df_dev = load_dataset("stsb_multi_mt", name="fr", split="dev") |
|
df_test = load_dataset("stsb_multi_mt", name="fr", split="test") |
|
|
|
# Convert the dataset for evaluation |
|
|
|
# For Dev set: |
|
dev_samples = convert_dataset(df_dev) |
|
val_evaluator = EmbeddingSimilarityEvaluator.from_input_examples(dev_samples, name='sts-dev') |
|
val_evaluator(model, output_path="./") |
|
|
|
# For Test set: |
|
test_samples = convert_dataset(df_test) |
|
test_evaluator = EmbeddingSimilarityEvaluator.from_input_examples(test_samples, name='sts-test') |
|
test_evaluator(model, output_path="./") |
|
``` |
|
|
|
**Test Result**: |
|
The performance is measured using Pearson and Spearman correlation: |
|
- On dev |
|
|
|
|
|
| Model | Pearson correlation | Spearman correlation | #params | |
|
| ------------- | ------------- | ------------- |------------- | |
|
| [Lajavaness/sentence-camembert-large](https://huggingface.co/dangvantuan/sentence-camembert-large)| **88.63** |**88.46** | 336M| |
|
| [dangvantuan/sentence-camembert-large](https://huggingface.co/dangvantuan/sentence-camembert-large)| 88.2 |88.02 | 336M| |
|
| [Sahajtomar/french_semanti](https://huggingface.co/Sahajtomar/french_semantic)| 87.44 |87.30 | 336M| |
|
| [Lajavaness/sentence-flaubert-base](https://huggingface.co/Lajavaness/sentence-flaubert-base)| 87.14 |87.10 | 137M | |
|
| [GPT-3 (text-davinci-003)](https://platform.openai.com/docs/models) | 85 | NaN|175B | |
|
| [GPT-(text-embedding-ada-002)](https://platform.openai.com/docs/models) | 79.75 | 80.44|NaN | |
|
|
|
|
|
- On test, Pearson and Spearman correlation are evaluated on many different benchmark datasets: |
|
|
|
|
|
**Pearson score** |
|
| Model | [STS-B](https://huggingface.co/datasets/stsb_multi_mt/viewer/fr/train) | [STS12-fr ](https://huggingface.co/datasets/Lajavaness/STS12-fr)| [STS13-fr](https://huggingface.co/datasets/Lajavaness/STS13-fr) | [STS14-fr](https://huggingface.co/datasets/Lajavaness/STS14-fr) | [STS15-fr](https://huggingface.co/datasets/Lajavaness/STS15-fr) | [STS16-fr](https://huggingface.co/datasets/Lajavaness/STS16-fr) | [SICK-fr](https://huggingface.co/datasets/Lajavaness/SICK-fr) | params | |
|
|------------------------------------------|-------|----------|----------|----------|----------|----------|---------|--------| |
|
| [Lajavaness/sentence-camembert-large](https://huggingface.co/dangvantuan/sentence-camembert-large) | **86.26** | **87.42** | **89.34** | **88.05** | **88.91** | 77.15 | 83.13 | 336M | |
|
| [dangvantuan/sentence-camembert-large](https://huggingface.co/dangvantuan/sentence-camembert-large) | 85.88 | 87.28 | 89.25 | 87.91 | 88.54 | 76.90 | 83.26 | 336M | |
|
| [Sahajtomar/french_semantic](https://huggingface.co/Sahajtomar/french_semantic) | 85.80 | 86.05 | 88.50 | 86.57 | 87.49 | 77.85 | 83.27 | 336M | |
|
| [Lajavaness/sentence-flaubert-base](https://huggingface.co/Lajavaness/sentence-flaubert-base) | 85.39 | 86.64 | 87.24 | 85.68 | 87.99 | 75.78 | 82.84 | 137M | |
|
| [GPT3 (text-embedding-ada-002)](https://platform.openai.com/docs/models) | 79.03 | 66.16 | 75.48 | 70.69 | 77.88 | 65.18 | - | - | |
|
|
|
|
|
**Spearman score** |
|
| Model | [STS-B](https://huggingface.co/datasets/stsb_multi_mt/viewer/fr/train) | [STS12-fr ](https://huggingface.co/datasets/Lajavaness/STS12-fr)| [STS13-fr](https://huggingface.co/datasets/Lajavaness/STS13-fr) | [STS14-fr](https://huggingface.co/datasets/Lajavaness/STS14-fr) | [STS15-fr](https://huggingface.co/datasets/Lajavaness/STS15-fr) | [STS16-fr](https://huggingface.co/datasets/Lajavaness/STS16-fr) | [SICK-fr](https://huggingface.co/datasets/Lajavaness/SICK-fr) | params | |
|
|:-------------------------------------|-------:|---------:|---------:|---------:|---------:|---------:|--------:|:-------| |
|
| [Lajavaness/sentence-camembert-large](https://huggingface.co/dangvantuan/sentence-camembert-large) | **86.14** | **81.22** | 88.61 | **86.28** | **89.01** | 78.65 | **77.71** | 336M | |
|
| [dangvantuan/sentence-camembert-large](https://huggingface.co/dangvantuan/sentence-camembert-large) | 85.78 | 81.09 | 88.68 | 85.81 | 88.56 | 78.49 | 77.70 | 336M | |
|
| [Sahajtomar/french_semantic](https://huggingface.co/Sahajtomar/french_semantic) | 85.55 | 77.92 | 87.85 | 83.96 | 87.63 | 79.07 | 77.14 | 336M | |
|
| [Lajavaness/sentence-flaubert-base](https://huggingface.co/Lajavaness/sentence-flaubert-base) | 85.67 | 79.97 | 86.91 | 84.57 | 88.10 | 77.84 | 77.55 | 137M | |
|
| [GPT3 (text-embedding-ada-002)](https://platform.openai.com/docs/models) | 77.53 | 64.27 | 76.41 | 69.63 | 78.65 | 75.30 | - | - | |
|
|
|
|
|
|
|
## Citation |
|
|
|
|
|
@article{reimers2019sentence, |
|
title={Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks}, |
|
author={Nils Reimers, Iryna Gurevych}, |
|
journal={https://arxiv.org/abs/1908.10084}, |
|
year={2019} |
|
} |
|
|
|
|
|
@article{martin2020camembert, |
|
title={CamemBERT: a Tasty French Language Mode}, |
|
author={Martin, Louis and Muller, Benjamin and Su{\'a}rez, Pedro Javier Ortiz and Dupont, Yoann and Romary, Laurent and de la Clergerie, {\'E}ric Villemonte and Seddah, Djam{\'e} and Sagot, Beno{\^\i}t}, |
|
journal={Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics}, |
|
year={2020} |
|
} |
