Duplicate from Lajavaness/sentence-camembert-large

Co-authored-by: Van Tuan DANG <dangvantuan@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

1_Pooling/config.json

@@ -0,0 +1,7 @@
+{
+  "word_embedding_dimension": 1024,
+  "pooling_mode_cls_token": false,
+  "pooling_mode_mean_tokens": true,
+  "pooling_mode_max_tokens": false,
+  "pooling_mode_mean_sqrt_len_tokens": false
+}

README.md

@@ -0,0 +1,139 @@
+---
+pipeline_tag: sentence-similarity
+language: fr
+datasets:
+- stsb_multi_mt
+tags:
+- Text
+- Sentence Similarity
+- Sentence-Embedding
+- camembert-large
+license: apache-2.0
+model-index:
+- name: sentence-camembert-large by Van Tuan DANG
+  results:
+  - task: 
+      name: Sentence-Embedding
+      type: Text Similarity
+    dataset:
+      name: Text Similarity fr
+      type: stsb_multi_mt
+      args: fr
+    metrics:
+       - name: Test Pearson correlation coefficient
+         type: Pearson_correlation_coefficient
+         value:  88.63
+---
+## Description:
+This [**Sentence-CamemBERT-Large**](https://huggingface.co/Lajavaness/sentence-camembert-large) Model is an Embedding Model for French developed by [La Javaness](https://www.lajavaness.com/).  The purpose of this embedding model is to represent the content and semantics of a French sentence as a mathematical vector, allowing it to understand the meaning of the text beyond individual words in queries and documents. It offers powerful semantic search capabilities.
+## Pre-trained sentence embedding models are state-of-the-art of Sentence Embeddings for French.
+
+The [Lajavaness/sentence-camembert-large](https://huggingface.co/Lajavaness/sentence-camembert-large) model is an improvement over the [dangvantuan/sentence-camembert-base](https://huggingface.co/dangvantuan/sentence-camembert-large)  offering greater robustness and better performance on all STS benchmark datasets. It has been fine-tuned using the pre-trained [facebook/camembert-large](https://huggingface.co/camembert/camembert-large) and
+[Siamese BERT-Networks with 'sentences-transformers'](https://www.sbert.net/) on dataset [stsb](https://huggingface.co/datasets/stsb_multi_mt/viewer/fr/train).  Additionally, it has been combined with [Augmented SBERT](https://aclanthology.org/2021.naacl-main.28.pdf) on dataset [stsb](https://huggingface.co/datasets/stsb_multi_mt/viewer/fr/train). The model benefits from Pair Sampling Strategies using two models: [CrossEncoder-camembert-large](https://huggingface.co/dangvantuan/CrossEncoder-camembert-large) and [dangvantuan/sentence-camembert-large](https://huggingface.co/dangvantuan/sentence-camembert-large)
+
+## Usage
+The model can be used directly (without a language model) as follows:
+
+```python
+from sentence_transformers import SentenceTransformer
+model =  SentenceTransformer("Lajavaness/sentence-camembert-large")
+
+sentences = ["Un avion est en train de décoller.",
+          "Un homme joue d'une grande flûte.",
+          "Un homme étale du fromage râpé sur une pizza.",
+          "Une personne jette un chat au plafond.",
+          "Une personne est en train de plier un morceau de papier.",
+          ]
+
+embeddings = model.encode(sentences)
+```
+
+## Evaluation
+The model can be evaluated as follows on the French test data of stsb.
+
+```python
+from sentence_transformers import SentenceTransformer
+from sentence_transformers.readers import InputExample
+from datasets import load_dataset
+def convert_dataset(dataset):
+    dataset_samples=[]
+    for df in dataset:
+        score = float(df['similarity_score'])/5.0  # Normalize score to range 0 ... 1
+        inp_example = InputExample(texts=[df['sentence1'], 
+                                    df['sentence2']], label=score)
+        dataset_samples.append(inp_example)
+    return dataset_samples
+
+# Loading the dataset for evaluation
+df_dev = load_dataset("stsb_multi_mt", name="fr", split="dev")
+df_test = load_dataset("stsb_multi_mt", name="fr", split="test")
+
+# Convert the dataset for evaluation
+
+# For Dev set:
+dev_samples = convert_dataset(df_dev)
+val_evaluator = EmbeddingSimilarityEvaluator.from_input_examples(dev_samples, name='sts-dev')
+val_evaluator(model, output_path="./")
+
+# For Test set:
+test_samples = convert_dataset(df_test)
+test_evaluator = EmbeddingSimilarityEvaluator.from_input_examples(test_samples, name='sts-test')
+test_evaluator(model, output_path="./")
+```
+
+**Test Result**: 
+The performance is measured using Pearson and Spearman correlation:
+- On dev
+
+
+| Model  | Pearson correlation | Spearman correlation  | #params  |
+| ------------- | ------------- | ------------- |------------- |
+| [Lajavaness/sentence-camembert-large](https://huggingface.co/dangvantuan/sentence-camembert-large)| **88.63** |**88.46** | 336M| 
+| [dangvantuan/sentence-camembert-large](https://huggingface.co/dangvantuan/sentence-camembert-large)| 88.2 |88.02 | 336M| 
+| [Sahajtomar/french_semanti](https://huggingface.co/Sahajtomar/french_semantic)| 87.44 |87.30 | 336M| 
+| [Lajavaness/sentence-flaubert-base](https://huggingface.co/Lajavaness/sentence-flaubert-base)| 87.14 |87.10 | 137M |
+| [GPT-3 (text-davinci-003)](https://platform.openai.com/docs/models) | 85 | NaN|175B |
+| [GPT-(text-embedding-ada-002)](https://platform.openai.com/docs/models) | 79.75 | 80.44|NaN |
+
+
+- On test, Pearson and Spearman correlation are evaluated on many different benchmark datasets:
+
+
+**Pearson score**
+| Model                                                      | [STS-B](https://huggingface.co/datasets/stsb_multi_mt/viewer/fr/train)   | [STS12-fr ](https://huggingface.co/datasets/Lajavaness/STS12-fr)| [STS13-fr](https://huggingface.co/datasets/Lajavaness/STS13-fr) | [STS14-fr](https://huggingface.co/datasets/Lajavaness/STS14-fr) | [STS15-fr](https://huggingface.co/datasets/Lajavaness/STS15-fr) | [STS16-fr](https://huggingface.co/datasets/Lajavaness/STS16-fr) | [SICK-fr](https://huggingface.co/datasets/Lajavaness/SICK-fr) | params |
+|------------------------------------------|-------|----------|----------|----------|----------|----------|---------|--------|
+| [Lajavaness/sentence-camembert-large](https://huggingface.co/dangvantuan/sentence-camembert-large)      | **86.26** | **87.42**    | **89.34**    | **88.05**    | **88.91**    | 77.15    | 83.13   | 336M   |
+| [dangvantuan/sentence-camembert-large](https://huggingface.co/dangvantuan/sentence-camembert-large)     | 85.88 | 87.28    | 89.25    | 87.91    | 88.54    | 76.90    | 83.26   | 336M   |
+| [Sahajtomar/french_semantic](https://huggingface.co/Sahajtomar/french_semantic)               | 85.80 | 86.05    | 88.50    | 86.57    | 87.49    | 77.85    | 83.27   | 336M    |
+| [Lajavaness/sentence-flaubert-base](https://huggingface.co/Lajavaness/sentence-flaubert-base)        | 85.39 | 86.64    | 87.24    | 85.68    | 87.99    | 75.78    | 82.84   | 137M   |
+| [GPT3 (text-embedding-ada-002)](https://platform.openai.com/docs/models)            | 79.03 | 66.16    | 75.48    | 70.69    | 77.88    | 65.18    | -       | -      |
+
+
+**Spearman score**
+| Model                                                      | [STS-B](https://huggingface.co/datasets/stsb_multi_mt/viewer/fr/train)   | [STS12-fr ](https://huggingface.co/datasets/Lajavaness/STS12-fr)| [STS13-fr](https://huggingface.co/datasets/Lajavaness/STS13-fr) | [STS14-fr](https://huggingface.co/datasets/Lajavaness/STS14-fr) | [STS15-fr](https://huggingface.co/datasets/Lajavaness/STS15-fr) | [STS16-fr](https://huggingface.co/datasets/Lajavaness/STS16-fr) | [SICK-fr](https://huggingface.co/datasets/Lajavaness/SICK-fr) | params |
+|:-------------------------------------|-------:|---------:|---------:|---------:|---------:|---------:|--------:|:-------|
+| [Lajavaness/sentence-camembert-large](https://huggingface.co/dangvantuan/sentence-camembert-large)   | **86.14** | **81.22**    | 88.61    | **86.28**    | **89.01**    | 78.65    | **77.71**   | 336M   |
+| [dangvantuan/sentence-camembert-large](https://huggingface.co/dangvantuan/sentence-camembert-large) | 85.78 | 81.09    | 88.68    | 85.81    | 88.56    | 78.49    | 77.70   | 336M   |
+| [Sahajtomar/french_semantic](https://huggingface.co/Sahajtomar/french_semantic)          | 85.55 | 77.92    | 87.85    | 83.96    | 87.63    | 79.07    | 77.14   | 336M   |
+| [Lajavaness/sentence-flaubert-base](https://huggingface.co/Lajavaness/sentence-flaubert-base)    | 85.67 | 79.97    | 86.91    | 84.57    | 88.10    | 77.84    | 77.55   | 137M    |
+| [GPT3 (text-embedding-ada-002)](https://platform.openai.com/docs/models)         | 77.53 | 64.27    | 76.41    | 69.63    | 78.65    | 75.30    | -       | -      |
+
+
+
+## Citation
+
+
+	@article{reimers2019sentence,
+	   title={Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks},
+	   author={Nils Reimers, Iryna Gurevych},
+	   journal={https://arxiv.org/abs/1908.10084},
+	   year={2019}
+	}
+
+
+	@article{martin2020camembert,
+	   title={CamemBERT: a Tasty French Language Mode},
+	   author={Martin, Louis and Muller, Benjamin and Su{\'a}rez, Pedro Javier Ortiz and Dupont, Yoann and Romary, Laurent and de la Clergerie, {\'E}ric Villemonte and Seddah, Djam{\'e} and Sagot, Beno{\^\i}t},
+	   journal={Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
+	   year={2020}
+	}

added_tokens.json

@@ -0,0 +1,9 @@
+{
+  "</s>": 6,
+  "</s>NOTUSED": 2,
+  "<mask>": 32004,
+  "<pad>": 1,
+  "<s>": 5,
+  "<s>NOTUSED": 0,
+  "<unk>": 3
+}

config.json

@@ -0,0 +1,28 @@
+{
+  "_name_or_path": "Lajavaness/sentence-camembert-large",
+  "architectures": [
+    "CamembertModel"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": 0,
+  "classifier_dropout": null,
+  "eos_token_id": 2,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 1024,
+  "initializer_range": 0.02,
+  "intermediate_size": 4096,
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 514,
+  "model_type": "camembert",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 24,
+  "output_past": true,
+  "pad_token_id": 1,
+  "position_embedding_type": "absolute",
+  "torch_dtype": "float16",
+  "transformers_version": "4.38.2",
+  "type_vocab_size": 1,
+  "use_cache": true,
+  "vocab_size": 32005
+}

config_sentence_transformers.json

@@ -0,0 +1,7 @@
+{
+  "__version__": {
+    "sentence_transformers": "2.2.2",
+    "transformers": "4.34.0",
+    "pytorch": "2.0.1"
+  }
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:402e561f8da2b00dac77e4880062f02ac3057faf41ba51b93a2ac094086e5c23
+size 1346690896

modules.json

@@ -0,0 +1,14 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.models.Transformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_Pooling",
+    "type": "sentence_transformers.models.Pooling"
+  }
+]

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:21e2cf7956d639d06096d34da9c8b521969ba9ab822bc99393f0ced4314aec48
+size 1346777833

sentence_bert_config.json

@@ -0,0 +1,4 @@
+{
+  "max_seq_length": 514,
+  "do_lower_case": false
+}

sentencepiece.bpe.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f98f266fdc548c94216aaadc13ffaaafacf0c8793303e2195322d954549ea261
+size 808767

special_tokens_map.json

@@ -0,0 +1,13 @@
+{
+  "additional_special_tokens": [
+    "<s>NOTUSED",
+    "</s>NOTUSED"
+  ],
+  "bos_token": "<s>",
+  "cls_token": "<s>",
+  "eos_token": "</s>",
+  "mask_token": "<mask>",
+  "pad_token": "<pad>",
+  "sep_token": "</s>",
+  "unk_token": "<unk>"
+}

tokenizer_config.json

@@ -0,0 +1,75 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<s>NOTUSED",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>NOTUSED",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "5": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "6": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32004": {
+      "content": "<mask>",
+      "lstrip": true,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<s>NOTUSED",
+    "</s>NOTUSED"
+  ],
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "<s>",
+  "eos_token": "</s>",
+  "mask_token": "<mask>",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<pad>",
+  "sep_token": "</s>",
+  "sp_model_kwargs": {},
+  "tokenizer_class": "CamembertTokenizer",
+  "unk_token": "<unk>"
+}