philipp-zettl
commited on
Commit
•
b4ba8b3
1
Parent(s):
bbd7653
Add new SentenceTransformer model.
Browse files- README.md +106 -96
- model.safetensors +1 -1
README.md
CHANGED
@@ -6,7 +6,7 @@ tags:
|
|
6 |
- sentence-similarity
|
7 |
- feature-extraction
|
8 |
- generated_from_trainer
|
9 |
-
- dataset_size:
|
10 |
- loss:CoSENTLoss
|
11 |
base_model: sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
|
12 |
datasets: []
|
@@ -22,30 +22,30 @@ metrics:
|
|
22 |
- pearson_max
|
23 |
- spearman_max
|
24 |
widget:
|
25 |
-
- source_sentence:
|
26 |
sentences:
|
27 |
-
- order query
|
28 |
- faq query
|
29 |
-
-
|
30 |
-
-
|
|
|
31 |
sentences:
|
32 |
-
-
|
|
|
33 |
- feedback query
|
34 |
-
|
35 |
-
- source_sentence: 告诉我如何更改我的密码
|
36 |
sentences:
|
37 |
- support query
|
38 |
- product query
|
|
|
|
|
|
|
|
|
|
|
39 |
- faq query
|
40 |
-
- source_sentence:
|
41 |
sentences:
|
42 |
-
-
|
43 |
-
- service request
|
44 |
- account query
|
45 |
-
- source_sentence: Change the currency for my payment
|
46 |
-
sentences:
|
47 |
-
- product query
|
48 |
-
- payment query
|
49 |
- faq query
|
50 |
pipeline_tag: sentence-similarity
|
51 |
model-index:
|
@@ -59,34 +59,34 @@ model-index:
|
|
59 |
type: MiniLM-dev
|
60 |
metrics:
|
61 |
- type: pearson_cosine
|
62 |
-
value: 0.
|
63 |
name: Pearson Cosine
|
64 |
- type: spearman_cosine
|
65 |
-
value: 0.
|
66 |
name: Spearman Cosine
|
67 |
- type: pearson_manhattan
|
68 |
-
value: 0.
|
69 |
name: Pearson Manhattan
|
70 |
- type: spearman_manhattan
|
71 |
-
value: 0.
|
72 |
name: Spearman Manhattan
|
73 |
- type: pearson_euclidean
|
74 |
-
value: 0.
|
75 |
name: Pearson Euclidean
|
76 |
- type: spearman_euclidean
|
77 |
-
value: 0.
|
78 |
name: Spearman Euclidean
|
79 |
- type: pearson_dot
|
80 |
-
value: 0.
|
81 |
name: Pearson Dot
|
82 |
- type: spearman_dot
|
83 |
-
value: 0.
|
84 |
name: Spearman Dot
|
85 |
- type: pearson_max
|
86 |
-
value: 0.
|
87 |
name: Pearson Max
|
88 |
- type: spearman_max
|
89 |
-
value: 0.
|
90 |
name: Spearman Max
|
91 |
- task:
|
92 |
type: semantic-similarity
|
@@ -96,34 +96,34 @@ model-index:
|
|
96 |
type: MiniLM-test
|
97 |
metrics:
|
98 |
- type: pearson_cosine
|
99 |
-
value: 0.
|
100 |
name: Pearson Cosine
|
101 |
- type: spearman_cosine
|
102 |
-
value: 0.
|
103 |
name: Spearman Cosine
|
104 |
- type: pearson_manhattan
|
105 |
-
value: 0.
|
106 |
name: Pearson Manhattan
|
107 |
- type: spearman_manhattan
|
108 |
-
value: 0.
|
109 |
name: Spearman Manhattan
|
110 |
- type: pearson_euclidean
|
111 |
-
value: 0.
|
112 |
name: Pearson Euclidean
|
113 |
- type: spearman_euclidean
|
114 |
-
value: 0.
|
115 |
name: Spearman Euclidean
|
116 |
- type: pearson_dot
|
117 |
-
value: 0.
|
118 |
name: Pearson Dot
|
119 |
- type: spearman_dot
|
120 |
-
value: 0.
|
121 |
name: Spearman Dot
|
122 |
- type: pearson_max
|
123 |
-
value: 0.
|
124 |
name: Pearson Max
|
125 |
- type: spearman_max
|
126 |
-
value: 0.
|
127 |
name: Spearman Max
|
128 |
---
|
129 |
|
@@ -176,9 +176,9 @@ from sentence_transformers import SentenceTransformer
|
|
176 |
model = SentenceTransformer("philipp-zettl/MiniLM-similarity-small")
|
177 |
# Run inference
|
178 |
sentences = [
|
179 |
-
'
|
180 |
-
'payment query',
|
181 |
'faq query',
|
|
|
182 |
]
|
183 |
embeddings = model.encode(sentences)
|
184 |
print(embeddings.shape)
|
@@ -224,16 +224,16 @@ You can finetune this model on your own dataset.
|
|
224 |
|
225 |
| Metric | Value |
|
226 |
|:--------------------|:-----------|
|
227 |
-
| pearson_cosine | 0.
|
228 |
-
| **spearman_cosine** | **0.
|
229 |
-
| pearson_manhattan | 0.
|
230 |
-
| spearman_manhattan | 0.
|
231 |
-
| pearson_euclidean | 0.
|
232 |
-
| spearman_euclidean | 0.
|
233 |
-
| pearson_dot | 0.
|
234 |
-
| spearman_dot | 0.
|
235 |
-
| pearson_max | 0.
|
236 |
-
| spearman_max | 0.
|
237 |
|
238 |
#### Semantic Similarity
|
239 |
* Dataset: `MiniLM-test`
|
@@ -241,16 +241,16 @@ You can finetune this model on your own dataset.
|
|
241 |
|
242 |
| Metric | Value |
|
243 |
|:--------------------|:-----------|
|
244 |
-
| pearson_cosine | 0.
|
245 |
-
| **spearman_cosine** | **0.
|
246 |
-
| pearson_manhattan | 0.
|
247 |
-
| spearman_manhattan | 0.
|
248 |
-
| pearson_euclidean | 0.
|
249 |
-
| spearman_euclidean | 0.
|
250 |
-
| pearson_dot | 0.
|
251 |
-
| spearman_dot | 0.
|
252 |
-
| pearson_max | 0.
|
253 |
-
| spearman_max | 0.
|
254 |
|
255 |
<!--
|
256 |
## Bias, Risks and Limitations
|
@@ -271,19 +271,19 @@ You can finetune this model on your own dataset.
|
|
271 |
#### Unnamed Dataset
|
272 |
|
273 |
|
274 |
-
* Size:
|
275 |
* Columns: <code>sentence1</code>, <code>sentence2</code>, and <code>score</code>
|
276 |
* Approximate statistics based on the first 1000 samples:
|
277 |
-
| | sentence1
|
278 |
-
|
279 |
-
| type | string
|
280 |
-
| details | <ul><li>min: 6 tokens</li><li>mean: 10.
|
281 |
* Samples:
|
282 |
-
| sentence1
|
283 |
-
|
284 |
-
| <code>
|
285 |
-
| <code
|
286 |
-
| <code
|
287 |
* Loss: [<code>CoSENTLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cosentloss) with these parameters:
|
288 |
```json
|
289 |
{
|
@@ -297,19 +297,19 @@ You can finetune this model on your own dataset.
|
|
297 |
#### Unnamed Dataset
|
298 |
|
299 |
|
300 |
-
* Size:
|
301 |
* Columns: <code>sentence1</code>, <code>sentence2</code>, and <code>score</code>
|
302 |
* Approximate statistics based on the first 1000 samples:
|
303 |
| | sentence1 | sentence2 | score |
|
304 |
|:--------|:----------------------------------------------------------------------------------|:--------------------------------------------------------------------------------|:---------------------------------------------------------------|
|
305 |
| type | string | string | float |
|
306 |
-
| details | <ul><li>min: 6 tokens</li><li>mean: 10.
|
307 |
* Samples:
|
308 |
-
| sentence1
|
309 |
-
|
310 |
-
| <code
|
311 |
-
| <code>
|
312 |
-
| <code>
|
313 |
* Loss: [<code>CoSENTLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cosentloss) with these parameters:
|
314 |
```json
|
315 |
{
|
@@ -445,28 +445,38 @@ You can finetune this model on your own dataset.
|
|
445 |
### Training Logs
|
446 |
| Epoch | Step | Training Loss | loss | MiniLM-dev_spearman_cosine | MiniLM-test_spearman_cosine |
|
447 |
|:------:|:----:|:-------------:|:------:|:--------------------------:|:---------------------------:|
|
448 |
-
| 0.
|
449 |
-
| 0.
|
450 |
-
| 0.
|
451 |
-
| 0.
|
452 |
-
| 0.
|
453 |
-
| 0.
|
454 |
-
| 0.
|
455 |
-
| 0.
|
456 |
-
| 0.
|
457 |
-
| 0.
|
458 |
-
|
|
459 |
-
|
|
460 |
-
|
|
461 |
-
|
|
462 |
-
|
|
463 |
-
| 1.
|
464 |
-
| 1.
|
465 |
-
| 1.
|
466 |
-
| 1.
|
467 |
-
| 1.
|
468 |
-
| 1.
|
469 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
470 |
|
471 |
|
472 |
### Framework Versions
|
|
|
6 |
- sentence-similarity
|
7 |
- feature-extraction
|
8 |
- generated_from_trainer
|
9 |
+
- dataset_size:1267
|
10 |
- loss:CoSENTLoss
|
11 |
base_model: sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
|
12 |
datasets: []
|
|
|
22 |
- pearson_max
|
23 |
- spearman_max
|
24 |
widget:
|
25 |
+
- source_sentence: Give me suggestions for a high-quality DSLR camera
|
26 |
sentences:
|
|
|
27 |
- faq query
|
28 |
+
- subscription query
|
29 |
+
- faq query
|
30 |
+
- source_sentence: Aidez-moi à configurer une nouvelle adresse e-mail
|
31 |
sentences:
|
32 |
+
- order query
|
33 |
+
- faq query
|
34 |
- feedback query
|
35 |
+
- source_sentence: Как я могу изменить адрес доставки?
|
|
|
36 |
sentences:
|
37 |
- support query
|
38 |
- product query
|
39 |
+
- product query
|
40 |
+
- source_sentence: ساعدني في حذف الملفات الغير مرغوب فيها من هاتفي
|
41 |
+
sentences:
|
42 |
+
- technical support query
|
43 |
+
- product recommendation
|
44 |
- faq query
|
45 |
+
- source_sentence: Envoyez-moi la politique de garantie de ce produit
|
46 |
sentences:
|
47 |
+
- faq query
|
|
|
48 |
- account query
|
|
|
|
|
|
|
|
|
49 |
- faq query
|
50 |
pipeline_tag: sentence-similarity
|
51 |
model-index:
|
|
|
59 |
type: MiniLM-dev
|
60 |
metrics:
|
61 |
- type: pearson_cosine
|
62 |
+
value: 0.6538226572138826
|
63 |
name: Pearson Cosine
|
64 |
- type: spearman_cosine
|
65 |
+
value: 0.6336766646599241
|
66 |
name: Spearman Cosine
|
67 |
- type: pearson_manhattan
|
68 |
+
value: 0.5799895241429639
|
69 |
name: Pearson Manhattan
|
70 |
- type: spearman_manhattan
|
71 |
+
value: 0.5525776786782183
|
72 |
name: Spearman Manhattan
|
73 |
- type: pearson_euclidean
|
74 |
+
value: 0.5732001104236694
|
75 |
name: Pearson Euclidean
|
76 |
- type: spearman_euclidean
|
77 |
+
value: 0.5394971970682657
|
78 |
name: Spearman Euclidean
|
79 |
- type: pearson_dot
|
80 |
+
value: 0.6359725423136287
|
81 |
name: Pearson Dot
|
82 |
- type: spearman_dot
|
83 |
+
value: 0.6237936341101822
|
84 |
name: Spearman Dot
|
85 |
- type: pearson_max
|
86 |
+
value: 0.6538226572138826
|
87 |
name: Pearson Max
|
88 |
- type: spearman_max
|
89 |
+
value: 0.6336766646599241
|
90 |
name: Spearman Max
|
91 |
- task:
|
92 |
type: semantic-similarity
|
|
|
96 |
type: MiniLM-test
|
97 |
metrics:
|
98 |
- type: pearson_cosine
|
99 |
+
value: 0.6682368113711722
|
100 |
name: Pearson Cosine
|
101 |
- type: spearman_cosine
|
102 |
+
value: 0.6222011918428743
|
103 |
name: Spearman Cosine
|
104 |
- type: pearson_manhattan
|
105 |
+
value: 0.5714617063306076
|
106 |
name: Pearson Manhattan
|
107 |
- type: spearman_manhattan
|
108 |
+
value: 0.5481366191719228
|
109 |
name: Spearman Manhattan
|
110 |
- type: pearson_euclidean
|
111 |
+
value: 0.5726946277850402
|
112 |
name: Pearson Euclidean
|
113 |
- type: spearman_euclidean
|
114 |
+
value: 0.549312247309557
|
115 |
name: Spearman Euclidean
|
116 |
- type: pearson_dot
|
117 |
+
value: 0.6396412507506479
|
118 |
name: Pearson Dot
|
119 |
- type: spearman_dot
|
120 |
+
value: 0.6107388175009413
|
121 |
name: Spearman Dot
|
122 |
- type: pearson_max
|
123 |
+
value: 0.6682368113711722
|
124 |
name: Pearson Max
|
125 |
- type: spearman_max
|
126 |
+
value: 0.6222011918428743
|
127 |
name: Spearman Max
|
128 |
---
|
129 |
|
|
|
176 |
model = SentenceTransformer("philipp-zettl/MiniLM-similarity-small")
|
177 |
# Run inference
|
178 |
sentences = [
|
179 |
+
'Envoyez-moi la politique de garantie de ce produit',
|
|
|
180 |
'faq query',
|
181 |
+
'account query',
|
182 |
]
|
183 |
embeddings = model.encode(sentences)
|
184 |
print(embeddings.shape)
|
|
|
224 |
|
225 |
| Metric | Value |
|
226 |
|:--------------------|:-----------|
|
227 |
+
| pearson_cosine | 0.6538 |
|
228 |
+
| **spearman_cosine** | **0.6337** |
|
229 |
+
| pearson_manhattan | 0.58 |
|
230 |
+
| spearman_manhattan | 0.5526 |
|
231 |
+
| pearson_euclidean | 0.5732 |
|
232 |
+
| spearman_euclidean | 0.5395 |
|
233 |
+
| pearson_dot | 0.636 |
|
234 |
+
| spearman_dot | 0.6238 |
|
235 |
+
| pearson_max | 0.6538 |
|
236 |
+
| spearman_max | 0.6337 |
|
237 |
|
238 |
#### Semantic Similarity
|
239 |
* Dataset: `MiniLM-test`
|
|
|
241 |
|
242 |
| Metric | Value |
|
243 |
|:--------------------|:-----------|
|
244 |
+
| pearson_cosine | 0.6682 |
|
245 |
+
| **spearman_cosine** | **0.6222** |
|
246 |
+
| pearson_manhattan | 0.5715 |
|
247 |
+
| spearman_manhattan | 0.5481 |
|
248 |
+
| pearson_euclidean | 0.5727 |
|
249 |
+
| spearman_euclidean | 0.5493 |
|
250 |
+
| pearson_dot | 0.6396 |
|
251 |
+
| spearman_dot | 0.6107 |
|
252 |
+
| pearson_max | 0.6682 |
|
253 |
+
| spearman_max | 0.6222 |
|
254 |
|
255 |
<!--
|
256 |
## Bias, Risks and Limitations
|
|
|
271 |
#### Unnamed Dataset
|
272 |
|
273 |
|
274 |
+
* Size: 1,267 training samples
|
275 |
* Columns: <code>sentence1</code>, <code>sentence2</code>, and <code>score</code>
|
276 |
* Approximate statistics based on the first 1000 samples:
|
277 |
+
| | sentence1 | sentence2 | score |
|
278 |
+
|:--------|:----------------------------------------------------------------------------------|:--------------------------------------------------------------------------------|:---------------------------------------------------------------|
|
279 |
+
| type | string | string | float |
|
280 |
+
| details | <ul><li>min: 6 tokens</li><li>mean: 10.77 tokens</li><li>max: 18 tokens</li></ul> | <ul><li>min: 4 tokens</li><li>mean: 5.31 tokens</li><li>max: 6 tokens</li></ul> | <ul><li>min: 0.0</li><li>mean: 0.67</li><li>max: 1.0</li></ul> |
|
281 |
* Samples:
|
282 |
+
| sentence1 | sentence2 | score |
|
283 |
+
|:--------------------------------------------------------------|:---------------------------|:-----------------|
|
284 |
+
| <code>Get information on the next art exhibition</code> | <code>product query</code> | <code>0.0</code> |
|
285 |
+
| <code>Show me how to update my profile</code> | <code>product query</code> | <code>0.0</code> |
|
286 |
+
| <code>Покажите мне доступные варианты полетов в Турцию</code> | <code>faq query</code> | <code>0.0</code> |
|
287 |
* Loss: [<code>CoSENTLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cosentloss) with these parameters:
|
288 |
```json
|
289 |
{
|
|
|
297 |
#### Unnamed Dataset
|
298 |
|
299 |
|
300 |
+
* Size: 159 evaluation samples
|
301 |
* Columns: <code>sentence1</code>, <code>sentence2</code>, and <code>score</code>
|
302 |
* Approximate statistics based on the first 1000 samples:
|
303 |
| | sentence1 | sentence2 | score |
|
304 |
|:--------|:----------------------------------------------------------------------------------|:--------------------------------------------------------------------------------|:---------------------------------------------------------------|
|
305 |
| type | string | string | float |
|
306 |
+
| details | <ul><li>min: 6 tokens</li><li>mean: 10.65 tokens</li><li>max: 17 tokens</li></ul> | <ul><li>min: 4 tokens</li><li>mean: 5.35 tokens</li><li>max: 6 tokens</li></ul> | <ul><li>min: 0.0</li><li>mean: 0.67</li><li>max: 1.0</li></ul> |
|
307 |
* Samples:
|
308 |
+
| sentence1 | sentence2 | score |
|
309 |
+
|:---------------------------------------------------------------|:---------------------------|:-----------------|
|
310 |
+
| <code>Sende mir die Bestellbestätigung per E-Mail</code> | <code>order query</code> | <code>0.0</code> |
|
311 |
+
| <code>How do I add a new payment method?</code> | <code>faq query</code> | <code>1.0</code> |
|
312 |
+
| <code>No puedo conectar mi impresora, ¿puedes ayudarme?</code> | <code>support query</code> | <code>1.0</code> |
|
313 |
* Loss: [<code>CoSENTLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cosentloss) with these parameters:
|
314 |
```json
|
315 |
{
|
|
|
445 |
### Training Logs
|
446 |
| Epoch | Step | Training Loss | loss | MiniLM-dev_spearman_cosine | MiniLM-test_spearman_cosine |
|
447 |
|:------:|:----:|:-------------:|:------:|:--------------------------:|:---------------------------:|
|
448 |
+
| 0.0629 | 10 | 6.2479 | 2.5890 | 0.1448 | - |
|
449 |
+
| 0.1258 | 20 | 4.3549 | 2.2787 | 0.1965 | - |
|
450 |
+
| 0.1887 | 30 | 3.5969 | 2.0104 | 0.2599 | - |
|
451 |
+
| 0.2516 | 40 | 2.4979 | 1.7269 | 0.3357 | - |
|
452 |
+
| 0.3145 | 50 | 2.5551 | 1.5747 | 0.4439 | - |
|
453 |
+
| 0.3774 | 60 | 3.1446 | 1.4892 | 0.4750 | - |
|
454 |
+
| 0.4403 | 70 | 2.1353 | 1.5305 | 0.4662 | - |
|
455 |
+
| 0.5031 | 80 | 2.9341 | 1.3718 | 0.4848 | - |
|
456 |
+
| 0.5660 | 90 | 2.8709 | 1.2469 | 0.5316 | - |
|
457 |
+
| 0.6289 | 100 | 2.1367 | 1.2558 | 0.5436 | - |
|
458 |
+
| 0.6918 | 110 | 2.2735 | 1.2939 | 0.5392 | - |
|
459 |
+
| 0.7547 | 120 | 2.8646 | 1.1206 | 0.5616 | - |
|
460 |
+
| 0.8176 | 130 | 3.3204 | 1.0213 | 0.5662 | - |
|
461 |
+
| 0.8805 | 140 | 0.8989 | 0.9866 | 0.5738 | - |
|
462 |
+
| 0.9434 | 150 | 0.0057 | 0.9961 | 0.5674 | - |
|
463 |
+
| 1.0063 | 160 | 0.0019 | 1.0111 | 0.5674 | - |
|
464 |
+
| 1.0692 | 170 | 0.4617 | 1.0275 | 0.5747 | - |
|
465 |
+
| 1.1321 | 180 | 0.0083 | 1.0746 | 0.5732 | - |
|
466 |
+
| 1.1950 | 190 | 0.5048 | 1.0968 | 0.5753 | - |
|
467 |
+
| 1.2579 | 200 | 0.0002 | 1.0840 | 0.5738 | - |
|
468 |
+
| 1.3208 | 210 | 0.07 | 1.0364 | 0.5753 | - |
|
469 |
+
| 1.3836 | 220 | 0.0 | 0.9952 | 0.5750 | - |
|
470 |
+
| 1.4465 | 230 | 0.0 | 0.9922 | 0.5744 | - |
|
471 |
+
| 1.5094 | 240 | 0.0 | 0.9923 | 0.5726 | - |
|
472 |
+
| 1.0126 | 250 | 0.229 | 0.9930 | 0.5729 | - |
|
473 |
+
| 1.0755 | 260 | 2.2061 | 0.9435 | 0.5880 | - |
|
474 |
+
| 1.1384 | 270 | 2.7711 | 0.8892 | 0.6078 | - |
|
475 |
+
| 1.2013 | 280 | 0.7528 | 0.8886 | 0.6148 | - |
|
476 |
+
| 1.2642 | 290 | 0.386 | 0.8927 | 0.6162 | - |
|
477 |
+
| 1.3270 | 300 | 0.8902 | 0.8710 | 0.6267 | - |
|
478 |
+
| 1.3899 | 310 | 0.9534 | 0.8429 | 0.6337 | - |
|
479 |
+
| 1.4403 | 318 | - | - | - | 0.6222 |
|
480 |
|
481 |
|
482 |
### Framework Versions
|
model.safetensors
CHANGED
@@ -1,3 +1,3 @@
|
|
1 |
version https://git-lfs.github.com/spec/v1
|
2 |
-
oid sha256:
|
3 |
size 470637416
|
|
|
1 |
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:a480f8a3b0abde34feef318b982835792b5781f388c0cbeb144e8d54ef77f2a3
|
3 |
size 470637416
|