philipp-zettl commited on
Commit
b4ba8b3
1 Parent(s): bbd7653

Add new SentenceTransformer model.

Browse files
Files changed (2) hide show
  1. README.md +106 -96
  2. model.safetensors +1 -1
README.md CHANGED
@@ -6,7 +6,7 @@ tags:
6
  - sentence-similarity
7
  - feature-extraction
8
  - generated_from_trainer
9
- - dataset_size:844
10
  - loss:CoSENTLoss
11
  base_model: sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
12
  datasets: []
@@ -22,30 +22,30 @@ metrics:
22
  - pearson_max
23
  - spearman_max
24
  widget:
25
- - source_sentence: Help fix a problem with my device’s battery life
26
  sentences:
27
- - order query
28
  - faq query
29
- - technical support query
30
- - source_sentence: 订购一双运动鞋
 
31
  sentences:
32
- - service request
 
33
  - feedback query
34
- - product query
35
- - source_sentence: 告诉我如何更改我的密码
36
  sentences:
37
  - support query
38
  - product query
 
 
 
 
 
39
  - faq query
40
- - source_sentence: Get information on the next local festival
41
  sentences:
42
- - event inquiry
43
- - service request
44
  - account query
45
- - source_sentence: Change the currency for my payment
46
- sentences:
47
- - product query
48
- - payment query
49
  - faq query
50
  pipeline_tag: sentence-similarity
51
  model-index:
@@ -59,34 +59,34 @@ model-index:
59
  type: MiniLM-dev
60
  metrics:
61
  - type: pearson_cosine
62
- value: 0.7356955662825808
63
  name: Pearson Cosine
64
  - type: spearman_cosine
65
- value: 0.7320761390174187
66
  name: Spearman Cosine
67
  - type: pearson_manhattan
68
- value: 0.6240041985776243
69
  name: Pearson Manhattan
70
  - type: spearman_manhattan
71
- value: 0.6179783414452009
72
  name: Spearman Manhattan
73
  - type: pearson_euclidean
74
- value: 0.6321466982201008
75
  name: Pearson Euclidean
76
  - type: spearman_euclidean
77
- value: 0.6296964936282937
78
  name: Spearman Euclidean
79
  - type: pearson_dot
80
- value: 0.7491168439451736
81
  name: Pearson Dot
82
  - type: spearman_dot
83
- value: 0.7592129124940543
84
  name: Spearman Dot
85
  - type: pearson_max
86
- value: 0.7491168439451736
87
  name: Pearson Max
88
  - type: spearman_max
89
- value: 0.7592129124940543
90
  name: Spearman Max
91
  - task:
92
  type: semantic-similarity
@@ -96,34 +96,34 @@ model-index:
96
  type: MiniLM-test
97
  metrics:
98
  - type: pearson_cosine
99
- value: 0.7687106130417081
100
  name: Pearson Cosine
101
  - type: spearman_cosine
102
- value: 0.7552108666502075
103
  name: Spearman Cosine
104
  - type: pearson_manhattan
105
- value: 0.7462708006775693
106
  name: Pearson Manhattan
107
  - type: spearman_manhattan
108
- value: 0.7365483246407295
109
  name: Spearman Manhattan
110
  - type: pearson_euclidean
111
- value: 0.7545194410402545
112
  name: Pearson Euclidean
113
  - type: spearman_euclidean
114
- value: 0.7465016803791179
115
  name: Spearman Euclidean
116
  - type: pearson_dot
117
- value: 0.7251488155932073
118
  name: Pearson Dot
119
  - type: spearman_dot
120
- value: 0.7390366635753267
121
  name: Spearman Dot
122
  - type: pearson_max
123
- value: 0.7687106130417081
124
  name: Pearson Max
125
  - type: spearman_max
126
- value: 0.7552108666502075
127
  name: Spearman Max
128
  ---
129
 
@@ -176,9 +176,9 @@ from sentence_transformers import SentenceTransformer
176
  model = SentenceTransformer("philipp-zettl/MiniLM-similarity-small")
177
  # Run inference
178
  sentences = [
179
- 'Change the currency for my payment',
180
- 'payment query',
181
  'faq query',
 
182
  ]
183
  embeddings = model.encode(sentences)
184
  print(embeddings.shape)
@@ -224,16 +224,16 @@ You can finetune this model on your own dataset.
224
 
225
  | Metric | Value |
226
  |:--------------------|:-----------|
227
- | pearson_cosine | 0.7357 |
228
- | **spearman_cosine** | **0.7321** |
229
- | pearson_manhattan | 0.624 |
230
- | spearman_manhattan | 0.618 |
231
- | pearson_euclidean | 0.6321 |
232
- | spearman_euclidean | 0.6297 |
233
- | pearson_dot | 0.7491 |
234
- | spearman_dot | 0.7592 |
235
- | pearson_max | 0.7491 |
236
- | spearman_max | 0.7592 |
237
 
238
  #### Semantic Similarity
239
  * Dataset: `MiniLM-test`
@@ -241,16 +241,16 @@ You can finetune this model on your own dataset.
241
 
242
  | Metric | Value |
243
  |:--------------------|:-----------|
244
- | pearson_cosine | 0.7687 |
245
- | **spearman_cosine** | **0.7552** |
246
- | pearson_manhattan | 0.7463 |
247
- | spearman_manhattan | 0.7365 |
248
- | pearson_euclidean | 0.7545 |
249
- | spearman_euclidean | 0.7465 |
250
- | pearson_dot | 0.7251 |
251
- | spearman_dot | 0.739 |
252
- | pearson_max | 0.7687 |
253
- | spearman_max | 0.7552 |
254
 
255
  <!--
256
  ## Bias, Risks and Limitations
@@ -271,19 +271,19 @@ You can finetune this model on your own dataset.
271
  #### Unnamed Dataset
272
 
273
 
274
- * Size: 844 training samples
275
  * Columns: <code>sentence1</code>, <code>sentence2</code>, and <code>score</code>
276
  * Approximate statistics based on the first 1000 samples:
277
- | | sentence1 | sentence2 | score |
278
- |:--------|:---------------------------------------------------------------------------------|:--------------------------------------------------------------------------------|:---------------------------------------------------------------|
279
- | type | string | string | float |
280
- | details | <ul><li>min: 6 tokens</li><li>mean: 10.8 tokens</li><li>max: 19 tokens</li></ul> | <ul><li>min: 4 tokens</li><li>mean: 5.33 tokens</li><li>max: 6 tokens</li></ul> | <ul><li>min: 0.0</li><li>mean: 0.49</li><li>max: 1.0</li></ul> |
281
  * Samples:
282
- | sentence1 | sentence2 | score |
283
- |:----------------------------------------------------------------|:---------------------------|:-----------------|
284
- | <code>Update the payment method for my order</code> | <code>order query</code> | <code>1.0</code> |
285
- | <code>Не могу установить новое обновление, помогите!</code> | <code>support query</code> | <code>1.0</code> |
286
- | <code>Помогите мне изменить настройки конфиденциальности</code> | <code>support query</code> | <code>1.0</code> |
287
  * Loss: [<code>CoSENTLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cosentloss) with these parameters:
288
  ```json
289
  {
@@ -297,19 +297,19 @@ You can finetune this model on your own dataset.
297
  #### Unnamed Dataset
298
 
299
 
300
- * Size: 106 evaluation samples
301
  * Columns: <code>sentence1</code>, <code>sentence2</code>, and <code>score</code>
302
  * Approximate statistics based on the first 1000 samples:
303
  | | sentence1 | sentence2 | score |
304
  |:--------|:----------------------------------------------------------------------------------|:--------------------------------------------------------------------------------|:---------------------------------------------------------------|
305
  | type | string | string | float |
306
- | details | <ul><li>min: 6 tokens</li><li>mean: 10.79 tokens</li><li>max: 15 tokens</li></ul> | <ul><li>min: 4 tokens</li><li>mean: 5.27 tokens</li><li>max: 6 tokens</li></ul> | <ul><li>min: 0.0</li><li>mean: 0.51</li><li>max: 1.0</li></ul> |
307
  * Samples:
308
- | sentence1 | sentence2 | score |
309
- |:----------------------------------------------------------------|:-------------------------------------|:-----------------|
310
- | <code>帮我修复系统错误</code> | <code>support query</code> | <code>1.0</code> |
311
- | <code>Je veux commander une pizza</code> | <code>product query</code> | <code>1.0</code> |
312
- | <code>Fix problems with my device’s Bluetooth connection</code> | <code>technical support query</code> | <code>1.0</code> |
313
  * Loss: [<code>CoSENTLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cosentloss) with these parameters:
314
  ```json
315
  {
@@ -445,28 +445,38 @@ You can finetune this model on your own dataset.
445
  ### Training Logs
446
  | Epoch | Step | Training Loss | loss | MiniLM-dev_spearman_cosine | MiniLM-test_spearman_cosine |
447
  |:------:|:----:|:-------------:|:------:|:--------------------------:|:---------------------------:|
448
- | 0.0943 | 10 | 4.0771 | 2.2054 | 0.2529 | - |
449
- | 0.1887 | 20 | 4.4668 | 1.8221 | 0.3528 | - |
450
- | 0.2830 | 30 | 2.5459 | 1.5545 | 0.4638 | - |
451
- | 0.3774 | 40 | 2.1926 | 1.3145 | 0.5569 | - |
452
- | 0.4717 | 50 | 0.9001 | 1.1653 | 0.6285 | - |
453
- | 0.5660 | 60 | 1.4049 | 1.0734 | 0.6834 | - |
454
- | 0.6604 | 70 | 0.7204 | 0.9951 | 0.6988 | - |
455
- | 0.7547 | 80 | 1.4023 | 1.1213 | 0.6945 | - |
456
- | 0.8491 | 90 | 0.2315 | 1.2931 | 0.6414 | - |
457
- | 0.9434 | 100 | 0.0018 | 1.3904 | 0.6180 | - |
458
- | 1.0377 | 110 | 0.0494 | 1.2889 | 0.6322 | - |
459
- | 1.1321 | 120 | 0.3156 | 1.2461 | 0.6402 | - |
460
- | 1.2264 | 130 | 1.8153 | 1.0844 | 0.6716 | - |
461
- | 1.3208 | 140 | 0.2638 | 0.9939 | 0.6957 | - |
462
- | 1.4151 | 150 | 0.5454 | 0.9545 | 0.7056 | - |
463
- | 1.5094 | 160 | 0.3421 | 0.9699 | 0.7062 | - |
464
- | 1.6038 | 170 | 0.0035 | 0.9521 | 0.7093 | - |
465
- | 1.6981 | 180 | 0.0401 | 0.8988 | 0.7160 | - |
466
- | 1.7925 | 190 | 0.8138 | 0.8619 | 0.7271 | - |
467
- | 1.8868 | 200 | 0.0236 | 0.8449 | 0.7315 | - |
468
- | 1.9811 | 210 | 0.0012 | 0.8438 | 0.7321 | - |
469
- | 2.0 | 212 | - | - | - | 0.7552 |
 
 
 
 
 
 
 
 
 
 
470
 
471
 
472
  ### Framework Versions
 
6
  - sentence-similarity
7
  - feature-extraction
8
  - generated_from_trainer
9
+ - dataset_size:1267
10
  - loss:CoSENTLoss
11
  base_model: sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
12
  datasets: []
 
22
  - pearson_max
23
  - spearman_max
24
  widget:
25
+ - source_sentence: Give me suggestions for a high-quality DSLR camera
26
  sentences:
 
27
  - faq query
28
+ - subscription query
29
+ - faq query
30
+ - source_sentence: Aidez-moi à configurer une nouvelle adresse e-mail
31
  sentences:
32
+ - order query
33
+ - faq query
34
  - feedback query
35
+ - source_sentence: Как я могу изменить адрес доставки?
 
36
  sentences:
37
  - support query
38
  - product query
39
+ - product query
40
+ - source_sentence: ساعدني في حذف الملفات الغير مرغوب فيها من هاتفي
41
+ sentences:
42
+ - technical support query
43
+ - product recommendation
44
  - faq query
45
+ - source_sentence: Envoyez-moi la politique de garantie de ce produit
46
  sentences:
47
+ - faq query
 
48
  - account query
 
 
 
 
49
  - faq query
50
  pipeline_tag: sentence-similarity
51
  model-index:
 
59
  type: MiniLM-dev
60
  metrics:
61
  - type: pearson_cosine
62
+ value: 0.6538226572138826
63
  name: Pearson Cosine
64
  - type: spearman_cosine
65
+ value: 0.6336766646599241
66
  name: Spearman Cosine
67
  - type: pearson_manhattan
68
+ value: 0.5799895241429639
69
  name: Pearson Manhattan
70
  - type: spearman_manhattan
71
+ value: 0.5525776786782183
72
  name: Spearman Manhattan
73
  - type: pearson_euclidean
74
+ value: 0.5732001104236694
75
  name: Pearson Euclidean
76
  - type: spearman_euclidean
77
+ value: 0.5394971970682657
78
  name: Spearman Euclidean
79
  - type: pearson_dot
80
+ value: 0.6359725423136287
81
  name: Pearson Dot
82
  - type: spearman_dot
83
+ value: 0.6237936341101822
84
  name: Spearman Dot
85
  - type: pearson_max
86
+ value: 0.6538226572138826
87
  name: Pearson Max
88
  - type: spearman_max
89
+ value: 0.6336766646599241
90
  name: Spearman Max
91
  - task:
92
  type: semantic-similarity
 
96
  type: MiniLM-test
97
  metrics:
98
  - type: pearson_cosine
99
+ value: 0.6682368113711722
100
  name: Pearson Cosine
101
  - type: spearman_cosine
102
+ value: 0.6222011918428743
103
  name: Spearman Cosine
104
  - type: pearson_manhattan
105
+ value: 0.5714617063306076
106
  name: Pearson Manhattan
107
  - type: spearman_manhattan
108
+ value: 0.5481366191719228
109
  name: Spearman Manhattan
110
  - type: pearson_euclidean
111
+ value: 0.5726946277850402
112
  name: Pearson Euclidean
113
  - type: spearman_euclidean
114
+ value: 0.549312247309557
115
  name: Spearman Euclidean
116
  - type: pearson_dot
117
+ value: 0.6396412507506479
118
  name: Pearson Dot
119
  - type: spearman_dot
120
+ value: 0.6107388175009413
121
  name: Spearman Dot
122
  - type: pearson_max
123
+ value: 0.6682368113711722
124
  name: Pearson Max
125
  - type: spearman_max
126
+ value: 0.6222011918428743
127
  name: Spearman Max
128
  ---
129
 
 
176
  model = SentenceTransformer("philipp-zettl/MiniLM-similarity-small")
177
  # Run inference
178
  sentences = [
179
+ 'Envoyez-moi la politique de garantie de ce produit',
 
180
  'faq query',
181
+ 'account query',
182
  ]
183
  embeddings = model.encode(sentences)
184
  print(embeddings.shape)
 
224
 
225
  | Metric | Value |
226
  |:--------------------|:-----------|
227
+ | pearson_cosine | 0.6538 |
228
+ | **spearman_cosine** | **0.6337** |
229
+ | pearson_manhattan | 0.58 |
230
+ | spearman_manhattan | 0.5526 |
231
+ | pearson_euclidean | 0.5732 |
232
+ | spearman_euclidean | 0.5395 |
233
+ | pearson_dot | 0.636 |
234
+ | spearman_dot | 0.6238 |
235
+ | pearson_max | 0.6538 |
236
+ | spearman_max | 0.6337 |
237
 
238
  #### Semantic Similarity
239
  * Dataset: `MiniLM-test`
 
241
 
242
  | Metric | Value |
243
  |:--------------------|:-----------|
244
+ | pearson_cosine | 0.6682 |
245
+ | **spearman_cosine** | **0.6222** |
246
+ | pearson_manhattan | 0.5715 |
247
+ | spearman_manhattan | 0.5481 |
248
+ | pearson_euclidean | 0.5727 |
249
+ | spearman_euclidean | 0.5493 |
250
+ | pearson_dot | 0.6396 |
251
+ | spearman_dot | 0.6107 |
252
+ | pearson_max | 0.6682 |
253
+ | spearman_max | 0.6222 |
254
 
255
  <!--
256
  ## Bias, Risks and Limitations
 
271
  #### Unnamed Dataset
272
 
273
 
274
+ * Size: 1,267 training samples
275
  * Columns: <code>sentence1</code>, <code>sentence2</code>, and <code>score</code>
276
  * Approximate statistics based on the first 1000 samples:
277
+ | | sentence1 | sentence2 | score |
278
+ |:--------|:----------------------------------------------------------------------------------|:--------------------------------------------------------------------------------|:---------------------------------------------------------------|
279
+ | type | string | string | float |
280
+ | details | <ul><li>min: 6 tokens</li><li>mean: 10.77 tokens</li><li>max: 18 tokens</li></ul> | <ul><li>min: 4 tokens</li><li>mean: 5.31 tokens</li><li>max: 6 tokens</li></ul> | <ul><li>min: 0.0</li><li>mean: 0.67</li><li>max: 1.0</li></ul> |
281
  * Samples:
282
+ | sentence1 | sentence2 | score |
283
+ |:--------------------------------------------------------------|:---------------------------|:-----------------|
284
+ | <code>Get information on the next art exhibition</code> | <code>product query</code> | <code>0.0</code> |
285
+ | <code>Show me how to update my profile</code> | <code>product query</code> | <code>0.0</code> |
286
+ | <code>Покажите мне доступные варианты полетов в Турцию</code> | <code>faq query</code> | <code>0.0</code> |
287
  * Loss: [<code>CoSENTLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cosentloss) with these parameters:
288
  ```json
289
  {
 
297
  #### Unnamed Dataset
298
 
299
 
300
+ * Size: 159 evaluation samples
301
  * Columns: <code>sentence1</code>, <code>sentence2</code>, and <code>score</code>
302
  * Approximate statistics based on the first 1000 samples:
303
  | | sentence1 | sentence2 | score |
304
  |:--------|:----------------------------------------------------------------------------------|:--------------------------------------------------------------------------------|:---------------------------------------------------------------|
305
  | type | string | string | float |
306
+ | details | <ul><li>min: 6 tokens</li><li>mean: 10.65 tokens</li><li>max: 17 tokens</li></ul> | <ul><li>min: 4 tokens</li><li>mean: 5.35 tokens</li><li>max: 6 tokens</li></ul> | <ul><li>min: 0.0</li><li>mean: 0.67</li><li>max: 1.0</li></ul> |
307
  * Samples:
308
+ | sentence1 | sentence2 | score |
309
+ |:---------------------------------------------------------------|:---------------------------|:-----------------|
310
+ | <code>Sende mir die Bestellbestätigung per E-Mail</code> | <code>order query</code> | <code>0.0</code> |
311
+ | <code>How do I add a new payment method?</code> | <code>faq query</code> | <code>1.0</code> |
312
+ | <code>No puedo conectar mi impresora, ¿puedes ayudarme?</code> | <code>support query</code> | <code>1.0</code> |
313
  * Loss: [<code>CoSENTLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cosentloss) with these parameters:
314
  ```json
315
  {
 
445
  ### Training Logs
446
  | Epoch | Step | Training Loss | loss | MiniLM-dev_spearman_cosine | MiniLM-test_spearman_cosine |
447
  |:------:|:----:|:-------------:|:------:|:--------------------------:|:---------------------------:|
448
+ | 0.0629 | 10 | 6.2479 | 2.5890 | 0.1448 | - |
449
+ | 0.1258 | 20 | 4.3549 | 2.2787 | 0.1965 | - |
450
+ | 0.1887 | 30 | 3.5969 | 2.0104 | 0.2599 | - |
451
+ | 0.2516 | 40 | 2.4979 | 1.7269 | 0.3357 | - |
452
+ | 0.3145 | 50 | 2.5551 | 1.5747 | 0.4439 | - |
453
+ | 0.3774 | 60 | 3.1446 | 1.4892 | 0.4750 | - |
454
+ | 0.4403 | 70 | 2.1353 | 1.5305 | 0.4662 | - |
455
+ | 0.5031 | 80 | 2.9341 | 1.3718 | 0.4848 | - |
456
+ | 0.5660 | 90 | 2.8709 | 1.2469 | 0.5316 | - |
457
+ | 0.6289 | 100 | 2.1367 | 1.2558 | 0.5436 | - |
458
+ | 0.6918 | 110 | 2.2735 | 1.2939 | 0.5392 | - |
459
+ | 0.7547 | 120 | 2.8646 | 1.1206 | 0.5616 | - |
460
+ | 0.8176 | 130 | 3.3204 | 1.0213 | 0.5662 | - |
461
+ | 0.8805 | 140 | 0.8989 | 0.9866 | 0.5738 | - |
462
+ | 0.9434 | 150 | 0.0057 | 0.9961 | 0.5674 | - |
463
+ | 1.0063 | 160 | 0.0019 | 1.0111 | 0.5674 | - |
464
+ | 1.0692 | 170 | 0.4617 | 1.0275 | 0.5747 | - |
465
+ | 1.1321 | 180 | 0.0083 | 1.0746 | 0.5732 | - |
466
+ | 1.1950 | 190 | 0.5048 | 1.0968 | 0.5753 | - |
467
+ | 1.2579 | 200 | 0.0002 | 1.0840 | 0.5738 | - |
468
+ | 1.3208 | 210 | 0.07 | 1.0364 | 0.5753 | - |
469
+ | 1.3836 | 220 | 0.0 | 0.9952 | 0.5750 | - |
470
+ | 1.4465 | 230 | 0.0 | 0.9922 | 0.5744 | - |
471
+ | 1.5094 | 240 | 0.0 | 0.9923 | 0.5726 | - |
472
+ | 1.0126 | 250 | 0.229 | 0.9930 | 0.5729 | - |
473
+ | 1.0755 | 260 | 2.2061 | 0.9435 | 0.5880 | - |
474
+ | 1.1384 | 270 | 2.7711 | 0.8892 | 0.6078 | - |
475
+ | 1.2013 | 280 | 0.7528 | 0.8886 | 0.6148 | - |
476
+ | 1.2642 | 290 | 0.386 | 0.8927 | 0.6162 | - |
477
+ | 1.3270 | 300 | 0.8902 | 0.8710 | 0.6267 | - |
478
+ | 1.3899 | 310 | 0.9534 | 0.8429 | 0.6337 | - |
479
+ | 1.4403 | 318 | - | - | - | 0.6222 |
480
 
481
 
482
  ### Framework Versions
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:944d69b0e22c70edbadcb4a35df9b7c8243f8601d9962798cbea41342b1c6406
3
  size 470637416
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a480f8a3b0abde34feef318b982835792b5781f388c0cbeb144e8d54ef77f2a3
3
  size 470637416