|
--- |
|
tags: |
|
- mteb |
|
model-index: |
|
- name: zpoint_large_embedding_zh |
|
results: |
|
- task: |
|
type: STS |
|
dataset: |
|
type: C-MTEB/AFQMC |
|
name: MTEB AFQMC |
|
config: default |
|
split: validation |
|
revision: None |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 56.52479321107392 |
|
- type: cos_sim_spearman |
|
value: 60.72175935031135 |
|
- type: euclidean_pearson |
|
value: 59.40990657564856 |
|
- type: euclidean_spearman |
|
value: 60.72175934804556 |
|
- type: manhattan_pearson |
|
value: 59.4134322847349 |
|
- type: manhattan_spearman |
|
value: 60.724413114688225 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: C-MTEB/ATEC |
|
name: MTEB ATEC |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 56.492631347325464 |
|
- type: cos_sim_spearman |
|
value: 58.765171687177656 |
|
- type: euclidean_pearson |
|
value: 63.236364373113844 |
|
- type: euclidean_spearman |
|
value: 58.765171686714865 |
|
- type: manhattan_pearson |
|
value: 63.22241814845751 |
|
- type: manhattan_spearman |
|
value: 58.762780342648234 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_reviews_multi |
|
name: MTEB AmazonReviewsClassification (zh) |
|
config: zh |
|
split: test |
|
revision: 1399c76144fd37290681b995c656ef9b2e06e26d |
|
metrics: |
|
- type: accuracy |
|
value: 49.72 |
|
- type: f1 |
|
value: 46.588683657317084 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: C-MTEB/BQ |
|
name: MTEB BQ |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 73.07779128771674 |
|
- type: cos_sim_spearman |
|
value: 75.03682691328844 |
|
- type: euclidean_pearson |
|
value: 73.68098259699073 |
|
- type: euclidean_spearman |
|
value: 75.03683037648963 |
|
- type: manhattan_pearson |
|
value: 73.66963332679124 |
|
- type: manhattan_spearman |
|
value: 75.02269337817758 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: C-MTEB/CLSClusteringP2P |
|
name: MTEB CLSClusteringP2P |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: v_measure |
|
value: 58.2897067752906 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: C-MTEB/CLSClusteringS2S |
|
name: MTEB CLSClusteringS2S |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: v_measure |
|
value: 48.79170511177673 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: C-MTEB/CMedQAv1 |
|
name: MTEB CMedQAv1 |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map |
|
value: 91.10738371185181 |
|
- type: mrr |
|
value: 92.82496031746031 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: C-MTEB/CMedQAv2 |
|
name: MTEB CMedQAv2 |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map |
|
value: 90.06959035874831 |
|
- type: mrr |
|
value: 92.00789682539683 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: C-MTEB/CmedqaRetrieval |
|
name: MTEB CmedqaRetrieval |
|
config: default |
|
split: dev |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 27.132 |
|
- type: map_at_10 |
|
value: 40.400999999999996 |
|
- type: map_at_100 |
|
value: 42.246 |
|
- type: map_at_1000 |
|
value: 42.351 |
|
- type: map_at_3 |
|
value: 35.94 |
|
- type: map_at_5 |
|
value: 38.527 |
|
- type: mrr_at_1 |
|
value: 41.285 |
|
- type: mrr_at_10 |
|
value: 49.474000000000004 |
|
- type: mrr_at_100 |
|
value: 50.4 |
|
- type: mrr_at_1000 |
|
value: 50.438 |
|
- type: mrr_at_3 |
|
value: 46.891 |
|
- type: mrr_at_5 |
|
value: 48.353 |
|
- type: ndcg_at_1 |
|
value: 41.285 |
|
- type: ndcg_at_10 |
|
value: 47.159 |
|
- type: ndcg_at_100 |
|
value: 54.163 |
|
- type: ndcg_at_1000 |
|
value: 55.921 |
|
- type: ndcg_at_3 |
|
value: 41.678 |
|
- type: ndcg_at_5 |
|
value: 44.069 |
|
- type: precision_at_1 |
|
value: 41.285 |
|
- type: precision_at_10 |
|
value: 10.468 |
|
- type: precision_at_100 |
|
value: 1.611 |
|
- type: precision_at_1000 |
|
value: 0.183 |
|
- type: precision_at_3 |
|
value: 23.648 |
|
- type: precision_at_5 |
|
value: 17.229 |
|
- type: recall_at_1 |
|
value: 27.132 |
|
- type: recall_at_10 |
|
value: 57.977999999999994 |
|
- type: recall_at_100 |
|
value: 86.88 |
|
- type: recall_at_1000 |
|
value: 98.586 |
|
- type: recall_at_3 |
|
value: 41.487 |
|
- type: recall_at_5 |
|
value: 48.79 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: C-MTEB/CMNLI |
|
name: MTEB Cmnli |
|
config: default |
|
split: validation |
|
revision: None |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 86.06133493686109 |
|
- type: cos_sim_ap |
|
value: 92.54288511740305 |
|
- type: cos_sim_f1 |
|
value: 86.85572811163628 |
|
- type: cos_sim_precision |
|
value: 83.72748969407681 |
|
- type: cos_sim_recall |
|
value: 90.22679448211363 |
|
- type: dot_accuracy |
|
value: 86.06133493686109 |
|
- type: dot_ap |
|
value: 92.53922591080917 |
|
- type: dot_f1 |
|
value: 86.85572811163628 |
|
- type: dot_precision |
|
value: 83.72748969407681 |
|
- type: dot_recall |
|
value: 90.22679448211363 |
|
- type: euclidean_accuracy |
|
value: 86.06133493686109 |
|
- type: euclidean_ap |
|
value: 92.54287994398305 |
|
- type: euclidean_f1 |
|
value: 86.85572811163628 |
|
- type: euclidean_precision |
|
value: 83.72748969407681 |
|
- type: euclidean_recall |
|
value: 90.22679448211363 |
|
- type: manhattan_accuracy |
|
value: 86.01322910402887 |
|
- type: manhattan_ap |
|
value: 92.53060255301997 |
|
- type: manhattan_f1 |
|
value: 86.81441683456458 |
|
- type: manhattan_precision |
|
value: 83.27249302125833 |
|
- type: manhattan_recall |
|
value: 90.67103109656301 |
|
- type: max_accuracy |
|
value: 86.06133493686109 |
|
- type: max_ap |
|
value: 92.54288511740305 |
|
- type: max_f1 |
|
value: 86.85572811163628 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: C-MTEB/CovidRetrieval |
|
name: MTEB CovidRetrieval |
|
config: default |
|
split: dev |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 78.899 |
|
- type: map_at_10 |
|
value: 86.232 |
|
- type: map_at_100 |
|
value: 86.331 |
|
- type: map_at_1000 |
|
value: 86.332 |
|
- type: map_at_3 |
|
value: 85.256 |
|
- type: map_at_5 |
|
value: 85.883 |
|
- type: mrr_at_1 |
|
value: 79.347 |
|
- type: mrr_at_10 |
|
value: 86.252 |
|
- type: mrr_at_100 |
|
value: 86.342 |
|
- type: mrr_at_1000 |
|
value: 86.343 |
|
- type: mrr_at_3 |
|
value: 85.283 |
|
- type: mrr_at_5 |
|
value: 85.91 |
|
- type: ndcg_at_1 |
|
value: 79.347 |
|
- type: ndcg_at_10 |
|
value: 89.143 |
|
- type: ndcg_at_100 |
|
value: 89.541 |
|
- type: ndcg_at_1000 |
|
value: 89.58 |
|
- type: ndcg_at_3 |
|
value: 87.227 |
|
- type: ndcg_at_5 |
|
value: 88.31400000000001 |
|
- type: precision_at_1 |
|
value: 79.347 |
|
- type: precision_at_10 |
|
value: 9.905 |
|
- type: precision_at_100 |
|
value: 1.0070000000000001 |
|
- type: precision_at_1000 |
|
value: 0.101 |
|
- type: precision_at_3 |
|
value: 31.261 |
|
- type: precision_at_5 |
|
value: 19.305 |
|
- type: recall_at_1 |
|
value: 78.899 |
|
- type: recall_at_10 |
|
value: 97.99799999999999 |
|
- type: recall_at_100 |
|
value: 99.684 |
|
- type: recall_at_1000 |
|
value: 100 |
|
- type: recall_at_3 |
|
value: 92.808 |
|
- type: recall_at_5 |
|
value: 95.46900000000001 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: C-MTEB/DuRetrieval |
|
name: MTEB DuRetrieval |
|
config: default |
|
split: dev |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 27.107999999999997 |
|
- type: map_at_10 |
|
value: 82.525 |
|
- type: map_at_100 |
|
value: 85.168 |
|
- type: map_at_1000 |
|
value: 85.194 |
|
- type: map_at_3 |
|
value: 57.74399999999999 |
|
- type: map_at_5 |
|
value: 72.53699999999999 |
|
- type: mrr_at_1 |
|
value: 92.30000000000001 |
|
- type: mrr_at_10 |
|
value: 94.705 |
|
- type: mrr_at_100 |
|
value: 94.76599999999999 |
|
- type: mrr_at_1000 |
|
value: 94.76599999999999 |
|
- type: mrr_at_3 |
|
value: 94.55 |
|
- type: mrr_at_5 |
|
value: 94.64 |
|
- type: ndcg_at_1 |
|
value: 92.30000000000001 |
|
- type: ndcg_at_10 |
|
value: 89.23100000000001 |
|
- type: ndcg_at_100 |
|
value: 91.556 |
|
- type: ndcg_at_1000 |
|
value: 91.81700000000001 |
|
- type: ndcg_at_3 |
|
value: 88.558 |
|
- type: ndcg_at_5 |
|
value: 87.316 |
|
- type: precision_at_1 |
|
value: 92.30000000000001 |
|
- type: precision_at_10 |
|
value: 42.38 |
|
- type: precision_at_100 |
|
value: 4.818 |
|
- type: precision_at_1000 |
|
value: 0.488 |
|
- type: precision_at_3 |
|
value: 79.14999999999999 |
|
- type: precision_at_5 |
|
value: 66.63 |
|
- type: recall_at_1 |
|
value: 27.107999999999997 |
|
- type: recall_at_10 |
|
value: 89.914 |
|
- type: recall_at_100 |
|
value: 97.658 |
|
- type: recall_at_1000 |
|
value: 99.00099999999999 |
|
- type: recall_at_3 |
|
value: 59.673 |
|
- type: recall_at_5 |
|
value: 76.437 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: C-MTEB/EcomRetrieval |
|
name: MTEB EcomRetrieval |
|
config: default |
|
split: dev |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 55.00000000000001 |
|
- type: map_at_10 |
|
value: 65.57600000000001 |
|
- type: map_at_100 |
|
value: 66.096 |
|
- type: map_at_1000 |
|
value: 66.103 |
|
- type: map_at_3 |
|
value: 63.217 |
|
- type: map_at_5 |
|
value: 64.562 |
|
- type: mrr_at_1 |
|
value: 55.00000000000001 |
|
- type: mrr_at_10 |
|
value: 65.57600000000001 |
|
- type: mrr_at_100 |
|
value: 66.096 |
|
- type: mrr_at_1000 |
|
value: 66.103 |
|
- type: mrr_at_3 |
|
value: 63.217 |
|
- type: mrr_at_5 |
|
value: 64.562 |
|
- type: ndcg_at_1 |
|
value: 55.00000000000001 |
|
- type: ndcg_at_10 |
|
value: 70.74000000000001 |
|
- type: ndcg_at_100 |
|
value: 73.001 |
|
- type: ndcg_at_1000 |
|
value: 73.223 |
|
- type: ndcg_at_3 |
|
value: 65.837 |
|
- type: ndcg_at_5 |
|
value: 68.264 |
|
- type: precision_at_1 |
|
value: 55.00000000000001 |
|
- type: precision_at_10 |
|
value: 8.7 |
|
- type: precision_at_100 |
|
value: 0.97 |
|
- type: precision_at_1000 |
|
value: 0.099 |
|
- type: precision_at_3 |
|
value: 24.467 |
|
- type: precision_at_5 |
|
value: 15.86 |
|
- type: recall_at_1 |
|
value: 55.00000000000001 |
|
- type: recall_at_10 |
|
value: 87 |
|
- type: recall_at_100 |
|
value: 97 |
|
- type: recall_at_1000 |
|
value: 98.8 |
|
- type: recall_at_3 |
|
value: 73.4 |
|
- type: recall_at_5 |
|
value: 79.3 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: C-MTEB/IFlyTek-classification |
|
name: MTEB IFlyTek |
|
config: default |
|
split: validation |
|
revision: None |
|
metrics: |
|
- type: accuracy |
|
value: 51.696806464024625 |
|
- type: f1 |
|
value: 40.02655259854763 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: C-MTEB/JDReview-classification |
|
name: MTEB JDReview |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: accuracy |
|
value: 88.87429643527206 |
|
- type: ap |
|
value: 59.89821610336161 |
|
- type: f1 |
|
value: 83.98100504939507 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: C-MTEB/LCQMC |
|
name: MTEB LCQMC |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 72.59510783330644 |
|
- type: cos_sim_spearman |
|
value: 79.75022839599451 |
|
- type: euclidean_pearson |
|
value: 79.54475341768782 |
|
- type: euclidean_spearman |
|
value: 79.75021730266204 |
|
- type: manhattan_pearson |
|
value: 79.53741020350834 |
|
- type: manhattan_spearman |
|
value: 79.74152434784455 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: C-MTEB/Mmarco-reranking |
|
name: MTEB MMarcoReranking |
|
config: default |
|
split: dev |
|
revision: None |
|
metrics: |
|
- type: map |
|
value: 38.86925357762224 |
|
- type: mrr |
|
value: 38.17460317460318 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: C-MTEB/MMarcoRetrieval |
|
name: MTEB MMarcoRetrieval |
|
config: default |
|
split: dev |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 68.731 |
|
- type: map_at_10 |
|
value: 78.52 |
|
- type: map_at_100 |
|
value: 78.792 |
|
- type: map_at_1000 |
|
value: 78.797 |
|
- type: map_at_3 |
|
value: 76.586 |
|
- type: map_at_5 |
|
value: 77.876 |
|
- type: mrr_at_1 |
|
value: 71.003 |
|
- type: mrr_at_10 |
|
value: 79.03 |
|
- type: mrr_at_100 |
|
value: 79.27 |
|
- type: mrr_at_1000 |
|
value: 79.274 |
|
- type: mrr_at_3 |
|
value: 77.373 |
|
- type: mrr_at_5 |
|
value: 78.46600000000001 |
|
- type: ndcg_at_1 |
|
value: 71.003 |
|
- type: ndcg_at_10 |
|
value: 82.381 |
|
- type: ndcg_at_100 |
|
value: 83.504 |
|
- type: ndcg_at_1000 |
|
value: 83.627 |
|
- type: ndcg_at_3 |
|
value: 78.78699999999999 |
|
- type: ndcg_at_5 |
|
value: 80.94 |
|
- type: precision_at_1 |
|
value: 71.003 |
|
- type: precision_at_10 |
|
value: 9.961 |
|
- type: precision_at_100 |
|
value: 1.05 |
|
- type: precision_at_1000 |
|
value: 0.106 |
|
- type: precision_at_3 |
|
value: 29.694 |
|
- type: precision_at_5 |
|
value: 18.963 |
|
- type: recall_at_1 |
|
value: 68.731 |
|
- type: recall_at_10 |
|
value: 93.697 |
|
- type: recall_at_100 |
|
value: 98.546 |
|
- type: recall_at_1000 |
|
value: 99.515 |
|
- type: recall_at_3 |
|
value: 84.328 |
|
- type: recall_at_5 |
|
value: 89.42 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (zh-CN) |
|
config: zh-CN |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 76.79219905850707 |
|
- type: f1 |
|
value: 73.15228001501512 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (zh-CN) |
|
config: zh-CN |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 84.9562878278413 |
|
- type: f1 |
|
value: 84.0910677219451 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: C-MTEB/MedicalRetrieval |
|
name: MTEB MedicalRetrieval |
|
config: default |
|
split: dev |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 57.8 |
|
- type: map_at_10 |
|
value: 64.732 |
|
- type: map_at_100 |
|
value: 65.315 |
|
- type: map_at_1000 |
|
value: 65.347 |
|
- type: map_at_3 |
|
value: 63.14999999999999 |
|
- type: map_at_5 |
|
value: 63.934999999999995 |
|
- type: mrr_at_1 |
|
value: 57.99999999999999 |
|
- type: mrr_at_10 |
|
value: 64.852 |
|
- type: mrr_at_100 |
|
value: 65.435 |
|
- type: mrr_at_1000 |
|
value: 65.467 |
|
- type: mrr_at_3 |
|
value: 63.266999999999996 |
|
- type: mrr_at_5 |
|
value: 64.072 |
|
- type: ndcg_at_1 |
|
value: 57.8 |
|
- type: ndcg_at_10 |
|
value: 68.14 |
|
- type: ndcg_at_100 |
|
value: 71.04899999999999 |
|
- type: ndcg_at_1000 |
|
value: 71.856 |
|
- type: ndcg_at_3 |
|
value: 64.813 |
|
- type: ndcg_at_5 |
|
value: 66.241 |
|
- type: precision_at_1 |
|
value: 57.8 |
|
- type: precision_at_10 |
|
value: 7.89 |
|
- type: precision_at_100 |
|
value: 0.927 |
|
- type: precision_at_1000 |
|
value: 0.099 |
|
- type: precision_at_3 |
|
value: 23.200000000000003 |
|
- type: precision_at_5 |
|
value: 14.62 |
|
- type: recall_at_1 |
|
value: 57.8 |
|
- type: recall_at_10 |
|
value: 78.9 |
|
- type: recall_at_100 |
|
value: 92.7 |
|
- type: recall_at_1000 |
|
value: 99 |
|
- type: recall_at_3 |
|
value: 69.6 |
|
- type: recall_at_5 |
|
value: 73.1 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: C-MTEB/MultilingualSentiment-classification |
|
name: MTEB MultilingualSentiment |
|
config: default |
|
split: validation |
|
revision: None |
|
metrics: |
|
- type: accuracy |
|
value: 79.22333333333333 |
|
- type: f1 |
|
value: 79.01276765455862 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: C-MTEB/OCNLI |
|
name: MTEB Ocnli |
|
config: default |
|
split: validation |
|
revision: None |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 85.32755820249052 |
|
- type: cos_sim_ap |
|
value: 90.56118966152913 |
|
- type: cos_sim_f1 |
|
value: 86.28428927680798 |
|
- type: cos_sim_precision |
|
value: 81.75803402646503 |
|
- type: cos_sim_recall |
|
value: 91.34107708553326 |
|
- type: dot_accuracy |
|
value: 85.32755820249052 |
|
- type: dot_ap |
|
value: 90.56120405888693 |
|
- type: dot_f1 |
|
value: 86.28428927680798 |
|
- type: dot_precision |
|
value: 81.75803402646503 |
|
- type: dot_recall |
|
value: 91.34107708553326 |
|
- type: euclidean_accuracy |
|
value: 85.32755820249052 |
|
- type: euclidean_ap |
|
value: 90.56118966152913 |
|
- type: euclidean_f1 |
|
value: 86.28428927680798 |
|
- type: euclidean_precision |
|
value: 81.75803402646503 |
|
- type: euclidean_recall |
|
value: 91.34107708553326 |
|
- type: manhattan_accuracy |
|
value: 85.43584190579317 |
|
- type: manhattan_ap |
|
value: 90.52296007826511 |
|
- type: manhattan_f1 |
|
value: 86.42099949520444 |
|
- type: manhattan_precision |
|
value: 82.7852998065764 |
|
- type: manhattan_recall |
|
value: 90.3907074973601 |
|
- type: max_accuracy |
|
value: 85.43584190579317 |
|
- type: max_ap |
|
value: 90.56120405888693 |
|
- type: max_f1 |
|
value: 86.42099949520444 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: C-MTEB/OnlineShopping-classification |
|
name: MTEB OnlineShopping |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: accuracy |
|
value: 94.87999999999998 |
|
- type: ap |
|
value: 93.12892276945414 |
|
- type: f1 |
|
value: 94.86921245385685 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: C-MTEB/PAWSX |
|
name: MTEB PAWSX |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 38.4367277229591 |
|
- type: cos_sim_spearman |
|
value: 45.942712312151656 |
|
- type: euclidean_pearson |
|
value: 44.96055989566686 |
|
- type: euclidean_spearman |
|
value: 45.94279939044163 |
|
- type: manhattan_pearson |
|
value: 44.979762134562925 |
|
- type: manhattan_spearman |
|
value: 45.96004430328375 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: C-MTEB/QBQTC |
|
name: MTEB QBQTC |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 41.45428416733968 |
|
- type: cos_sim_spearman |
|
value: 43.462057455255845 |
|
- type: euclidean_pearson |
|
value: 38.20089604291246 |
|
- type: euclidean_spearman |
|
value: 43.46288438624811 |
|
- type: manhattan_pearson |
|
value: 38.175045608320694 |
|
- type: manhattan_spearman |
|
value: 43.468885824666344 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts22-crosslingual-sts |
|
name: MTEB STS22 (zh) |
|
config: zh |
|
split: test |
|
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 65.61911213187778 |
|
- type: cos_sim_spearman |
|
value: 66.70525921118497 |
|
- type: euclidean_pearson |
|
value: 65.35554462551515 |
|
- type: euclidean_spearman |
|
value: 66.70525921118497 |
|
- type: manhattan_pearson |
|
value: 65.25174169329627 |
|
- type: manhattan_spearman |
|
value: 66.6550752269368 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: C-MTEB/STSB |
|
name: MTEB STSB |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 81.27160581568329 |
|
- type: cos_sim_spearman |
|
value: 83.34482829304406 |
|
- type: euclidean_pearson |
|
value: 82.98079434913451 |
|
- type: euclidean_spearman |
|
value: 83.34503180775212 |
|
- type: manhattan_pearson |
|
value: 82.95256917013506 |
|
- type: manhattan_spearman |
|
value: 83.31034894907503 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: C-MTEB/T2Reranking |
|
name: MTEB T2Reranking |
|
config: default |
|
split: dev |
|
revision: None |
|
metrics: |
|
- type: map |
|
value: 69.29054152015013 |
|
- type: mrr |
|
value: 79.73472208788729 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: C-MTEB/T2Retrieval |
|
name: MTEB T2Retrieval |
|
config: default |
|
split: dev |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 27 |
|
- type: map_at_10 |
|
value: 75.871 |
|
- type: map_at_100 |
|
value: 79.664 |
|
- type: map_at_1000 |
|
value: 79.725 |
|
- type: map_at_3 |
|
value: 53.14 |
|
- type: map_at_5 |
|
value: 65.365 |
|
- type: mrr_at_1 |
|
value: 88.642 |
|
- type: mrr_at_10 |
|
value: 91.732 |
|
- type: mrr_at_100 |
|
value: 91.818 |
|
- type: mrr_at_1000 |
|
value: 91.821 |
|
- type: mrr_at_3 |
|
value: 91.217 |
|
- type: mrr_at_5 |
|
value: 91.561 |
|
- type: ndcg_at_1 |
|
value: 88.642 |
|
- type: ndcg_at_10 |
|
value: 83.815 |
|
- type: ndcg_at_100 |
|
value: 87.689 |
|
- type: ndcg_at_1000 |
|
value: 88.266 |
|
- type: ndcg_at_3 |
|
value: 84.807 |
|
- type: ndcg_at_5 |
|
value: 83.53699999999999 |
|
- type: precision_at_1 |
|
value: 88.642 |
|
- type: precision_at_10 |
|
value: 41.725 |
|
- type: precision_at_100 |
|
value: 5.024 |
|
- type: precision_at_1000 |
|
value: 0.516 |
|
- type: precision_at_3 |
|
value: 74.10600000000001 |
|
- type: precision_at_5 |
|
value: 62.192 |
|
- type: recall_at_1 |
|
value: 27 |
|
- type: recall_at_10 |
|
value: 83.292 |
|
- type: recall_at_100 |
|
value: 95.66799999999999 |
|
- type: recall_at_1000 |
|
value: 98.56 |
|
- type: recall_at_3 |
|
value: 55.111 |
|
- type: recall_at_5 |
|
value: 69.327 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: C-MTEB/TNews-classification |
|
name: MTEB TNews |
|
config: default |
|
split: validation |
|
revision: None |
|
metrics: |
|
- type: accuracy |
|
value: 54.346 |
|
- type: f1 |
|
value: 52.302508458396055 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: C-MTEB/ThuNewsClusteringP2P |
|
name: MTEB ThuNewsClusteringP2P |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: v_measure |
|
value: 72.47709523787981 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: C-MTEB/ThuNewsClusteringS2S |
|
name: MTEB ThuNewsClusteringS2S |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: v_measure |
|
value: 69.35293863978707 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: C-MTEB/VideoRetrieval |
|
name: MTEB VideoRetrieval |
|
config: default |
|
split: dev |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 64.60000000000001 |
|
- type: map_at_10 |
|
value: 75.683 |
|
- type: map_at_100 |
|
value: 75.961 |
|
- type: map_at_1000 |
|
value: 75.96199999999999 |
|
- type: map_at_3 |
|
value: 74.083 |
|
- type: map_at_5 |
|
value: 75.03800000000001 |
|
- type: mrr_at_1 |
|
value: 64.60000000000001 |
|
- type: mrr_at_10 |
|
value: 75.683 |
|
- type: mrr_at_100 |
|
value: 75.961 |
|
- type: mrr_at_1000 |
|
value: 75.96199999999999 |
|
- type: mrr_at_3 |
|
value: 74.083 |
|
- type: mrr_at_5 |
|
value: 75.03800000000001 |
|
- type: ndcg_at_1 |
|
value: 64.60000000000001 |
|
- type: ndcg_at_10 |
|
value: 80.26299999999999 |
|
- type: ndcg_at_100 |
|
value: 81.487 |
|
- type: ndcg_at_1000 |
|
value: 81.5 |
|
- type: ndcg_at_3 |
|
value: 77.003 |
|
- type: ndcg_at_5 |
|
value: 78.708 |
|
- type: precision_at_1 |
|
value: 64.60000000000001 |
|
- type: precision_at_10 |
|
value: 9.43 |
|
- type: precision_at_100 |
|
value: 0.997 |
|
- type: precision_at_1000 |
|
value: 0.1 |
|
- type: precision_at_3 |
|
value: 28.467 |
|
- type: precision_at_5 |
|
value: 17.9 |
|
- type: recall_at_1 |
|
value: 64.60000000000001 |
|
- type: recall_at_10 |
|
value: 94.3 |
|
- type: recall_at_100 |
|
value: 99.7 |
|
- type: recall_at_1000 |
|
value: 99.8 |
|
- type: recall_at_3 |
|
value: 85.39999999999999 |
|
- type: recall_at_5 |
|
value: 89.5 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: C-MTEB/waimai-classification |
|
name: MTEB Waimai |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: accuracy |
|
value: 89.36 |
|
- type: ap |
|
value: 75.26507519569006 |
|
- type: f1 |
|
value: 87.89845508858562 |
|
language: |
|
- zh |
|
license: mit |
|
library_name: sentence-transformers |
|
--- |
|
<h2 align="left">ZPoint Large Embedding for Chinese</h2> |
|
|
|
- **[2024-06-04]** Release zpoint_large_embedding_zh, and upload model weight to huggingface |
|
- **[2024-06-05]** Add training details |
|
|
|
### Training Details |
|
|
|
**Base Model** |
|
1) We chose [Stella](https://huggingface.co/infgrad/stella-mrl-large-zh-v3.5-1792d) as our base model. |
|
|
|
**Training Data** |
|
1) **Hard negative samping** |
|
- For retrieval task, We sampled 10 hard negative passages/answers from top50-top200 related passages/answers for each query. |
|
- For classification/clustering tasks, we sampled 5 hard negative samples from other classes/cluster for each sample. |
|
- For classification/clustering tasks, we also used the category names of each class and cluster as positive and negative samples. |
|
|
|
2) **Data synthesis by LLM (ZPoint-72B)** |
|
- For retrieval tasks, we used LLM to rewrite each query, generating five different rewritten results. |
|
- For retrieval tasks, we also generated five new queries for some documents by LLM. |
|
- For non-retrieval tasks, we used LLM to rewrite the queries, generating five rewritten results for each query. |
|
- Finally, total amount of synthesized data is about 30 million. |
|
|
|
3) **Collect more data for retrieval-type tasks** |
|
- [miracl/miracl](https://huggingface.co/datasets/miracl/miracl) |
|
- [FreedomIntelligence/Huatuo26M-Lite](https://huggingface.co/datasets/FreedomIntelligence/Huatuo26M-Lite) |
|
- [PaddlePaddle/dureader_robust](https://huggingface.co/datasets/PaddlePaddle/dureader_robust) **C-MTEB test filtered** |
|
- [THUIR/T2Ranking](https://huggingface.co/datasets/THUIR/T2Ranking) **C-MTEB test filtered** |
|
- [Shitao/bge-reranker-data](https://huggingface.co/datasets/Shitao/bge-reranker-data) |
|
- [Shitao/MLDR](https://huggingface.co/datasets/Shitao/MLDR) |
|
- ... |
|
|
|
***We constructed a dataset of approximately 100 million training samples through collection, machine translation, and LLM synthesis. This dataset includes data from various fields such as healthcare, law, electricity, automotive, and 3C (Consumer Electronics).*** |
|
|
|
|
|
**Training loss** |
|
1) Multi-Task loss like [Piccolo](https://huggingface.co/sensenova/piccolo-large-zh-v2) |
|
2) Matryoshka Representation Learning |
|
|
|
|
|
### Example |
|
|
|
```python |
|
from sentence_transformers import SentenceTransformer |
|
sentences1 = ["这个产品真垃圾"] |
|
sentences2 = ["我太喜欢这个产品了"] |
|
model = SentenceTransformer('iampanda/zpoint_large_embedding_zh') |
|
embeddings_1 = model.encode(sentences1, normalize_embeddings=True) |
|
embeddings_2 = model.encode(sentences2, normalize_embeddings=True) |
|
similarity = embeddings_1 @ embeddings_2.T |
|
print(similarity) |
|
``` |
