diff --git "a/README.md" "b/README.md" new file mode 100644--- /dev/null +++ "b/README.md" @@ -0,0 +1,9028 @@ +--- +library_name: model2vec +license: mit +model-index: +- name: potion-base-8M + results: + - dataset: + config: en-ext + name: MTEB AmazonCounterfactualClassification (en-ext) + revision: e8379541af4e31359cca9fbcf4b00f2671dba205 + split: test + type: mteb/amazon_counterfactual + metrics: + - type: accuracy + value: 72.15142428785607 + - type: ap + value: 20.626102291010103 + - type: ap_weighted + value: 20.626102291010103 + - type: f1 + value: 59.187001923736894 + - type: f1_weighted + value: 77.34906471545477 + - type: main_score + value: 72.15142428785607 + task: + type: Classification + - dataset: + config: en + name: MTEB AmazonCounterfactualClassification (en) + revision: e8379541af4e31359cca9fbcf4b00f2671dba205 + split: test + type: mteb/amazon_counterfactual + metrics: + - type: accuracy + value: 71.7910447761194 + - type: ap + value: 33.038020188116036 + - type: ap_weighted + value: 33.038020188116036 + - type: f1 + value: 65.03799728338926 + - type: f1_weighted + value: 74.32788084269461 + - type: main_score + value: 71.7910447761194 + task: + type: Classification + - dataset: + config: default + name: MTEB AmazonPolarityClassification (default) + revision: e2d317d38cd51312af73b3d32a06d1a08b442046 + split: test + type: mteb/amazon_polarity + metrics: + - type: accuracy + value: 72.47644999999999 + - type: ap + value: 66.91002822830875 + - type: ap_weighted + value: 66.91002822830875 + - type: f1 + value: 72.2600863044581 + - type: f1_weighted + value: 72.2600863044581 + - type: main_score + value: 72.47644999999999 + task: + type: Classification + - dataset: + config: en + name: MTEB AmazonReviewsClassification (en) + revision: 1399c76144fd37290681b995c656ef9b2e06e26d + split: test + type: mteb/amazon_reviews_multi + metrics: + - type: accuracy + value: 36.012 + - type: f1 + value: 35.38209336470206 + - type: f1_weighted + value: 35.38209336470206 + - type: main_score + value: 36.012 + task: + type: Classification + - dataset: + config: default + name: MTEB ArguAna (default) + revision: c22ab2a51041ffd869aaddef7af8d8215647e41a + split: test + type: mteb/arguana + metrics: + - type: main_score + value: 41.966 + - type: map_at_1 + value: 21.124000000000002 + - type: map_at_10 + value: 34.335 + - type: map_at_100 + value: 35.618 + - type: map_at_1000 + value: 35.653 + - type: map_at_20 + value: 35.21 + - type: map_at_3 + value: 30.287 + - type: map_at_5 + value: 32.364 + - type: mrr_at_1 + value: 21.62162162162162 + - type: mrr_at_10 + value: 34.509104969631224 + - type: mrr_at_100 + value: 35.79229946325059 + - type: mrr_at_1000 + value: 35.82767320968403 + - type: mrr_at_20 + value: 35.38485605181455 + - type: mrr_at_3 + value: 30.405405405405343 + - type: mrr_at_5 + value: 32.539118065433755 + - type: nauc_map_at_1000_diff1 + value: 7.960826255212609 + - type: nauc_map_at_1000_max + value: -0.036381315067780806 + - type: nauc_map_at_1000_std + value: 4.317766293607543 + - type: nauc_map_at_100_diff1 + value: 7.96318422584977 + - type: nauc_map_at_100_max + value: -0.007800758201736421 + - type: nauc_map_at_100_std + value: 4.362078927714198 + - type: nauc_map_at_10_diff1 + value: 7.718022643886373 + - type: nauc_map_at_10_max + value: -0.28312250079415263 + - type: nauc_map_at_10_std + value: 4.079196099329437 + - type: nauc_map_at_1_diff1 + value: 9.240393281366906 + - type: nauc_map_at_1_max + value: -4.35798405693968 + - type: nauc_map_at_1_std + value: 1.5076565659508505 + - type: nauc_map_at_20_diff1 + value: 8.028053857747947 + - type: nauc_map_at_20_max + value: 0.0719807687813251 + - type: nauc_map_at_20_std + value: 4.394812024847373 + - type: nauc_map_at_3_diff1 + value: 7.953781299828595 + - type: nauc_map_at_3_max + value: -0.573072664182506 + - type: nauc_map_at_3_std + value: 3.110821611511372 + - type: nauc_map_at_5_diff1 + value: 7.3135486297676415 + - type: nauc_map_at_5_max + value: -1.2456304709603878 + - type: nauc_map_at_5_std + value: 3.2332006196074805 + - type: nauc_mrr_at_1000_diff1 + value: 6.511595076207588 + - type: nauc_mrr_at_1000_max + value: -0.4777573692286575 + - type: nauc_mrr_at_1000_std + value: 4.19518565742107 + - type: nauc_mrr_at_100_diff1 + value: 6.515632481906436 + - type: nauc_mrr_at_100_max + value: -0.44877259463397945 + - type: nauc_mrr_at_100_std + value: 4.23945026873963 + - type: nauc_mrr_at_10_diff1 + value: 6.325261150908693 + - type: nauc_mrr_at_10_max + value: -0.6968688229450172 + - type: nauc_mrr_at_10_std + value: 3.9631303923167294 + - type: nauc_mrr_at_1_diff1 + value: 7.4844946822832785 + - type: nauc_mrr_at_1_max + value: -4.0195803039697315 + - type: nauc_mrr_at_1_std + value: 1.3908984330415426 + - type: nauc_mrr_at_20_diff1 + value: 6.596479652899773 + - type: nauc_mrr_at_20_max + value: -0.3643520262705732 + - type: nauc_mrr_at_20_std + value: 4.273437423781988 + - type: nauc_mrr_at_3_diff1 + value: 6.3669450211955745 + - type: nauc_mrr_at_3_max + value: -1.2252447747465325 + - type: nauc_mrr_at_3_std + value: 2.941708547001192 + - type: nauc_mrr_at_5_diff1 + value: 5.907234785613739 + - type: nauc_mrr_at_5_max + value: -1.6860364992754489 + - type: nauc_mrr_at_5_std + value: 3.0737345356263406 + - type: nauc_ndcg_at_1000_diff1 + value: 7.9706658500975704 + - type: nauc_ndcg_at_1000_max + value: 1.5533941879318276 + - type: nauc_ndcg_at_1000_std + value: 5.933724413159287 + - type: nauc_ndcg_at_100_diff1 + value: 8.107414913432397 + - type: nauc_ndcg_at_100_max + value: 2.5869418793842778 + - type: nauc_ndcg_at_100_std + value: 7.322146884970876 + - type: nauc_ndcg_at_10_diff1 + value: 7.669807780113455 + - type: nauc_ndcg_at_10_max + value: 1.886214180834648 + - type: nauc_ndcg_at_10_std + value: 6.055781567147952 + - type: nauc_ndcg_at_1_diff1 + value: 9.240393281366906 + - type: nauc_ndcg_at_1_max + value: -4.35798405693968 + - type: nauc_ndcg_at_1_std + value: 1.5076565659508505 + - type: nauc_ndcg_at_20_diff1 + value: 8.661303229272372 + - type: nauc_ndcg_at_20_max + value: 3.303174862536166 + - type: nauc_ndcg_at_20_std + value: 7.493758825967179 + - type: nauc_ndcg_at_3_diff1 + value: 7.858281169135036 + - type: nauc_ndcg_at_3_max + value: 0.7079724865506055 + - type: nauc_ndcg_at_3_std + value: 3.7402042497720958 + - type: nauc_ndcg_at_5_diff1 + value: 6.68694262946663 + - type: nauc_ndcg_at_5_max + value: -0.43002529778264326 + - type: nauc_ndcg_at_5_std + value: 3.9597009492387265 + - type: nauc_precision_at_1000_diff1 + value: -28.217119971169463 + - type: nauc_precision_at_1000_max + value: 17.425278660692022 + - type: nauc_precision_at_1000_std + value: 46.7473304347162 + - type: nauc_precision_at_100_diff1 + value: 8.738254686624805 + - type: nauc_precision_at_100_max + value: 32.88945783040687 + - type: nauc_precision_at_100_std + value: 48.42583030760342 + - type: nauc_precision_at_10_diff1 + value: 7.873361516017592 + - type: nauc_precision_at_10_max + value: 9.802552072953949 + - type: nauc_precision_at_10_std + value: 13.506647301311148 + - type: nauc_precision_at_1_diff1 + value: 9.240393281366906 + - type: nauc_precision_at_1_max + value: -4.35798405693968 + - type: nauc_precision_at_1_std + value: 1.5076565659508505 + - type: nauc_precision_at_20_diff1 + value: 13.008220519097161 + - type: nauc_precision_at_20_max + value: 20.829507014709748 + - type: nauc_precision_at_20_std + value: 25.02998005000373 + - type: nauc_precision_at_3_diff1 + value: 7.685752623087433 + - type: nauc_precision_at_3_max + value: 4.126629771323765 + - type: nauc_precision_at_3_std + value: 5.440817692025366 + - type: nauc_precision_at_5_diff1 + value: 4.879990376967901 + - type: nauc_precision_at_5_max + value: 1.7076492862153407 + - type: nauc_precision_at_5_std + value: 6.009634283832547 + - type: nauc_recall_at_1000_diff1 + value: -28.217119971166543 + - type: nauc_recall_at_1000_max + value: 17.425278660689965 + - type: nauc_recall_at_1000_std + value: 46.74733043471749 + - type: nauc_recall_at_100_diff1 + value: 8.738254686625181 + - type: nauc_recall_at_100_max + value: 32.8894578304071 + - type: nauc_recall_at_100_std + value: 48.425830307603746 + - type: nauc_recall_at_10_diff1 + value: 7.87336151601764 + - type: nauc_recall_at_10_max + value: 9.802552072953997 + - type: nauc_recall_at_10_std + value: 13.506647301311201 + - type: nauc_recall_at_1_diff1 + value: 9.240393281366906 + - type: nauc_recall_at_1_max + value: -4.35798405693968 + - type: nauc_recall_at_1_std + value: 1.5076565659508505 + - type: nauc_recall_at_20_diff1 + value: 13.008220519097097 + - type: nauc_recall_at_20_max + value: 20.82950701470975 + - type: nauc_recall_at_20_std + value: 25.02998005000377 + - type: nauc_recall_at_3_diff1 + value: 7.685752623087458 + - type: nauc_recall_at_3_max + value: 4.126629771323791 + - type: nauc_recall_at_3_std + value: 5.440817692025401 + - type: nauc_recall_at_5_diff1 + value: 4.879990376967856 + - type: nauc_recall_at_5_max + value: 1.7076492862153638 + - type: nauc_recall_at_5_std + value: 6.009634283832578 + - type: ndcg_at_1 + value: 21.124000000000002 + - type: ndcg_at_10 + value: 41.966 + - type: ndcg_at_100 + value: 47.751 + - type: ndcg_at_1000 + value: 48.635 + - type: ndcg_at_20 + value: 45.08 + - type: ndcg_at_3 + value: 33.505 + - type: ndcg_at_5 + value: 37.266 + - type: precision_at_1 + value: 21.124000000000002 + - type: precision_at_10 + value: 6.643000000000001 + - type: precision_at_100 + value: 0.9249999999999999 + - type: precision_at_1000 + value: 0.099 + - type: precision_at_20 + value: 3.93 + - type: precision_at_3 + value: 14.296000000000001 + - type: precision_at_5 + value: 10.413 + - type: recall_at_1 + value: 21.124000000000002 + - type: recall_at_10 + value: 66.43 + - type: recall_at_100 + value: 92.461 + - type: recall_at_1000 + value: 99.289 + - type: recall_at_20 + value: 78.592 + - type: recall_at_3 + value: 42.888 + - type: recall_at_5 + value: 52.063 + task: + type: Retrieval + - dataset: + config: default + name: MTEB ArxivClusteringP2P (default) + revision: a122ad7f3f0291bf49cc6f4d32aa80929df69d5d + split: test + type: mteb/arxiv-clustering-p2p + metrics: + - type: main_score + value: 35.387660145946825 + - type: v_measure + value: 35.387660145946825 + - type: v_measure_std + value: 14.022525689022785 + task: + type: Clustering + - dataset: + config: default + name: MTEB ArxivClusteringS2S (default) + revision: f910caf1a6075f7329cdf8c1a6135696f37dbd53 + split: test + type: mteb/arxiv-clustering-s2s + metrics: + - type: main_score + value: 25.26058942964131 + - type: v_measure + value: 25.26058942964131 + - type: v_measure_std + value: 14.850432186356857 + task: + type: Clustering + - dataset: + config: default + name: MTEB AskUbuntuDupQuestions (default) + revision: 2000358ca161889fa9c082cb41daa8dcfb161a54 + split: test + type: mteb/askubuntudupquestions-reranking + metrics: + - type: main_score + value: 54.13950871400633 + - type: map + value: 54.13950871400633 + - type: mrr + value: 68.87437892978059 + - type: nAUC_map_diff1 + value: 3.489277672557011 + - type: nAUC_map_max + value: 15.848457273691064 + - type: nAUC_map_std + value: 5.166813098270773 + - type: nAUC_mrr_diff1 + value: 4.9924344024669765 + - type: nAUC_mrr_max + value: 21.861692980537956 + - type: nAUC_mrr_std + value: 8.256966784037171 + task: + type: Reranking + - dataset: + config: default + name: MTEB BIOSSES (default) + revision: d3fb88f8f02e40887cd149695127462bbcf29b4a + split: test + type: mteb/biosses-sts + metrics: + - type: cosine_pearson + value: 79.11612010879227 + - type: cosine_spearman + value: 75.85775256673794 + - type: euclidean_pearson + value: 77.46080265077437 + - type: euclidean_spearman + value: 75.85775256673794 + - type: main_score + value: 75.85775256673794 + - type: manhattan_pearson + value: 77.73191375456281 + - type: manhattan_spearman + value: 75.98908086034702 + - type: pearson + value: 79.11612010879227 + - type: spearman + value: 75.85775256673794 + task: + type: STS + - dataset: + config: default + name: MTEB Banking77Classification (default) + revision: 0fd18e25b25c072e09e0d92ab615fda904d66300 + split: test + type: mteb/banking77 + metrics: + - type: accuracy + value: 72.63636363636363 + - type: f1 + value: 71.69751597573539 + - type: f1_weighted + value: 71.69751597573539 + - type: main_score + value: 72.63636363636363 + task: + type: Classification + - dataset: + config: default + name: MTEB BiorxivClusteringP2P (default) + revision: 65b79d1d13f80053f67aca9498d9402c2d9f1f40 + split: test + type: mteb/biorxiv-clustering-p2p + metrics: + - type: main_score + value: 30.861840536151014 + - type: v_measure + value: 30.861840536151014 + - type: v_measure_std + value: 0.8096483751274005 + task: + type: Clustering + - dataset: + config: default + name: MTEB BiorxivClusteringS2S (default) + revision: 258694dd0231531bc1fd9de6ceb52a0853c6d908 + split: test + type: mteb/biorxiv-clustering-s2s + metrics: + - type: main_score + value: 20.219544420664455 + - type: v_measure + value: 20.219544420664455 + - type: v_measure_std + value: 0.7431903039116942 + task: + type: Clustering + - dataset: + config: default + name: MTEB CQADupstackAndroidRetrieval (default) + revision: f46a197baaae43b4f621051089b82a364682dfeb + split: test + type: mteb/cqadupstack-android + metrics: + - type: main_score + value: 31.835 + - type: map_at_1 + value: 19.939 + - type: map_at_10 + value: 26.924 + - type: map_at_100 + value: 28.16 + - type: map_at_1000 + value: 28.316999999999997 + - type: map_at_20 + value: 27.554000000000002 + - type: map_at_3 + value: 24.45 + - type: map_at_5 + value: 25.751 + - type: mrr_at_1 + value: 25.894134477825464 + - type: mrr_at_10 + value: 32.65152031246451 + - type: mrr_at_100 + value: 33.58362210177363 + - type: mrr_at_1000 + value: 33.66415578481638 + - type: mrr_at_20 + value: 33.158616397714056 + - type: mrr_at_3 + value: 30.51979017644255 + - type: mrr_at_5 + value: 31.67143538388174 + - type: nauc_map_at_1000_diff1 + value: 43.61649840733464 + - type: nauc_map_at_1000_max + value: 27.361709993841355 + - type: nauc_map_at_1000_std + value: -1.47509416166404 + - type: nauc_map_at_100_diff1 + value: 43.63694784277137 + - type: nauc_map_at_100_max + value: 27.3675446795805 + - type: nauc_map_at_100_std + value: -1.4918015679743737 + - type: nauc_map_at_10_diff1 + value: 43.85263484013946 + - type: nauc_map_at_10_max + value: 26.810142038619045 + - type: nauc_map_at_10_std + value: -1.9884710880957612 + - type: nauc_map_at_1_diff1 + value: 48.66149039458694 + - type: nauc_map_at_1_max + value: 25.719796249226828 + - type: nauc_map_at_1_std + value: -3.291830544258096 + - type: nauc_map_at_20_diff1 + value: 43.70511471916722 + - type: nauc_map_at_20_max + value: 27.211922285560092 + - type: nauc_map_at_20_std + value: -1.621254133243609 + - type: nauc_map_at_3_diff1 + value: 45.678378884966854 + - type: nauc_map_at_3_max + value: 26.263363796878807 + - type: nauc_map_at_3_std + value: -3.067861673919005 + - type: nauc_map_at_5_diff1 + value: 44.28820868486158 + - type: nauc_map_at_5_max + value: 27.02028586800064 + - type: nauc_map_at_5_std + value: -2.8993536712942554 + - type: nauc_mrr_at_1000_diff1 + value: 41.91452307309703 + - type: nauc_mrr_at_1000_max + value: 28.25542784321284 + - type: nauc_mrr_at_1000_std + value: -1.2881473492995474 + - type: nauc_mrr_at_100_diff1 + value: 41.887361041816355 + - type: nauc_mrr_at_100_max + value: 28.242674898536045 + - type: nauc_mrr_at_100_std + value: -1.2962789057617752 + - type: nauc_mrr_at_10_diff1 + value: 41.839392429152184 + - type: nauc_mrr_at_10_max + value: 28.18109937160502 + - type: nauc_mrr_at_10_std + value: -1.760338307129395 + - type: nauc_mrr_at_1_diff1 + value: 46.97337896088234 + - type: nauc_mrr_at_1_max + value: 28.47299575870196 + - type: nauc_mrr_at_1_std + value: -2.699423724792112 + - type: nauc_mrr_at_20_diff1 + value: 41.87609128070427 + - type: nauc_mrr_at_20_max + value: 28.275298954521837 + - type: nauc_mrr_at_20_std + value: -1.3019240483529069 + - type: nauc_mrr_at_3_diff1 + value: 43.7337496151517 + - type: nauc_mrr_at_3_max + value: 27.798267478018285 + - type: nauc_mrr_at_3_std + value: -2.840593072947404 + - type: nauc_mrr_at_5_diff1 + value: 42.334483231228894 + - type: nauc_mrr_at_5_max + value: 28.312298246235912 + - type: nauc_mrr_at_5_std + value: -2.4627148837425574 + - type: nauc_ndcg_at_1000_diff1 + value: 41.15727539315947 + - type: nauc_ndcg_at_1000_max + value: 28.221291832726013 + - type: nauc_ndcg_at_1000_std + value: 2.0023108110987686 + - type: nauc_ndcg_at_100_diff1 + value: 40.696711368737986 + - type: nauc_ndcg_at_100_max + value: 28.3380433133816 + - type: nauc_ndcg_at_100_std + value: 1.6747741379499974 + - type: nauc_ndcg_at_10_diff1 + value: 40.68084799209197 + - type: nauc_ndcg_at_10_max + value: 27.001668531808047 + - type: nauc_ndcg_at_10_std + value: -0.6698055635076909 + - type: nauc_ndcg_at_1_diff1 + value: 46.97337896088234 + - type: nauc_ndcg_at_1_max + value: 28.47299575870196 + - type: nauc_ndcg_at_1_std + value: -2.699423724792112 + - type: nauc_ndcg_at_20_diff1 + value: 40.66080469225681 + - type: nauc_ndcg_at_20_max + value: 27.65886977082646 + - type: nauc_ndcg_at_20_std + value: 0.7450066458769301 + - type: nauc_ndcg_at_3_diff1 + value: 42.76104820392522 + - type: nauc_ndcg_at_3_max + value: 26.519613853147632 + - type: nauc_ndcg_at_3_std + value: -2.4350130293906034 + - type: nauc_ndcg_at_5_diff1 + value: 41.019172353488194 + - type: nauc_ndcg_at_5_max + value: 27.496046368143357 + - type: nauc_ndcg_at_5_std + value: -2.2882580326645177 + - type: nauc_precision_at_1000_diff1 + value: -14.261675661323125 + - type: nauc_precision_at_1000_max + value: -1.183805005826827 + - type: nauc_precision_at_1000_std + value: 3.344837871953594 + - type: nauc_precision_at_100_diff1 + value: 2.705968352361474 + - type: nauc_precision_at_100_max + value: 15.123914801051598 + - type: nauc_precision_at_100_std + value: 6.622282531987529 + - type: nauc_precision_at_10_diff1 + value: 21.143497652137974 + - type: nauc_precision_at_10_max + value: 22.754667045964673 + - type: nauc_precision_at_10_std + value: 2.56769270957959 + - type: nauc_precision_at_1_diff1 + value: 46.97337896088234 + - type: nauc_precision_at_1_max + value: 28.47299575870196 + - type: nauc_precision_at_1_std + value: -2.699423724792112 + - type: nauc_precision_at_20_diff1 + value: 15.750482341955857 + - type: nauc_precision_at_20_max + value: 22.860380841938827 + - type: nauc_precision_at_20_std + value: 4.22745838192582 + - type: nauc_precision_at_3_diff1 + value: 35.61809209460161 + - type: nauc_precision_at_3_max + value: 27.0006337531976 + - type: nauc_precision_at_3_std + value: -1.4556398881692423 + - type: nauc_precision_at_5_diff1 + value: 28.851808861899496 + - type: nauc_precision_at_5_max + value: 27.469054608601784 + - type: nauc_precision_at_5_std + value: -1.1421142808937477 + - type: nauc_recall_at_1000_diff1 + value: 33.27567106545891 + - type: nauc_recall_at_1000_max + value: 30.098997951989325 + - type: nauc_recall_at_1000_std + value: 37.339251250157766 + - type: nauc_recall_at_100_diff1 + value: 29.072377336992822 + - type: nauc_recall_at_100_max + value: 28.48476566182903 + - type: nauc_recall_at_100_std + value: 14.360417936748082 + - type: nauc_recall_at_10_diff1 + value: 32.83564819819592 + - type: nauc_recall_at_10_max + value: 24.465508171060677 + - type: nauc_recall_at_10_std + value: 3.332253149508536 + - type: nauc_recall_at_1_diff1 + value: 48.66149039458694 + - type: nauc_recall_at_1_max + value: 25.719796249226828 + - type: nauc_recall_at_1_std + value: -3.291830544258096 + - type: nauc_recall_at_20_diff1 + value: 31.185350107155045 + - type: nauc_recall_at_20_max + value: 25.812923152751406 + - type: nauc_recall_at_20_std + value: 8.353054109145367 + - type: nauc_recall_at_3_diff1 + value: 40.27297484569938 + - type: nauc_recall_at_3_max + value: 23.81327189620511 + - type: nauc_recall_at_3_std + value: -2.526830052534271 + - type: nauc_recall_at_5_diff1 + value: 34.64896359382995 + - type: nauc_recall_at_5_max + value: 25.750218989139317 + - type: nauc_recall_at_5_std + value: -1.3789317138918638 + - type: ndcg_at_1 + value: 25.894000000000002 + - type: ndcg_at_10 + value: 31.835 + - type: ndcg_at_100 + value: 37.325 + - type: ndcg_at_1000 + value: 40.586 + - type: ndcg_at_20 + value: 33.714 + - type: ndcg_at_3 + value: 28.143 + - type: ndcg_at_5 + value: 29.648999999999997 + - type: precision_at_1 + value: 25.894000000000002 + - type: precision_at_10 + value: 6.194999999999999 + - type: precision_at_100 + value: 1.126 + - type: precision_at_1000 + value: 0.173 + - type: precision_at_20 + value: 3.7199999999999998 + - type: precision_at_3 + value: 13.543 + - type: precision_at_5 + value: 9.757 + - type: recall_at_1 + value: 19.939 + - type: recall_at_10 + value: 40.537 + - type: recall_at_100 + value: 64.717 + - type: recall_at_1000 + value: 87.01299999999999 + - type: recall_at_20 + value: 47.677 + - type: recall_at_3 + value: 29.301 + - type: recall_at_5 + value: 33.918 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackEnglishRetrieval (default) + revision: ad9991cb51e31e31e430383c75ffb2885547b5f0 + split: test + type: mteb/cqadupstack-english + metrics: + - type: main_score + value: 25.734 + - type: map_at_1 + value: 16.601 + - type: map_at_10 + value: 22.07 + - type: map_at_100 + value: 22.958000000000002 + - type: map_at_1000 + value: 23.074 + - type: map_at_20 + value: 22.52 + - type: map_at_3 + value: 20.137 + - type: map_at_5 + value: 21.315 + - type: mrr_at_1 + value: 20.382165605095544 + - type: mrr_at_10 + value: 25.95447881912849 + - type: mrr_at_100 + value: 26.72268332839149 + - type: mrr_at_1000 + value: 26.79228081014276 + - type: mrr_at_20 + value: 26.372942687112676 + - type: mrr_at_3 + value: 24.097664543524406 + - type: mrr_at_5 + value: 25.269639065817373 + - type: nauc_map_at_1000_diff1 + value: 39.97979443324452 + - type: nauc_map_at_1000_max + value: 13.65503993855689 + - type: nauc_map_at_1000_std + value: -2.0265680574493286 + - type: nauc_map_at_100_diff1 + value: 40.04134376146643 + - type: nauc_map_at_100_max + value: 13.602473622919186 + - type: nauc_map_at_100_std + value: -2.1531627932652073 + - type: nauc_map_at_10_diff1 + value: 40.321538712092966 + - type: nauc_map_at_10_max + value: 13.5001803982381 + - type: nauc_map_at_10_std + value: -2.628320244096416 + - type: nauc_map_at_1_diff1 + value: 47.528556920568896 + - type: nauc_map_at_1_max + value: 15.848152314768068 + - type: nauc_map_at_1_std + value: -3.8515029742454763 + - type: nauc_map_at_20_diff1 + value: 40.22452252482904 + - type: nauc_map_at_20_max + value: 13.501820277821633 + - type: nauc_map_at_20_std + value: -2.4849480670127835 + - type: nauc_map_at_3_diff1 + value: 41.68152420395297 + - type: nauc_map_at_3_max + value: 13.993359536648425 + - type: nauc_map_at_3_std + value: -4.120472655476033 + - type: nauc_map_at_5_diff1 + value: 40.72541498326932 + - type: nauc_map_at_5_max + value: 13.706855573979945 + - type: nauc_map_at_5_std + value: -3.168857069165899 + - type: nauc_mrr_at_1000_diff1 + value: 37.9361528126572 + - type: nauc_mrr_at_1000_max + value: 14.435169065764649 + - type: nauc_mrr_at_1000_std + value: -0.3672502634006242 + - type: nauc_mrr_at_100_diff1 + value: 37.94986436229442 + - type: nauc_mrr_at_100_max + value: 14.435994989813192 + - type: nauc_mrr_at_100_std + value: -0.37576385382293837 + - type: nauc_mrr_at_10_diff1 + value: 38.11900316449423 + - type: nauc_mrr_at_10_max + value: 14.472293540608746 + - type: nauc_mrr_at_10_std + value: -0.43716209085613345 + - type: nauc_mrr_at_1_diff1 + value: 44.21976115137286 + - type: nauc_mrr_at_1_max + value: 17.82290497090946 + - type: nauc_mrr_at_1_std + value: -1.547820761457578 + - type: nauc_mrr_at_20_diff1 + value: 38.024147471792524 + - type: nauc_mrr_at_20_max + value: 14.385378851779368 + - type: nauc_mrr_at_20_std + value: -0.47797312999005215 + - type: nauc_mrr_at_3_diff1 + value: 39.15186528374059 + - type: nauc_mrr_at_3_max + value: 15.21927102759239 + - type: nauc_mrr_at_3_std + value: -1.5215890424003806 + - type: nauc_mrr_at_5_diff1 + value: 38.45626599850357 + - type: nauc_mrr_at_5_max + value: 14.640408888284732 + - type: nauc_mrr_at_5_std + value: -0.7311075454359176 + - type: nauc_ndcg_at_1000_diff1 + value: 36.09833573033763 + - type: nauc_ndcg_at_1000_max + value: 13.245365815282575 + - type: nauc_ndcg_at_1000_std + value: 1.5761746506032988 + - type: nauc_ndcg_at_100_diff1 + value: 36.904025539005644 + - type: nauc_ndcg_at_100_max + value: 12.957957928970645 + - type: nauc_ndcg_at_100_std + value: 0.4532239536005292 + - type: nauc_ndcg_at_10_diff1 + value: 37.32497182133629 + - type: nauc_ndcg_at_10_max + value: 12.490853969491074 + - type: nauc_ndcg_at_10_std + value: -0.7416415504597471 + - type: nauc_ndcg_at_1_diff1 + value: 44.21976115137286 + - type: nauc_ndcg_at_1_max + value: 17.82290497090946 + - type: nauc_ndcg_at_1_std + value: -1.547820761457578 + - type: nauc_ndcg_at_20_diff1 + value: 37.28170904668032 + - type: nauc_ndcg_at_20_max + value: 12.268080858587759 + - type: nauc_ndcg_at_20_std + value: -0.7360183931126623 + - type: nauc_ndcg_at_3_diff1 + value: 39.02888999235542 + - type: nauc_ndcg_at_3_max + value: 13.901334459489329 + - type: nauc_ndcg_at_3_std + value: -2.7172751935367647 + - type: nauc_ndcg_at_5_diff1 + value: 38.02752207740974 + - type: nauc_ndcg_at_5_max + value: 13.02646174038431 + - type: nauc_ndcg_at_5_std + value: -1.609904028585218 + - type: nauc_precision_at_1000_diff1 + value: -6.66757757004073 + - type: nauc_precision_at_1000_max + value: 9.0023204523236 + - type: nauc_precision_at_1000_std + value: 23.5060357363243 + - type: nauc_precision_at_100_diff1 + value: 6.113195112414238 + - type: nauc_precision_at_100_max + value: 11.685619926894306 + - type: nauc_precision_at_100_std + value: 19.46517809799074 + - type: nauc_precision_at_10_diff1 + value: 20.39466712905433 + - type: nauc_precision_at_10_max + value: 11.42898255449916 + - type: nauc_precision_at_10_std + value: 9.716462445452729 + - type: nauc_precision_at_1_diff1 + value: 44.21976115137286 + - type: nauc_precision_at_1_max + value: 17.82290497090946 + - type: nauc_precision_at_1_std + value: -1.547820761457578 + - type: nauc_precision_at_20_diff1 + value: 16.658730057271427 + - type: nauc_precision_at_20_max + value: 11.1652114440581 + - type: nauc_precision_at_20_std + value: 11.300027272107469 + - type: nauc_precision_at_3_diff1 + value: 30.28030907617402 + - type: nauc_precision_at_3_max + value: 13.794055418422083 + - type: nauc_precision_at_3_std + value: 0.6048823642224063 + - type: nauc_precision_at_5_diff1 + value: 25.663334758638058 + - type: nauc_precision_at_5_max + value: 12.249908938864056 + - type: nauc_precision_at_5_std + value: 5.0045410071189425 + - type: nauc_recall_at_1000_diff1 + value: 21.220572448408245 + - type: nauc_recall_at_1000_max + value: 9.691420267810058 + - type: nauc_recall_at_1000_std + value: 12.85759827330056 + - type: nauc_recall_at_100_diff1 + value: 28.21527141094479 + - type: nauc_recall_at_100_max + value: 9.83831880254868 + - type: nauc_recall_at_100_std + value: 5.435149253402134 + - type: nauc_recall_at_10_diff1 + value: 30.716014201487262 + - type: nauc_recall_at_10_max + value: 8.051593782800182 + - type: nauc_recall_at_10_std + value: 0.4471610378184442 + - type: nauc_recall_at_1_diff1 + value: 47.528556920568896 + - type: nauc_recall_at_1_max + value: 15.848152314768068 + - type: nauc_recall_at_1_std + value: -3.8515029742454763 + - type: nauc_recall_at_20_diff1 + value: 29.800603042147905 + - type: nauc_recall_at_20_max + value: 7.042808403898776 + - type: nauc_recall_at_20_std + value: 0.8179034283502986 + - type: nauc_recall_at_3_diff1 + value: 36.05311584515151 + - type: nauc_recall_at_3_max + value: 11.03138015792514 + - type: nauc_recall_at_3_std + value: -4.298332543889119 + - type: nauc_recall_at_5_diff1 + value: 33.34542113435848 + - type: nauc_recall_at_5_max + value: 9.391429367517976 + - type: nauc_recall_at_5_std + value: -1.5174868347878459 + - type: ndcg_at_1 + value: 20.382 + - type: ndcg_at_10 + value: 25.734 + - type: ndcg_at_100 + value: 29.952 + - type: ndcg_at_1000 + value: 32.618 + - type: ndcg_at_20 + value: 27.181 + - type: ndcg_at_3 + value: 22.445999999999998 + - type: ndcg_at_5 + value: 24.162 + - type: precision_at_1 + value: 20.382 + - type: precision_at_10 + value: 4.662 + - type: precision_at_100 + value: 0.8580000000000001 + - type: precision_at_1000 + value: 0.133 + - type: precision_at_20 + value: 2.828 + - type: precision_at_3 + value: 10.446 + - type: precision_at_5 + value: 7.682 + - type: recall_at_1 + value: 16.601 + - type: recall_at_10 + value: 32.882 + - type: recall_at_100 + value: 51.273 + - type: recall_at_1000 + value: 69.33200000000001 + - type: recall_at_20 + value: 38.22 + - type: recall_at_3 + value: 23.54 + - type: recall_at_5 + value: 28.054000000000002 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackGamingRetrieval (default) + revision: 4885aa143210c98657558c04aaf3dc47cfb54340 + split: test + type: mteb/cqadupstack-gaming + metrics: + - type: main_score + value: 39.235 + - type: map_at_1 + value: 25.386999999999997 + - type: map_at_10 + value: 34.183 + - type: map_at_100 + value: 35.198 + - type: map_at_1000 + value: 35.292 + - type: map_at_20 + value: 34.756 + - type: map_at_3 + value: 31.466 + - type: map_at_5 + value: 33.037 + - type: mrr_at_1 + value: 29.404388714733543 + - type: mrr_at_10 + value: 37.51880877742944 + - type: mrr_at_100 + value: 38.30457109532953 + - type: mrr_at_1000 + value: 38.3645245292866 + - type: mrr_at_20 + value: 37.94776237222878 + - type: mrr_at_3 + value: 35.15151515151513 + - type: mrr_at_5 + value: 36.530825496342715 + - type: nauc_map_at_1000_diff1 + value: 41.249973220934464 + - type: nauc_map_at_1000_max + value: 23.416302755877073 + - type: nauc_map_at_1000_std + value: -10.207899212437999 + - type: nauc_map_at_100_diff1 + value: 41.24390045906369 + - type: nauc_map_at_100_max + value: 23.393682611799267 + - type: nauc_map_at_100_std + value: -10.254556576082482 + - type: nauc_map_at_10_diff1 + value: 41.382354597936995 + - type: nauc_map_at_10_max + value: 23.176782265492363 + - type: nauc_map_at_10_std + value: -10.849718292221906 + - type: nauc_map_at_1_diff1 + value: 45.39686265513208 + - type: nauc_map_at_1_max + value: 19.620871905273706 + - type: nauc_map_at_1_std + value: -12.904987428165654 + - type: nauc_map_at_20_diff1 + value: 41.27244082919643 + - type: nauc_map_at_20_max + value: 23.302684773349597 + - type: nauc_map_at_20_std + value: -10.441842806985154 + - type: nauc_map_at_3_diff1 + value: 41.8919220244127 + - type: nauc_map_at_3_max + value: 22.254220793423723 + - type: nauc_map_at_3_std + value: -12.130298439753705 + - type: nauc_map_at_5_diff1 + value: 41.58025783631085 + - type: nauc_map_at_5_max + value: 22.90826213564573 + - type: nauc_map_at_5_std + value: -11.165811549758352 + - type: nauc_mrr_at_1000_diff1 + value: 40.53152598499822 + - type: nauc_mrr_at_1000_max + value: 25.11227665851315 + - type: nauc_mrr_at_1000_std + value: -8.08741271282522 + - type: nauc_mrr_at_100_diff1 + value: 40.51963005358264 + - type: nauc_mrr_at_100_max + value: 25.120293035347625 + - type: nauc_mrr_at_100_std + value: -8.08477757772673 + - type: nauc_mrr_at_10_diff1 + value: 40.630254919734845 + - type: nauc_mrr_at_10_max + value: 25.192263018985 + - type: nauc_mrr_at_10_std + value: -8.343786686430308 + - type: nauc_mrr_at_1_diff1 + value: 45.24802769641752 + - type: nauc_mrr_at_1_max + value: 22.81400229887994 + - type: nauc_mrr_at_1_std + value: -11.030374885452746 + - type: nauc_mrr_at_20_diff1 + value: 40.527874579465404 + - type: nauc_mrr_at_20_max + value: 25.09785309228408 + - type: nauc_mrr_at_20_std + value: -8.178961300984005 + - type: nauc_mrr_at_3_diff1 + value: 40.9982110047705 + - type: nauc_mrr_at_3_max + value: 24.89415486978485 + - type: nauc_mrr_at_3_std + value: -9.326777261347539 + - type: nauc_mrr_at_5_diff1 + value: 40.80630420274428 + - type: nauc_mrr_at_5_max + value: 25.27575084878062 + - type: nauc_mrr_at_5_std + value: -8.546736722404525 + - type: nauc_ndcg_at_1000_diff1 + value: 39.53378645935715 + - type: nauc_ndcg_at_1000_max + value: 25.526492849521226 + - type: nauc_ndcg_at_1000_std + value: -6.007063152931765 + - type: nauc_ndcg_at_100_diff1 + value: 39.0880907026097 + - type: nauc_ndcg_at_100_max + value: 25.27434977919565 + - type: nauc_ndcg_at_100_std + value: -6.494059729717049 + - type: nauc_ndcg_at_10_diff1 + value: 39.75643189392527 + - type: nauc_ndcg_at_10_max + value: 24.79335502116443 + - type: nauc_ndcg_at_10_std + value: -8.786781322519788 + - type: nauc_ndcg_at_1_diff1 + value: 45.24802769641752 + - type: nauc_ndcg_at_1_max + value: 22.81400229887994 + - type: nauc_ndcg_at_1_std + value: -11.030374885452746 + - type: nauc_ndcg_at_20_diff1 + value: 39.38115636990762 + - type: nauc_ndcg_at_20_max + value: 24.830948061340973 + - type: nauc_ndcg_at_20_std + value: -7.74514857483731 + - type: nauc_ndcg_at_3_diff1 + value: 40.597424968913295 + - type: nauc_ndcg_at_3_max + value: 23.83761797284813 + - type: nauc_ndcg_at_3_std + value: -10.826014984199753 + - type: nauc_ndcg_at_5_diff1 + value: 40.160243884240955 + - type: nauc_ndcg_at_5_max + value: 24.641005184802403 + - type: nauc_ndcg_at_5_std + value: -9.394573143721122 + - type: nauc_precision_at_1000_diff1 + value: -0.26775483855404 + - type: nauc_precision_at_1000_max + value: 23.052779599626216 + - type: nauc_precision_at_1000_std + value: 24.978867586645737 + - type: nauc_precision_at_100_diff1 + value: 9.73599417323489 + - type: nauc_precision_at_100_max + value: 26.664612833573067 + - type: nauc_precision_at_100_std + value: 15.747547424892522 + - type: nauc_precision_at_10_diff1 + value: 25.384143998683495 + - type: nauc_precision_at_10_max + value: 28.77515164969203 + - type: nauc_precision_at_10_std + value: 1.334799782027906 + - type: nauc_precision_at_1_diff1 + value: 45.24802769641752 + - type: nauc_precision_at_1_max + value: 22.81400229887994 + - type: nauc_precision_at_1_std + value: -11.030374885452746 + - type: nauc_precision_at_20_diff1 + value: 20.21252517032333 + - type: nauc_precision_at_20_max + value: 28.092242647209847 + - type: nauc_precision_at_20_std + value: 7.13292725544981 + - type: nauc_precision_at_3_diff1 + value: 33.31087126292084 + - type: nauc_precision_at_3_max + value: 28.144729235595268 + - type: nauc_precision_at_3_std + value: -6.680273865904818 + - type: nauc_precision_at_5_diff1 + value: 29.65876394876068 + - type: nauc_precision_at_5_max + value: 29.35126830830009 + - type: nauc_precision_at_5_std + value: -1.6373943088766274 + - type: nauc_recall_at_1000_diff1 + value: 28.93648565815677 + - type: nauc_recall_at_1000_max + value: 35.83681303333163 + - type: nauc_recall_at_1000_std + value: 33.065249002817446 + - type: nauc_recall_at_100_diff1 + value: 27.743019102171594 + - type: nauc_recall_at_100_max + value: 28.027951033595023 + - type: nauc_recall_at_100_std + value: 9.499502949546343 + - type: nauc_recall_at_10_diff1 + value: 33.975592980890205 + - type: nauc_recall_at_10_max + value: 25.654266106207007 + - type: nauc_recall_at_10_std + value: -4.889087003341999 + - type: nauc_recall_at_1_diff1 + value: 45.39686265513208 + - type: nauc_recall_at_1_max + value: 19.620871905273706 + - type: nauc_recall_at_1_std + value: -12.904987428165654 + - type: nauc_recall_at_20_diff1 + value: 32.428638046562156 + - type: nauc_recall_at_20_max + value: 25.811049662670854 + - type: nauc_recall_at_20_std + value: -1.084167664066214 + - type: nauc_recall_at_3_diff1 + value: 36.80239523147669 + - type: nauc_recall_at_3_max + value: 23.70115293826517 + - type: nauc_recall_at_3_std + value: -10.179865917816631 + - type: nauc_recall_at_5_diff1 + value: 35.481273082880385 + - type: nauc_recall_at_5_max + value: 25.22699895557444 + - type: nauc_recall_at_5_std + value: -6.928154160954265 + - type: ndcg_at_1 + value: 29.404000000000003 + - type: ndcg_at_10 + value: 39.235 + - type: ndcg_at_100 + value: 44.072 + - type: ndcg_at_1000 + value: 46.272999999999996 + - type: ndcg_at_20 + value: 40.983000000000004 + - type: ndcg_at_3 + value: 34.292 + - type: ndcg_at_5 + value: 36.735 + - type: precision_at_1 + value: 29.404000000000003 + - type: precision_at_10 + value: 6.539000000000001 + - type: precision_at_100 + value: 0.984 + - type: precision_at_1000 + value: 0.125 + - type: precision_at_20 + value: 3.752 + - type: precision_at_3 + value: 15.423 + - type: precision_at_5 + value: 10.984 + - type: recall_at_1 + value: 25.386999999999997 + - type: recall_at_10 + value: 51.256 + - type: recall_at_100 + value: 73.53699999999999 + - type: recall_at_1000 + value: 89.522 + - type: recall_at_20 + value: 57.687 + - type: recall_at_3 + value: 37.830999999999996 + - type: recall_at_5 + value: 43.811 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackGisRetrieval (default) + revision: 5003b3064772da1887988e05400cf3806fe491f2 + split: test + type: mteb/cqadupstack-gis + metrics: + - type: main_score + value: 19.197 + - type: map_at_1 + value: 10.832 + - type: map_at_10 + value: 16.154 + - type: map_at_100 + value: 16.863 + - type: map_at_1000 + value: 16.979 + - type: map_at_20 + value: 16.494 + - type: map_at_3 + value: 14.654 + - type: map_at_5 + value: 15.634 + - type: mrr_at_1 + value: 11.751412429378531 + - type: mrr_at_10 + value: 17.286476549188407 + - type: mrr_at_100 + value: 18.019080515365157 + - type: mrr_at_1000 + value: 18.122220740371624 + - type: mrr_at_20 + value: 17.643986643881693 + - type: mrr_at_3 + value: 15.70621468926553 + - type: mrr_at_5 + value: 16.774011299435024 + - type: nauc_map_at_1000_diff1 + value: 37.927063185916786 + - type: nauc_map_at_1000_max + value: 14.15651072891371 + - type: nauc_map_at_1000_std + value: -8.124962552251457 + - type: nauc_map_at_100_diff1 + value: 37.93525025821844 + - type: nauc_map_at_100_max + value: 14.131523699537288 + - type: nauc_map_at_100_std + value: -8.170583771371396 + - type: nauc_map_at_10_diff1 + value: 38.42813636094302 + - type: nauc_map_at_10_max + value: 14.282120499977891 + - type: nauc_map_at_10_std + value: -8.577031812934745 + - type: nauc_map_at_1_diff1 + value: 51.66692699481996 + - type: nauc_map_at_1_max + value: 17.664646674047123 + - type: nauc_map_at_1_std + value: -11.782621031162968 + - type: nauc_map_at_20_diff1 + value: 38.17853788871855 + - type: nauc_map_at_20_max + value: 14.256213676574742 + - type: nauc_map_at_20_std + value: -8.310926163301415 + - type: nauc_map_at_3_diff1 + value: 40.16070984262913 + - type: nauc_map_at_3_max + value: 14.268693118841725 + - type: nauc_map_at_3_std + value: -9.133251481752447 + - type: nauc_map_at_5_diff1 + value: 38.83714248320578 + - type: nauc_map_at_5_max + value: 14.547528919229999 + - type: nauc_map_at_5_std + value: -8.916871955060776 + - type: nauc_mrr_at_1000_diff1 + value: 36.5899689047331 + - type: nauc_mrr_at_1000_max + value: 15.113884206534985 + - type: nauc_mrr_at_1000_std + value: -7.170934224974719 + - type: nauc_mrr_at_100_diff1 + value: 36.58290352969189 + - type: nauc_mrr_at_100_max + value: 15.10461015425463 + - type: nauc_mrr_at_100_std + value: -7.193153133255972 + - type: nauc_mrr_at_10_diff1 + value: 36.886787941126755 + - type: nauc_mrr_at_10_max + value: 15.127743773603711 + - type: nauc_mrr_at_10_std + value: -7.450354111586159 + - type: nauc_mrr_at_1_diff1 + value: 50.4303551964735 + - type: nauc_mrr_at_1_max + value: 18.974353633454818 + - type: nauc_mrr_at_1_std + value: -10.667048661688531 + - type: nauc_mrr_at_20_diff1 + value: 36.748056497939466 + - type: nauc_mrr_at_20_max + value: 15.240859680475241 + - type: nauc_mrr_at_20_std + value: -7.288016407850428 + - type: nauc_mrr_at_3_diff1 + value: 38.37428302171742 + - type: nauc_mrr_at_3_max + value: 14.8093219575286 + - type: nauc_mrr_at_3_std + value: -7.809230035161326 + - type: nauc_mrr_at_5_diff1 + value: 37.2144623683964 + - type: nauc_mrr_at_5_max + value: 15.28601324524152 + - type: nauc_mrr_at_5_std + value: -7.7340060832485 + - type: nauc_ndcg_at_1000_diff1 + value: 32.12453348510246 + - type: nauc_ndcg_at_1000_max + value: 13.157455004954915 + - type: nauc_ndcg_at_1000_std + value: -4.92622356811411 + - type: nauc_ndcg_at_100_diff1 + value: 32.06154877919635 + - type: nauc_ndcg_at_100_max + value: 12.373862596941047 + - type: nauc_ndcg_at_100_std + value: -5.679273924705311 + - type: nauc_ndcg_at_10_diff1 + value: 34.0105889334877 + - type: nauc_ndcg_at_10_max + value: 13.45850179368671 + - type: nauc_ndcg_at_10_std + value: -7.129474197823981 + - type: nauc_ndcg_at_1_diff1 + value: 50.4303551964735 + - type: nauc_ndcg_at_1_max + value: 18.974353633454818 + - type: nauc_ndcg_at_1_std + value: -10.667048661688531 + - type: nauc_ndcg_at_20_diff1 + value: 33.17001669466592 + - type: nauc_ndcg_at_20_max + value: 13.32565385671001 + - type: nauc_ndcg_at_20_std + value: -6.284897809311489 + - type: nauc_ndcg_at_3_diff1 + value: 36.583009335894786 + - type: nauc_ndcg_at_3_max + value: 13.3100798018976 + - type: nauc_ndcg_at_3_std + value: -8.166653842277874 + - type: nauc_ndcg_at_5_diff1 + value: 34.663883470713714 + - type: nauc_ndcg_at_5_max + value: 13.925348847790179 + - type: nauc_ndcg_at_5_std + value: -7.8134139319246705 + - type: nauc_precision_at_1000_diff1 + value: 3.267820129824429 + - type: nauc_precision_at_1000_max + value: 13.475739290072998 + - type: nauc_precision_at_1000_std + value: 9.817456700342868 + - type: nauc_precision_at_100_diff1 + value: 14.543473928222502 + - type: nauc_precision_at_100_max + value: 9.536842145225432 + - type: nauc_precision_at_100_std + value: 2.367980716410962 + - type: nauc_precision_at_10_diff1 + value: 22.83690357863953 + - type: nauc_precision_at_10_max + value: 12.377338528340081 + - type: nauc_precision_at_10_std + value: -2.7413618512966442 + - type: nauc_precision_at_1_diff1 + value: 50.4303551964735 + - type: nauc_precision_at_1_max + value: 18.974353633454818 + - type: nauc_precision_at_1_std + value: -10.667048661688531 + - type: nauc_precision_at_20_diff1 + value: 20.379974384537427 + - type: nauc_precision_at_20_max + value: 12.277432490519853 + - type: nauc_precision_at_20_std + value: -0.023357415290595228 + - type: nauc_precision_at_3_diff1 + value: 28.00128059605776 + - type: nauc_precision_at_3_max + value: 12.115949162806704 + - type: nauc_precision_at_3_std + value: -5.111345494119332 + - type: nauc_precision_at_5_diff1 + value: 23.931333166517064 + - type: nauc_precision_at_5_max + value: 13.460490076263444 + - type: nauc_precision_at_5_std + value: -4.566369591299022 + - type: nauc_recall_at_1000_diff1 + value: 13.901980638817474 + - type: nauc_recall_at_1000_max + value: 8.169301488452522 + - type: nauc_recall_at_1000_std + value: 6.977530327014011 + - type: nauc_recall_at_100_diff1 + value: 18.54699849728289 + - type: nauc_recall_at_100_max + value: 5.40051681338299 + - type: nauc_recall_at_100_std + value: -0.2998165893044503 + - type: nauc_recall_at_10_diff1 + value: 25.158691029447162 + - type: nauc_recall_at_10_max + value: 10.698096715728344 + - type: nauc_recall_at_10_std + value: -4.90677955177619 + - type: nauc_recall_at_1_diff1 + value: 51.66692699481996 + - type: nauc_recall_at_1_max + value: 17.664646674047123 + - type: nauc_recall_at_1_std + value: -11.782621031162968 + - type: nauc_recall_at_20_diff1 + value: 22.315869507893193 + - type: nauc_recall_at_20_max + value: 9.799239845339486 + - type: nauc_recall_at_20_std + value: -2.255295176195769 + - type: nauc_recall_at_3_diff1 + value: 30.21846457670379 + - type: nauc_recall_at_3_max + value: 10.958491456074727 + - type: nauc_recall_at_3_std + value: -6.746808382770713 + - type: nauc_recall_at_5_diff1 + value: 26.24302256225738 + - type: nauc_recall_at_5_max + value: 11.682268465161725 + - type: nauc_recall_at_5_std + value: -6.292007648799524 + - type: ndcg_at_1 + value: 11.751000000000001 + - type: ndcg_at_10 + value: 19.197 + - type: ndcg_at_100 + value: 23.159 + - type: ndcg_at_1000 + value: 26.453 + - type: ndcg_at_20 + value: 20.448 + - type: ndcg_at_3 + value: 16.186 + - type: ndcg_at_5 + value: 17.936 + - type: precision_at_1 + value: 11.751000000000001 + - type: precision_at_10 + value: 3.1189999999999998 + - type: precision_at_100 + value: 0.54 + - type: precision_at_1000 + value: 0.086 + - type: precision_at_20 + value: 1.859 + - type: precision_at_3 + value: 7.194000000000001 + - type: precision_at_5 + value: 5.311 + - type: recall_at_1 + value: 10.832 + - type: recall_at_10 + value: 27.472 + - type: recall_at_100 + value: 46.471000000000004 + - type: recall_at_1000 + value: 71.91199999999999 + - type: recall_at_20 + value: 32.213 + - type: recall_at_3 + value: 19.417 + - type: recall_at_5 + value: 23.577 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackMathematicaRetrieval (default) + revision: 90fceea13679c63fe563ded68f3b6f06e50061de + split: test + type: mteb/cqadupstack-mathematica + metrics: + - type: main_score + value: 12.145 + - type: map_at_1 + value: 6.019 + - type: map_at_10 + value: 9.584 + - type: map_at_100 + value: 10.433 + - type: map_at_1000 + value: 10.562000000000001 + - type: map_at_20 + value: 10.024 + - type: map_at_3 + value: 8.351 + - type: map_at_5 + value: 9.005 + - type: mrr_at_1 + value: 7.213930348258707 + - type: mrr_at_10 + value: 11.619827450051332 + - type: mrr_at_100 + value: 12.469229814971346 + - type: mrr_at_1000 + value: 12.577286932589695 + - type: mrr_at_20 + value: 12.072514356821353 + - type: mrr_at_3 + value: 10.157545605306801 + - type: mrr_at_5 + value: 10.89759535655058 + - type: nauc_map_at_1000_diff1 + value: 18.60219400887139 + - type: nauc_map_at_1000_max + value: 6.951583595979727 + - type: nauc_map_at_1000_std + value: -0.36466683994108184 + - type: nauc_map_at_100_diff1 + value: 18.660733139389524 + - type: nauc_map_at_100_max + value: 6.903072765131549 + - type: nauc_map_at_100_std + value: -0.48390217802549257 + - type: nauc_map_at_10_diff1 + value: 18.573179595835647 + - type: nauc_map_at_10_max + value: 6.992666771720911 + - type: nauc_map_at_10_std + value: -0.8874423543023089 + - type: nauc_map_at_1_diff1 + value: 33.90106432523568 + - type: nauc_map_at_1_max + value: 9.289205840089235 + - type: nauc_map_at_1_std + value: 2.1852128418717705 + - type: nauc_map_at_20_diff1 + value: 18.334656889783485 + - type: nauc_map_at_20_max + value: 6.931684308001437 + - type: nauc_map_at_20_std + value: -0.7124186564380448 + - type: nauc_map_at_3_diff1 + value: 20.32895393313974 + - type: nauc_map_at_3_max + value: 5.887419026571198 + - type: nauc_map_at_3_std + value: -0.015273865884840596 + - type: nauc_map_at_5_diff1 + value: 19.15574225963634 + - type: nauc_map_at_5_max + value: 6.175933890525402 + - type: nauc_map_at_5_std + value: -1.468261999387673 + - type: nauc_mrr_at_1000_diff1 + value: 18.0560339880594 + - type: nauc_mrr_at_1000_max + value: 8.653214727915024 + - type: nauc_mrr_at_1000_std + value: 1.6650523107666824 + - type: nauc_mrr_at_100_diff1 + value: 18.067266124955946 + - type: nauc_mrr_at_100_max + value: 8.645444544074266 + - type: nauc_mrr_at_100_std + value: 1.605397143432772 + - type: nauc_mrr_at_10_diff1 + value: 18.227604303918422 + - type: nauc_mrr_at_10_max + value: 8.980990643614946 + - type: nauc_mrr_at_10_std + value: 1.625956129526598 + - type: nauc_mrr_at_1_diff1 + value: 33.145174271418576 + - type: nauc_mrr_at_1_max + value: 10.674348159869123 + - type: nauc_mrr_at_1_std + value: 2.5718912675260843 + - type: nauc_mrr_at_20_diff1 + value: 17.85361170315467 + - type: nauc_mrr_at_20_max + value: 8.689966423383293 + - type: nauc_mrr_at_20_std + value: 1.4845343622374683 + - type: nauc_mrr_at_3_diff1 + value: 19.72873972100882 + - type: nauc_mrr_at_3_max + value: 7.818757201820606 + - type: nauc_mrr_at_3_std + value: 2.317801166782217 + - type: nauc_mrr_at_5_diff1 + value: 18.70515159747826 + - type: nauc_mrr_at_5_max + value: 7.8553636278171055 + - type: nauc_mrr_at_5_std + value: 0.8593300223901442 + - type: nauc_ndcg_at_1000_diff1 + value: 14.777764985527059 + - type: nauc_ndcg_at_1000_max + value: 8.001133085293265 + - type: nauc_ndcg_at_1000_std + value: 2.715094827482056 + - type: nauc_ndcg_at_100_diff1 + value: 15.873494520058037 + - type: nauc_ndcg_at_100_max + value: 7.5190091115119 + - type: nauc_ndcg_at_100_std + value: 0.7430533500967327 + - type: nauc_ndcg_at_10_diff1 + value: 14.950829327092022 + - type: nauc_ndcg_at_10_max + value: 7.999425322307154 + - type: nauc_ndcg_at_10_std + value: -0.5911692617165382 + - type: nauc_ndcg_at_1_diff1 + value: 33.145174271418576 + - type: nauc_ndcg_at_1_max + value: 10.674348159869123 + - type: nauc_ndcg_at_1_std + value: 2.5718912675260843 + - type: nauc_ndcg_at_20_diff1 + value: 14.28695753335748 + - type: nauc_ndcg_at_20_max + value: 7.460341211112809 + - type: nauc_ndcg_at_20_std + value: -0.2734671370134216 + - type: nauc_ndcg_at_3_diff1 + value: 17.243393543205006 + - type: nauc_ndcg_at_3_max + value: 6.003682896861271 + - type: nauc_ndcg_at_3_std + value: 0.3923628664952013 + - type: nauc_ndcg_at_5_diff1 + value: 15.841455870049076 + - type: nauc_ndcg_at_5_max + value: 6.163583363661528 + - type: nauc_ndcg_at_5_std + value: -1.9411356710983478 + - type: nauc_precision_at_1000_diff1 + value: -3.399817676017686 + - type: nauc_precision_at_1000_max + value: 5.575723322824422 + - type: nauc_precision_at_1000_std + value: 5.63779109914318 + - type: nauc_precision_at_100_diff1 + value: 6.1555220193892435 + - type: nauc_precision_at_100_max + value: 6.7977343501791045 + - type: nauc_precision_at_100_std + value: 2.026960062764128 + - type: nauc_precision_at_10_diff1 + value: 5.864713737249161 + - type: nauc_precision_at_10_max + value: 10.987539143688663 + - type: nauc_precision_at_10_std + value: -0.12419185225065871 + - type: nauc_precision_at_1_diff1 + value: 33.145174271418576 + - type: nauc_precision_at_1_max + value: 10.674348159869123 + - type: nauc_precision_at_1_std + value: 2.5718912675260843 + - type: nauc_precision_at_20_diff1 + value: 4.994637980783556 + - type: nauc_precision_at_20_max + value: 7.522690866727933 + - type: nauc_precision_at_20_std + value: 0.027674551460471312 + - type: nauc_precision_at_3_diff1 + value: 8.451342681964578 + - type: nauc_precision_at_3_max + value: 5.343253356927528 + - type: nauc_precision_at_3_std + value: 1.6495845441147832 + - type: nauc_precision_at_5_diff1 + value: 6.193033041556517 + - type: nauc_precision_at_5_max + value: 5.77635145338238 + - type: nauc_precision_at_5_std + value: -3.421797113444559 + - type: nauc_recall_at_1000_diff1 + value: 7.437110169863727 + - type: nauc_recall_at_1000_max + value: 9.607314782406986 + - type: nauc_recall_at_1000_std + value: 13.320498460741362 + - type: nauc_recall_at_100_diff1 + value: 13.309966057961834 + - type: nauc_recall_at_100_max + value: 7.748170239579637 + - type: nauc_recall_at_100_std + value: 2.6798857378517864 + - type: nauc_recall_at_10_diff1 + value: 8.674278695378167 + - type: nauc_recall_at_10_max + value: 8.969918415623756 + - type: nauc_recall_at_10_std + value: -1.4597400700986853 + - type: nauc_recall_at_1_diff1 + value: 33.90106432523568 + - type: nauc_recall_at_1_max + value: 9.289205840089235 + - type: nauc_recall_at_1_std + value: 2.1852128418717705 + - type: nauc_recall_at_20_diff1 + value: 7.663555921211413 + - type: nauc_recall_at_20_max + value: 7.420494129425241 + - type: nauc_recall_at_20_std + value: -0.39971980929980877 + - type: nauc_recall_at_3_diff1 + value: 10.784631081908223 + - type: nauc_recall_at_3_max + value: 3.815625872455824 + - type: nauc_recall_at_3_std + value: -1.1614434404018152 + - type: nauc_recall_at_5_diff1 + value: 9.60638979119831 + - type: nauc_recall_at_5_max + value: 5.1710882220553405 + - type: nauc_recall_at_5_std + value: -4.572280393094789 + - type: ndcg_at_1 + value: 7.2139999999999995 + - type: ndcg_at_10 + value: 12.145 + - type: ndcg_at_100 + value: 16.672 + - type: ndcg_at_1000 + value: 20.342 + - type: ndcg_at_20 + value: 13.745 + - type: ndcg_at_3 + value: 9.607000000000001 + - type: ndcg_at_5 + value: 10.712000000000002 + - type: precision_at_1 + value: 7.2139999999999995 + - type: precision_at_10 + value: 2.338 + - type: precision_at_100 + value: 0.5459999999999999 + - type: precision_at_1000 + value: 0.099 + - type: precision_at_20 + value: 1.6039999999999999 + - type: precision_at_3 + value: 4.726 + - type: precision_at_5 + value: 3.5319999999999996 + - type: recall_at_1 + value: 6.019 + - type: recall_at_10 + value: 18.102999999999998 + - type: recall_at_100 + value: 38.482 + - type: recall_at_1000 + value: 65.436 + - type: recall_at_20 + value: 23.952 + - type: recall_at_3 + value: 11.178 + - type: recall_at_5 + value: 13.877 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackPhysicsRetrieval (default) + revision: 79531abbd1fb92d06c6d6315a0cbbbf5bb247ea4 + split: test + type: mteb/cqadupstack-physics + metrics: + - type: main_score + value: 26.667999999999996 + - type: map_at_1 + value: 16.822 + - type: map_at_10 + value: 22.476 + - type: map_at_100 + value: 23.69 + - type: map_at_1000 + value: 23.827 + - type: map_at_20 + value: 23.084 + - type: map_at_3 + value: 20.441000000000003 + - type: map_at_5 + value: 21.512 + - type: mrr_at_1 + value: 20.78922040423484 + - type: mrr_at_10 + value: 26.67445804115679 + - type: mrr_at_100 + value: 27.67534998291947 + - type: mrr_at_1000 + value: 27.752906060167692 + - type: mrr_at_20 + value: 27.19875968774574 + - type: mrr_at_3 + value: 24.4947064485082 + - type: mrr_at_5 + value: 25.630413859480278 + - type: nauc_map_at_1000_diff1 + value: 40.40492447320535 + - type: nauc_map_at_1000_max + value: 28.548119831633194 + - type: nauc_map_at_1000_std + value: -0.22424233207141148 + - type: nauc_map_at_100_diff1 + value: 40.39875847865982 + - type: nauc_map_at_100_max + value: 28.500575725413096 + - type: nauc_map_at_100_std + value: -0.2779979908842256 + - type: nauc_map_at_10_diff1 + value: 40.942304749094085 + - type: nauc_map_at_10_max + value: 28.429772938475008 + - type: nauc_map_at_10_std + value: -0.8049874864329988 + - type: nauc_map_at_1_diff1 + value: 47.17822553627135 + - type: nauc_map_at_1_max + value: 31.206514215995206 + - type: nauc_map_at_1_std + value: -1.8984121963184788 + - type: nauc_map_at_20_diff1 + value: 40.4346381000311 + - type: nauc_map_at_20_max + value: 28.458128761837536 + - type: nauc_map_at_20_std + value: -0.7321703207226834 + - type: nauc_map_at_3_diff1 + value: 42.2424427066743 + - type: nauc_map_at_3_max + value: 28.16537428952111 + - type: nauc_map_at_3_std + value: -2.298671243793284 + - type: nauc_map_at_5_diff1 + value: 41.32690925538059 + - type: nauc_map_at_5_max + value: 28.53162210264393 + - type: nauc_map_at_5_std + value: -1.1738320079845177 + - type: nauc_mrr_at_1000_diff1 + value: 37.69693278594645 + - type: nauc_mrr_at_1000_max + value: 29.49690742209793 + - type: nauc_mrr_at_1000_std + value: 3.1815473802020544 + - type: nauc_mrr_at_100_diff1 + value: 37.65946389835227 + - type: nauc_mrr_at_100_max + value: 29.479438074437127 + - type: nauc_mrr_at_100_std + value: 3.166552364873761 + - type: nauc_mrr_at_10_diff1 + value: 38.06473613801605 + - type: nauc_mrr_at_10_max + value: 29.79312016758447 + - type: nauc_mrr_at_10_std + value: 3.111988711521923 + - type: nauc_mrr_at_1_diff1 + value: 43.69553072839024 + - type: nauc_mrr_at_1_max + value: 32.142344513289025 + - type: nauc_mrr_at_1_std + value: 2.696048057380709 + - type: nauc_mrr_at_20_diff1 + value: 37.626141249327574 + - type: nauc_mrr_at_20_max + value: 29.559923833552347 + - type: nauc_mrr_at_20_std + value: 2.9860721770618697 + - type: nauc_mrr_at_3_diff1 + value: 39.324715416924974 + - type: nauc_mrr_at_3_max + value: 29.651196356282618 + - type: nauc_mrr_at_3_std + value: 1.9583884507428824 + - type: nauc_mrr_at_5_diff1 + value: 38.36691352781637 + - type: nauc_mrr_at_5_max + value: 29.939763561026002 + - type: nauc_mrr_at_5_std + value: 2.7317703526814214 + - type: nauc_ndcg_at_1000_diff1 + value: 36.523136783112406 + - type: nauc_ndcg_at_1000_max + value: 28.684387654497584 + - type: nauc_ndcg_at_1000_std + value: 4.732051883634089 + - type: nauc_ndcg_at_100_diff1 + value: 36.16154861613736 + - type: nauc_ndcg_at_100_max + value: 27.921202679602143 + - type: nauc_ndcg_at_100_std + value: 3.560040019944456 + - type: nauc_ndcg_at_10_diff1 + value: 37.774474422977896 + - type: nauc_ndcg_at_10_max + value: 27.68147817987237 + - type: nauc_ndcg_at_10_std + value: 0.8327502237036594 + - type: nauc_ndcg_at_1_diff1 + value: 43.69553072839024 + - type: nauc_ndcg_at_1_max + value: 32.142344513289025 + - type: nauc_ndcg_at_1_std + value: 2.696048057380709 + - type: nauc_ndcg_at_20_diff1 + value: 36.163233644690266 + - type: nauc_ndcg_at_20_max + value: 27.4164968525345 + - type: nauc_ndcg_at_20_std + value: 0.8376631121502218 + - type: nauc_ndcg_at_3_diff1 + value: 39.707715661307105 + - type: nauc_ndcg_at_3_max + value: 28.324727845444997 + - type: nauc_ndcg_at_3_std + value: -0.7238153399588456 + - type: nauc_ndcg_at_5_diff1 + value: 38.42323115018405 + - type: nauc_ndcg_at_5_max + value: 28.520234702176587 + - type: nauc_ndcg_at_5_std + value: 0.4337143091381524 + - type: nauc_precision_at_1000_diff1 + value: -1.7237517846851018 + - type: nauc_precision_at_1000_max + value: 16.20499296488572 + - type: nauc_precision_at_1000_std + value: 20.16360817424688 + - type: nauc_precision_at_100_diff1 + value: 7.455105305668386 + - type: nauc_precision_at_100_max + value: 23.35672119353681 + - type: nauc_precision_at_100_std + value: 18.66911905196039 + - type: nauc_precision_at_10_diff1 + value: 23.28265657395181 + - type: nauc_precision_at_10_max + value: 27.533659469131948 + - type: nauc_precision_at_10_std + value: 9.661356716727099 + - type: nauc_precision_at_1_diff1 + value: 43.69553072839024 + - type: nauc_precision_at_1_max + value: 32.142344513289025 + - type: nauc_precision_at_1_std + value: 2.696048057380709 + - type: nauc_precision_at_20_diff1 + value: 15.588844976640317 + - type: nauc_precision_at_20_max + value: 24.89373446940838 + - type: nauc_precision_at_20_std + value: 9.462736793529547 + - type: nauc_precision_at_3_diff1 + value: 31.24543977571387 + - type: nauc_precision_at_3_max + value: 27.88457380895888 + - type: nauc_precision_at_3_std + value: 3.0400582769598334 + - type: nauc_precision_at_5_diff1 + value: 27.621476771588156 + - type: nauc_precision_at_5_max + value: 29.344696084898647 + - type: nauc_precision_at_5_std + value: 6.279675749763937 + - type: nauc_recall_at_1000_diff1 + value: 20.19996493542523 + - type: nauc_recall_at_1000_max + value: 24.65244498292903 + - type: nauc_recall_at_1000_std + value: 35.312310075738125 + - type: nauc_recall_at_100_diff1 + value: 22.904431187357847 + - type: nauc_recall_at_100_max + value: 21.00955732817796 + - type: nauc_recall_at_100_std + value: 13.938151070174573 + - type: nauc_recall_at_10_diff1 + value: 30.03923096618402 + - type: nauc_recall_at_10_max + value: 22.353534397229048 + - type: nauc_recall_at_10_std + value: 1.2207088824681231 + - type: nauc_recall_at_1_diff1 + value: 47.17822553627135 + - type: nauc_recall_at_1_max + value: 31.206514215995206 + - type: nauc_recall_at_1_std + value: -1.8984121963184788 + - type: nauc_recall_at_20_diff1 + value: 24.682826207248283 + - type: nauc_recall_at_20_max + value: 20.777119838220408 + - type: nauc_recall_at_20_std + value: 1.2286788398315465 + - type: nauc_recall_at_3_diff1 + value: 35.715604782377035 + - type: nauc_recall_at_3_max + value: 23.7633639937056 + - type: nauc_recall_at_3_std + value: -2.868937897653619 + - type: nauc_recall_at_5_diff1 + value: 32.21252827575707 + - type: nauc_recall_at_5_max + value: 24.799142864683375 + - type: nauc_recall_at_5_std + value: 0.36296684299374204 + - type: ndcg_at_1 + value: 20.788999999999998 + - type: ndcg_at_10 + value: 26.667999999999996 + - type: ndcg_at_100 + value: 32.565 + - type: ndcg_at_1000 + value: 35.634 + - type: ndcg_at_20 + value: 28.642 + - type: ndcg_at_3 + value: 22.942 + - type: ndcg_at_5 + value: 24.514 + - type: precision_at_1 + value: 20.788999999999998 + - type: precision_at_10 + value: 4.947 + - type: precision_at_100 + value: 0.96 + - type: precision_at_1000 + value: 0.14100000000000001 + - type: precision_at_20 + value: 3.104 + - type: precision_at_3 + value: 10.748000000000001 + - type: precision_at_5 + value: 7.68 + - type: recall_at_1 + value: 16.822 + - type: recall_at_10 + value: 35.237 + - type: recall_at_100 + value: 61.219 + - type: recall_at_1000 + value: 82.499 + - type: recall_at_20 + value: 42.230000000000004 + - type: recall_at_3 + value: 24.524 + - type: recall_at_5 + value: 28.787000000000003 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackProgrammersRetrieval (default) + revision: 6184bc1440d2dbc7612be22b50686b8826d22b32 + split: test + type: mteb/cqadupstack-programmers + metrics: + - type: main_score + value: 21.66 + - type: map_at_1 + value: 12.416 + - type: map_at_10 + value: 17.684 + - type: map_at_100 + value: 18.851000000000003 + - type: map_at_1000 + value: 18.991 + - type: map_at_20 + value: 18.360000000000003 + - type: map_at_3 + value: 15.770999999999999 + - type: map_at_5 + value: 16.606 + - type: mrr_at_1 + value: 15.068493150684931 + - type: mrr_at_10 + value: 21.28823294919185 + - type: mrr_at_100 + value: 22.306240026063588 + - type: mrr_at_1000 + value: 22.395578374917164 + - type: mrr_at_20 + value: 21.90701850599165 + - type: mrr_at_3 + value: 19.273211567732123 + - type: mrr_at_5 + value: 20.397640791476412 + - type: nauc_map_at_1000_diff1 + value: 32.04680475392268 + - type: nauc_map_at_1000_max + value: 20.9527363509733 + - type: nauc_map_at_1000_std + value: 1.9775389393996066 + - type: nauc_map_at_100_diff1 + value: 32.05659071752874 + - type: nauc_map_at_100_max + value: 20.937669829415213 + - type: nauc_map_at_100_std + value: 1.8872130027911487 + - type: nauc_map_at_10_diff1 + value: 32.40493239661906 + - type: nauc_map_at_10_max + value: 20.24841030282171 + - type: nauc_map_at_10_std + value: 0.8873591420958411 + - type: nauc_map_at_1_diff1 + value: 39.50866679123135 + - type: nauc_map_at_1_max + value: 21.067083493139833 + - type: nauc_map_at_1_std + value: -1.255629309903365 + - type: nauc_map_at_20_diff1 + value: 32.06523680001786 + - type: nauc_map_at_20_max + value: 20.482809699946856 + - type: nauc_map_at_20_std + value: 1.2900775457613989 + - type: nauc_map_at_3_diff1 + value: 33.51328659054749 + - type: nauc_map_at_3_max + value: 19.351150884357097 + - type: nauc_map_at_3_std + value: -0.9449293271546024 + - type: nauc_map_at_5_diff1 + value: 32.672807388132 + - type: nauc_map_at_5_max + value: 19.888696407961916 + - type: nauc_map_at_5_std + value: -0.21370229639305732 + - type: nauc_mrr_at_1000_diff1 + value: 29.4702965330427 + - type: nauc_mrr_at_1000_max + value: 21.5485190959632 + - type: nauc_mrr_at_1000_std + value: 2.9474086643706716 + - type: nauc_mrr_at_100_diff1 + value: 29.444301031842237 + - type: nauc_mrr_at_100_max + value: 21.545652672940818 + - type: nauc_mrr_at_100_std + value: 2.930083417192537 + - type: nauc_mrr_at_10_diff1 + value: 29.839809988865028 + - type: nauc_mrr_at_10_max + value: 21.285084047773285 + - type: nauc_mrr_at_10_std + value: 2.3023735099948794 + - type: nauc_mrr_at_1_diff1 + value: 38.253685943964285 + - type: nauc_mrr_at_1_max + value: 23.506493457282993 + - type: nauc_mrr_at_1_std + value: 0.36623457899262024 + - type: nauc_mrr_at_20_diff1 + value: 29.359787332306013 + - type: nauc_mrr_at_20_max + value: 21.246732134190733 + - type: nauc_mrr_at_20_std + value: 2.6115784611487087 + - type: nauc_mrr_at_3_diff1 + value: 31.490392724228837 + - type: nauc_mrr_at_3_max + value: 21.643605643490904 + - type: nauc_mrr_at_3_std + value: 1.6756866672672965 + - type: nauc_mrr_at_5_diff1 + value: 30.18536933081793 + - type: nauc_mrr_at_5_max + value: 21.27264373907216 + - type: nauc_mrr_at_5_std + value: 1.7079689552978534 + - type: nauc_ndcg_at_1000_diff1 + value: 28.11169834333845 + - type: nauc_ndcg_at_1000_max + value: 22.65134504760621 + - type: nauc_ndcg_at_1000_std + value: 8.353986044564932 + - type: nauc_ndcg_at_100_diff1 + value: 28.265985165496417 + - type: nauc_ndcg_at_100_max + value: 22.530347672551887 + - type: nauc_ndcg_at_100_std + value: 6.968755339521627 + - type: nauc_ndcg_at_10_diff1 + value: 29.088878880551906 + - type: nauc_ndcg_at_10_max + value: 19.918818478137702 + - type: nauc_ndcg_at_10_std + value: 2.5519795248451795 + - type: nauc_ndcg_at_1_diff1 + value: 38.253685943964285 + - type: nauc_ndcg_at_1_max + value: 23.506493457282993 + - type: nauc_ndcg_at_1_std + value: 0.36623457899262024 + - type: nauc_ndcg_at_20_diff1 + value: 27.910656458566045 + - type: nauc_ndcg_at_20_max + value: 20.295061759944723 + - type: nauc_ndcg_at_20_std + value: 3.6145835770906833 + - type: nauc_ndcg_at_3_diff1 + value: 31.233680318242634 + - type: nauc_ndcg_at_3_max + value: 19.494683132285033 + - type: nauc_ndcg_at_3_std + value: 0.04355647255533374 + - type: nauc_ndcg_at_5_diff1 + value: 29.60761336088322 + - type: nauc_ndcg_at_5_max + value: 19.80719438136175 + - type: nauc_ndcg_at_5_std + value: 0.6195875169583498 + - type: nauc_precision_at_1000_diff1 + value: -4.9635863591586284 + - type: nauc_precision_at_1000_max + value: 10.205880001940644 + - type: nauc_precision_at_1000_std + value: 13.475741604004421 + - type: nauc_precision_at_100_diff1 + value: 7.633273326571685 + - type: nauc_precision_at_100_max + value: 23.151284304137622 + - type: nauc_precision_at_100_std + value: 20.405156194796863 + - type: nauc_precision_at_10_diff1 + value: 18.705937577794554 + - type: nauc_precision_at_10_max + value: 20.628035226019335 + - type: nauc_precision_at_10_std + value: 7.041902045527893 + - type: nauc_precision_at_1_diff1 + value: 38.253685943964285 + - type: nauc_precision_at_1_max + value: 23.506493457282993 + - type: nauc_precision_at_1_std + value: 0.36623457899262024 + - type: nauc_precision_at_20_diff1 + value: 14.129163643470525 + - type: nauc_precision_at_20_max + value: 20.39744876825584 + - type: nauc_precision_at_20_std + value: 10.808780160453079 + - type: nauc_precision_at_3_diff1 + value: 24.81724694529244 + - type: nauc_precision_at_3_max + value: 19.750250129235862 + - type: nauc_precision_at_3_std + value: 1.6383497722612925 + - type: nauc_precision_at_5_diff1 + value: 20.559816479129896 + - type: nauc_precision_at_5_max + value: 20.737938153703908 + - type: nauc_precision_at_5_std + value: 2.9329054609944767 + - type: nauc_recall_at_1000_diff1 + value: 14.657477263404504 + - type: nauc_recall_at_1000_max + value: 27.29789317523507 + - type: nauc_recall_at_1000_std + value: 41.54560242921126 + - type: nauc_recall_at_100_diff1 + value: 19.668816678808028 + - type: nauc_recall_at_100_max + value: 24.546392696829855 + - type: nauc_recall_at_100_std + value: 20.045457113413388 + - type: nauc_recall_at_10_diff1 + value: 22.57592036080691 + - type: nauc_recall_at_10_max + value: 17.30186041967476 + - type: nauc_recall_at_10_std + value: 5.75949108824036 + - type: nauc_recall_at_1_diff1 + value: 39.50866679123135 + - type: nauc_recall_at_1_max + value: 21.067083493139833 + - type: nauc_recall_at_1_std + value: -1.255629309903365 + - type: nauc_recall_at_20_diff1 + value: 18.597441888297915 + - type: nauc_recall_at_20_max + value: 17.76783323985467 + - type: nauc_recall_at_20_std + value: 7.756313900025849 + - type: nauc_recall_at_3_diff1 + value: 27.928359626631092 + - type: nauc_recall_at_3_max + value: 16.336637037641772 + - type: nauc_recall_at_3_std + value: -1.3417417785554366 + - type: nauc_recall_at_5_diff1 + value: 24.22251676423838 + - type: nauc_recall_at_5_max + value: 16.857422692031594 + - type: nauc_recall_at_5_std + value: 0.6185629064463674 + - type: ndcg_at_1 + value: 15.068000000000001 + - type: ndcg_at_10 + value: 21.66 + - type: ndcg_at_100 + value: 27.245 + - type: ndcg_at_1000 + value: 30.591 + - type: ndcg_at_20 + value: 23.955000000000002 + - type: ndcg_at_3 + value: 17.968999999999998 + - type: ndcg_at_5 + value: 19.352 + - type: precision_at_1 + value: 15.068000000000001 + - type: precision_at_10 + value: 4.326 + - type: precision_at_100 + value: 0.855 + - type: precision_at_1000 + value: 0.132 + - type: precision_at_20 + value: 2.8369999999999997 + - type: precision_at_3 + value: 8.713999999999999 + - type: precision_at_5 + value: 6.3469999999999995 + - type: recall_at_1 + value: 12.416 + - type: recall_at_10 + value: 30.008000000000003 + - type: recall_at_100 + value: 54.498999999999995 + - type: recall_at_1000 + value: 78.32000000000001 + - type: recall_at_20 + value: 38.378 + - type: recall_at_3 + value: 19.79 + - type: recall_at_5 + value: 23.376 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackRetrieval (default) + revision: CQADupstackRetrieval_is_a_combined_dataset + split: test + type: CQADupstackRetrieval_is_a_combined_dataset + metrics: + - type: main_score + value: 22.302333333333333 + - type: ndcg_at_10 + value: 22.302333333333333 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackStatsRetrieval (default) + revision: 65ac3a16b8e91f9cee4c9828cc7c335575432a2a + split: test + type: mteb/cqadupstack-stats + metrics: + - type: main_score + value: 17.253 + - type: map_at_1 + value: 9.722999999999999 + - type: map_at_10 + value: 14.280999999999999 + - type: map_at_100 + value: 15.065000000000001 + - type: map_at_1000 + value: 15.154 + - type: map_at_20 + value: 14.704999999999998 + - type: map_at_3 + value: 13.004 + - type: map_at_5 + value: 13.626 + - type: mrr_at_1 + value: 11.809815950920246 + - type: mrr_at_10 + value: 16.383959002824028 + - type: mrr_at_100 + value: 17.188709691814985 + - type: mrr_at_1000 + value: 17.269435610183017 + - type: mrr_at_20 + value: 16.836972625425393 + - type: mrr_at_3 + value: 15.081799591002035 + - type: mrr_at_5 + value: 15.710633946830258 + - type: nauc_map_at_1000_diff1 + value: 28.431623275634156 + - type: nauc_map_at_1000_max + value: 14.476316695164403 + - type: nauc_map_at_1000_std + value: 4.607998508591043 + - type: nauc_map_at_100_diff1 + value: 28.42367177875125 + - type: nauc_map_at_100_max + value: 14.394653506060012 + - type: nauc_map_at_100_std + value: 4.567472357591712 + - type: nauc_map_at_10_diff1 + value: 28.60653023312716 + - type: nauc_map_at_10_max + value: 14.78157644547682 + - type: nauc_map_at_10_std + value: 3.94994519901673 + - type: nauc_map_at_1_diff1 + value: 34.36968432094878 + - type: nauc_map_at_1_max + value: 17.456572010137457 + - type: nauc_map_at_1_std + value: 4.2640515305539415 + - type: nauc_map_at_20_diff1 + value: 28.510596490501573 + - type: nauc_map_at_20_max + value: 14.318541992037401 + - type: nauc_map_at_20_std + value: 4.254075392620963 + - type: nauc_map_at_3_diff1 + value: 30.539716169861936 + - type: nauc_map_at_3_max + value: 16.14471431902583 + - type: nauc_map_at_3_std + value: 4.973502209268125 + - type: nauc_map_at_5_diff1 + value: 29.261684655915225 + - type: nauc_map_at_5_max + value: 15.372748605327446 + - type: nauc_map_at_5_std + value: 4.39285622535654 + - type: nauc_mrr_at_1000_diff1 + value: 28.972718024301447 + - type: nauc_mrr_at_1000_max + value: 17.826835397341046 + - type: nauc_mrr_at_1000_std + value: 6.917284034347911 + - type: nauc_mrr_at_100_diff1 + value: 28.945997371755087 + - type: nauc_mrr_at_100_max + value: 17.739278412823893 + - type: nauc_mrr_at_100_std + value: 6.899424135908487 + - type: nauc_mrr_at_10_diff1 + value: 29.06935519309891 + - type: nauc_mrr_at_10_max + value: 18.21083753088906 + - type: nauc_mrr_at_10_std + value: 6.518493253737144 + - type: nauc_mrr_at_1_diff1 + value: 35.63041619844435 + - type: nauc_mrr_at_1_max + value: 22.830306049699338 + - type: nauc_mrr_at_1_std + value: 7.826683917417351 + - type: nauc_mrr_at_20_diff1 + value: 29.016004511022537 + - type: nauc_mrr_at_20_max + value: 17.788437345787926 + - type: nauc_mrr_at_20_std + value: 6.652263770077456 + - type: nauc_mrr_at_3_diff1 + value: 30.644333070723466 + - type: nauc_mrr_at_3_max + value: 19.667632613725225 + - type: nauc_mrr_at_3_std + value: 7.743380165559918 + - type: nauc_mrr_at_5_diff1 + value: 29.829376205828805 + - type: nauc_mrr_at_5_max + value: 18.722595091544253 + - type: nauc_mrr_at_5_std + value: 6.818524829545593 + - type: nauc_ndcg_at_1000_diff1 + value: 25.62248172657835 + - type: nauc_ndcg_at_1000_max + value: 14.223326419511073 + - type: nauc_ndcg_at_1000_std + value: 7.495752604082028 + - type: nauc_ndcg_at_100_diff1 + value: 25.499428653265642 + - type: nauc_ndcg_at_100_max + value: 12.585064293899102 + - type: nauc_ndcg_at_100_std + value: 6.664889384341954 + - type: nauc_ndcg_at_10_diff1 + value: 25.74972755098383 + - type: nauc_ndcg_at_10_max + value: 13.793434874824303 + - type: nauc_ndcg_at_10_std + value: 3.883648047462527 + - type: nauc_ndcg_at_1_diff1 + value: 35.63041619844435 + - type: nauc_ndcg_at_1_max + value: 22.830306049699338 + - type: nauc_ndcg_at_1_std + value: 7.826683917417351 + - type: nauc_ndcg_at_20_diff1 + value: 25.334745687494443 + - type: nauc_ndcg_at_20_max + value: 12.305607906859144 + - type: nauc_ndcg_at_20_std + value: 4.7413728340444505 + - type: nauc_ndcg_at_3_diff1 + value: 29.45395763143249 + - type: nauc_ndcg_at_3_max + value: 16.23690234046979 + - type: nauc_ndcg_at_3_std + value: 6.142105291678576 + - type: nauc_ndcg_at_5_diff1 + value: 27.444736442905455 + - type: nauc_ndcg_at_5_max + value: 14.93362615759676 + - type: nauc_ndcg_at_5_std + value: 4.7342440148611225 + - type: nauc_precision_at_1000_diff1 + value: 16.80575206659899 + - type: nauc_precision_at_1000_max + value: 17.66226703408546 + - type: nauc_precision_at_1000_std + value: 18.77422949877631 + - type: nauc_precision_at_100_diff1 + value: 21.105287938477233 + - type: nauc_precision_at_100_max + value: 13.591179380636214 + - type: nauc_precision_at_100_std + value: 16.55840962012843 + - type: nauc_precision_at_10_diff1 + value: 21.469758913525254 + - type: nauc_precision_at_10_max + value: 15.320780706573464 + - type: nauc_precision_at_10_std + value: 6.351289997170259 + - type: nauc_precision_at_1_diff1 + value: 35.63041619844435 + - type: nauc_precision_at_1_max + value: 22.830306049699338 + - type: nauc_precision_at_1_std + value: 7.826683917417351 + - type: nauc_precision_at_20_diff1 + value: 20.438996654370953 + - type: nauc_precision_at_20_max + value: 11.895395539109575 + - type: nauc_precision_at_20_std + value: 9.227372989467945 + - type: nauc_precision_at_3_diff1 + value: 27.958385745280534 + - type: nauc_precision_at_3_max + value: 18.76663358991842 + - type: nauc_precision_at_3_std + value: 8.804799926813658 + - type: nauc_precision_at_5_diff1 + value: 25.20756412916346 + - type: nauc_precision_at_5_max + value: 17.16752690039525 + - type: nauc_precision_at_5_std + value: 7.822524248176865 + - type: nauc_recall_at_1000_diff1 + value: 17.093227818066353 + - type: nauc_recall_at_1000_max + value: 12.628515233697735 + - type: nauc_recall_at_1000_std + value: 16.519924218447994 + - type: nauc_recall_at_100_diff1 + value: 18.19732935930814 + - type: nauc_recall_at_100_max + value: 4.740051109026774 + - type: nauc_recall_at_100_std + value: 10.729043783837753 + - type: nauc_recall_at_10_diff1 + value: 17.84235497242283 + - type: nauc_recall_at_10_max + value: 7.9110522988146155 + - type: nauc_recall_at_10_std + value: 1.147900198002905 + - type: nauc_recall_at_1_diff1 + value: 34.36968432094878 + - type: nauc_recall_at_1_max + value: 17.456572010137457 + - type: nauc_recall_at_1_std + value: 4.2640515305539415 + - type: nauc_recall_at_20_diff1 + value: 16.692476991368853 + - type: nauc_recall_at_20_max + value: 3.809776817661501 + - type: nauc_recall_at_20_std + value: 3.6575551737685954 + - type: nauc_recall_at_3_diff1 + value: 25.110591985459862 + - type: nauc_recall_at_3_max + value: 13.681824792451245 + - type: nauc_recall_at_3_std + value: 5.806771643452482 + - type: nauc_recall_at_5_diff1 + value: 21.0191985797923 + - type: nauc_recall_at_5_max + value: 10.837381063643834 + - type: nauc_recall_at_5_std + value: 3.228418252689027 + - type: ndcg_at_1 + value: 11.81 + - type: ndcg_at_10 + value: 17.253 + - type: ndcg_at_100 + value: 21.404 + - type: ndcg_at_1000 + value: 24.09 + - type: ndcg_at_20 + value: 18.801000000000002 + - type: ndcg_at_3 + value: 14.716999999999999 + - type: ndcg_at_5 + value: 15.706000000000001 + - type: precision_at_1 + value: 11.81 + - type: precision_at_10 + value: 2.9749999999999996 + - type: precision_at_100 + value: 0.543 + - type: precision_at_1000 + value: 0.084 + - type: precision_at_20 + value: 1.848 + - type: precision_at_3 + value: 6.902 + - type: precision_at_5 + value: 4.816 + - type: recall_at_1 + value: 9.722999999999999 + - type: recall_at_10 + value: 24.569 + - type: recall_at_100 + value: 43.997 + - type: recall_at_1000 + value: 64.44 + - type: recall_at_20 + value: 30.505 + - type: recall_at_3 + value: 17.134 + - type: recall_at_5 + value: 19.72 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackTexRetrieval (default) + revision: 46989137a86843e03a6195de44b09deda022eec7 + split: test + type: mteb/cqadupstack-tex + metrics: + - type: main_score + value: 13.308 + - type: map_at_1 + value: 7.497 + - type: map_at_10 + value: 10.846 + - type: map_at_100 + value: 11.498999999999999 + - type: map_at_1000 + value: 11.618 + - type: map_at_20 + value: 11.161999999999999 + - type: map_at_3 + value: 9.658999999999999 + - type: map_at_5 + value: 10.298 + - type: mrr_at_1 + value: 9.11906400550585 + - type: mrr_at_10 + value: 12.993232392750626 + - type: mrr_at_100 + value: 13.701403675494117 + - type: mrr_at_1000 + value: 13.798101712770123 + - type: mrr_at_20 + value: 13.360764217937035 + - type: mrr_at_3 + value: 11.6655196145905 + - type: mrr_at_5 + value: 12.362353750860274 + - type: nauc_map_at_1000_diff1 + value: 29.030158454163164 + - type: nauc_map_at_1000_max + value: 15.750545094681929 + - type: nauc_map_at_1000_std + value: -3.0798436292807834 + - type: nauc_map_at_100_diff1 + value: 29.05038743174521 + - type: nauc_map_at_100_max + value: 15.679082682471822 + - type: nauc_map_at_100_std + value: -3.2003921265004855 + - type: nauc_map_at_10_diff1 + value: 29.680682239615308 + - type: nauc_map_at_10_max + value: 15.532980267877802 + - type: nauc_map_at_10_std + value: -3.622076099535413 + - type: nauc_map_at_1_diff1 + value: 37.49924172327444 + - type: nauc_map_at_1_max + value: 14.852898999380606 + - type: nauc_map_at_1_std + value: -3.8871845491808403 + - type: nauc_map_at_20_diff1 + value: 29.440127025124063 + - type: nauc_map_at_20_max + value: 15.566926763278111 + - type: nauc_map_at_20_std + value: -3.5118135905883445 + - type: nauc_map_at_3_diff1 + value: 31.87407675131833 + - type: nauc_map_at_3_max + value: 16.133052442782088 + - type: nauc_map_at_3_std + value: -3.7331459743832536 + - type: nauc_map_at_5_diff1 + value: 30.702048393849918 + - type: nauc_map_at_5_max + value: 15.7292852737471 + - type: nauc_map_at_5_std + value: -3.72714036461797 + - type: nauc_mrr_at_1000_diff1 + value: 27.069591144268795 + - type: nauc_mrr_at_1000_max + value: 17.335323991978157 + - type: nauc_mrr_at_1000_std + value: -2.1443215489774863 + - type: nauc_mrr_at_100_diff1 + value: 27.06995261671637 + - type: nauc_mrr_at_100_max + value: 17.3285570198275 + - type: nauc_mrr_at_100_std + value: -2.1819679734953903 + - type: nauc_mrr_at_10_diff1 + value: 27.57687228309106 + - type: nauc_mrr_at_10_max + value: 17.166971785334017 + - type: nauc_mrr_at_10_std + value: -2.6000743496984526 + - type: nauc_mrr_at_1_diff1 + value: 35.22676568917156 + - type: nauc_mrr_at_1_max + value: 17.007211079819626 + - type: nauc_mrr_at_1_std + value: -4.214696308727653 + - type: nauc_mrr_at_20_diff1 + value: 27.374588178560465 + - type: nauc_mrr_at_20_max + value: 17.23758467893531 + - type: nauc_mrr_at_20_std + value: -2.4124837810565603 + - type: nauc_mrr_at_3_diff1 + value: 29.722577971696918 + - type: nauc_mrr_at_3_max + value: 18.07384167733403 + - type: nauc_mrr_at_3_std + value: -3.003414797443647 + - type: nauc_mrr_at_5_diff1 + value: 28.45980370469956 + - type: nauc_mrr_at_5_max + value: 17.511976658495847 + - type: nauc_mrr_at_5_std + value: -2.5924858663986745 + - type: nauc_ndcg_at_1000_diff1 + value: 23.077231893052307 + - type: nauc_ndcg_at_1000_max + value: 16.93593483664181 + - type: nauc_ndcg_at_1000_std + value: 1.2092406562986315 + - type: nauc_ndcg_at_100_diff1 + value: 23.549727836162358 + - type: nauc_ndcg_at_100_max + value: 15.750436011474273 + - type: nauc_ndcg_at_100_std + value: -0.9019324316165611 + - type: nauc_ndcg_at_10_diff1 + value: 26.053761788639434 + - type: nauc_ndcg_at_10_max + value: 15.3669306793647 + - type: nauc_ndcg_at_10_std + value: -3.193779292269917 + - type: nauc_ndcg_at_1_diff1 + value: 35.22676568917156 + - type: nauc_ndcg_at_1_max + value: 17.007211079819626 + - type: nauc_ndcg_at_1_std + value: -4.214696308727653 + - type: nauc_ndcg_at_20_diff1 + value: 25.425326574435168 + - type: nauc_ndcg_at_20_max + value: 15.385189154016906 + - type: nauc_ndcg_at_20_std + value: -2.7870454259014545 + - type: nauc_ndcg_at_3_diff1 + value: 29.685264931512716 + - type: nauc_ndcg_at_3_max + value: 17.07409526298788 + - type: nauc_ndcg_at_3_std + value: -3.4063850629923293 + - type: nauc_ndcg_at_5_diff1 + value: 27.89860104840894 + - type: nauc_ndcg_at_5_max + value: 15.996740122854927 + - type: nauc_ndcg_at_5_std + value: -3.3146899970251873 + - type: nauc_precision_at_1000_diff1 + value: 6.214195083416471 + - type: nauc_precision_at_1000_max + value: 24.273670809985404 + - type: nauc_precision_at_1000_std + value: 17.553556491344104 + - type: nauc_precision_at_100_diff1 + value: 11.6615588663656 + - type: nauc_precision_at_100_max + value: 20.59244105372682 + - type: nauc_precision_at_100_std + value: 8.072189089366798 + - type: nauc_precision_at_10_diff1 + value: 18.279161444567706 + - type: nauc_precision_at_10_max + value: 17.664508142320727 + - type: nauc_precision_at_10_std + value: -1.0218966605840407 + - type: nauc_precision_at_1_diff1 + value: 35.22676568917156 + - type: nauc_precision_at_1_max + value: 17.007211079819626 + - type: nauc_precision_at_1_std + value: -4.214696308727653 + - type: nauc_precision_at_20_diff1 + value: 16.855549347544613 + - type: nauc_precision_at_20_max + value: 18.640589054149743 + - type: nauc_precision_at_20_std + value: 0.7553558754796067 + - type: nauc_precision_at_3_diff1 + value: 25.61293747306704 + - type: nauc_precision_at_3_max + value: 20.254901193584562 + - type: nauc_precision_at_3_std + value: -2.9517852127763153 + - type: nauc_precision_at_5_diff1 + value: 22.32451285561962 + - type: nauc_precision_at_5_max + value: 18.709490300571886 + - type: nauc_precision_at_5_std + value: -2.0702847848899615 + - type: nauc_recall_at_1000_diff1 + value: 8.102081393478185 + - type: nauc_recall_at_1000_max + value: 17.111395305264892 + - type: nauc_recall_at_1000_std + value: 14.340291614611578 + - type: nauc_recall_at_100_diff1 + value: 12.480368811829736 + - type: nauc_recall_at_100_max + value: 12.879220685006636 + - type: nauc_recall_at_100_std + value: 3.650162252310097 + - type: nauc_recall_at_10_diff1 + value: 19.461318204968205 + - type: nauc_recall_at_10_max + value: 12.823289358103562 + - type: nauc_recall_at_10_std + value: -3.1960264321653895 + - type: nauc_recall_at_1_diff1 + value: 37.49924172327444 + - type: nauc_recall_at_1_max + value: 14.852898999380606 + - type: nauc_recall_at_1_std + value: -3.8871845491808403 + - type: nauc_recall_at_20_diff1 + value: 17.698352862902524 + - type: nauc_recall_at_20_max + value: 12.409413309293047 + - type: nauc_recall_at_20_std + value: -2.0913697847507136 + - type: nauc_recall_at_3_diff1 + value: 26.236763474946116 + - type: nauc_recall_at_3_max + value: 15.89287407458128 + - type: nauc_recall_at_3_std + value: -3.776018275852628 + - type: nauc_recall_at_5_diff1 + value: 23.10472386873395 + - type: nauc_recall_at_5_max + value: 14.09706657151941 + - type: nauc_recall_at_5_std + value: -3.7053105237887296 + - type: ndcg_at_1 + value: 9.119 + - type: ndcg_at_10 + value: 13.308 + - type: ndcg_at_100 + value: 16.98 + - type: ndcg_at_1000 + value: 20.488 + - type: ndcg_at_20 + value: 14.455000000000002 + - type: ndcg_at_3 + value: 10.982 + - type: ndcg_at_5 + value: 12.003 + - type: precision_at_1 + value: 9.119 + - type: precision_at_10 + value: 2.4979999999999998 + - type: precision_at_100 + value: 0.519 + - type: precision_at_1000 + value: 0.099 + - type: precision_at_20 + value: 1.5779999999999998 + - type: precision_at_3 + value: 5.288 + - type: precision_at_5 + value: 3.8890000000000002 + - type: recall_at_1 + value: 7.497 + - type: recall_at_10 + value: 18.817999999999998 + - type: recall_at_100 + value: 35.893 + - type: recall_at_1000 + value: 61.966 + - type: recall_at_20 + value: 23.017000000000003 + - type: recall_at_3 + value: 12.199 + - type: recall_at_5 + value: 14.87 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackUnixRetrieval (default) + revision: 6c6430d3a6d36f8d2a829195bc5dc94d7e063e53 + split: test + type: mteb/cqadupstack-unix + metrics: + - type: main_score + value: 20.061999999999998 + - type: map_at_1 + value: 11.856 + - type: map_at_10 + value: 16.685 + - type: map_at_100 + value: 17.433 + - type: map_at_1000 + value: 17.558 + - type: map_at_20 + value: 17.041999999999998 + - type: map_at_3 + value: 15.021 + - type: map_at_5 + value: 15.931999999999999 + - type: mrr_at_1 + value: 14.17910447761194 + - type: mrr_at_10 + value: 19.398468964700307 + - type: mrr_at_100 + value: 20.153361230634783 + - type: mrr_at_1000 + value: 20.25140420668968 + - type: mrr_at_20 + value: 19.79354704809282 + - type: mrr_at_3 + value: 17.63059701492538 + - type: mrr_at_5 + value: 18.516791044776127 + - type: nauc_map_at_1000_diff1 + value: 39.29033459612684 + - type: nauc_map_at_1000_max + value: 27.17416795511821 + - type: nauc_map_at_1000_std + value: -6.92127611795475 + - type: nauc_map_at_100_diff1 + value: 39.32396099754708 + - type: nauc_map_at_100_max + value: 27.09334212594238 + - type: nauc_map_at_100_std + value: -7.039062385443858 + - type: nauc_map_at_10_diff1 + value: 39.94340086930468 + - type: nauc_map_at_10_max + value: 27.423789336152417 + - type: nauc_map_at_10_std + value: -7.508495669216843 + - type: nauc_map_at_1_diff1 + value: 47.64613699501138 + - type: nauc_map_at_1_max + value: 31.632492599268748 + - type: nauc_map_at_1_std + value: -7.883784832592304 + - type: nauc_map_at_20_diff1 + value: 39.45107288329592 + - type: nauc_map_at_20_max + value: 27.15650902645131 + - type: nauc_map_at_20_std + value: -7.301916707077087 + - type: nauc_map_at_3_diff1 + value: 41.801336320148984 + - type: nauc_map_at_3_max + value: 28.342684341392683 + - type: nauc_map_at_3_std + value: -8.213654438632787 + - type: nauc_map_at_5_diff1 + value: 40.973958128612786 + - type: nauc_map_at_5_max + value: 28.355847958983126 + - type: nauc_map_at_5_std + value: -7.204454459764011 + - type: nauc_mrr_at_1000_diff1 + value: 39.68737143543835 + - type: nauc_mrr_at_1000_max + value: 28.74366308891808 + - type: nauc_mrr_at_1000_std + value: -5.74519909264754 + - type: nauc_mrr_at_100_diff1 + value: 39.696965050178875 + - type: nauc_mrr_at_100_max + value: 28.71065540406762 + - type: nauc_mrr_at_100_std + value: -5.8117683155682895 + - type: nauc_mrr_at_10_diff1 + value: 40.22891666712493 + - type: nauc_mrr_at_10_max + value: 28.97882832718155 + - type: nauc_mrr_at_10_std + value: -6.167061574555064 + - type: nauc_mrr_at_1_diff1 + value: 48.39795549312159 + - type: nauc_mrr_at_1_max + value: 33.31270433423697 + - type: nauc_mrr_at_1_std + value: -5.8264509798445925 + - type: nauc_mrr_at_20_diff1 + value: 39.75516014377185 + - type: nauc_mrr_at_20_max + value: 28.762238070807676 + - type: nauc_mrr_at_20_std + value: -6.015233094372284 + - type: nauc_mrr_at_3_diff1 + value: 42.39647678330573 + - type: nauc_mrr_at_3_max + value: 29.854246402890674 + - type: nauc_mrr_at_3_std + value: -6.989062488249666 + - type: nauc_mrr_at_5_diff1 + value: 41.32547115377251 + - type: nauc_mrr_at_5_max + value: 29.756253662694554 + - type: nauc_mrr_at_5_std + value: -5.989324088608618 + - type: nauc_ndcg_at_1000_diff1 + value: 33.24611188020779 + - type: nauc_ndcg_at_1000_max + value: 25.5685050419863 + - type: nauc_ndcg_at_1000_std + value: -2.1838171971216838 + - type: nauc_ndcg_at_100_diff1 + value: 34.12429897480726 + - type: nauc_ndcg_at_100_max + value: 24.386449655174115 + - type: nauc_ndcg_at_100_std + value: -4.463092158837694 + - type: nauc_ndcg_at_10_diff1 + value: 36.7514146310574 + - type: nauc_ndcg_at_10_max + value: 25.816604124438165 + - type: nauc_ndcg_at_10_std + value: -6.864047505974296 + - type: nauc_ndcg_at_1_diff1 + value: 48.39795549312159 + - type: nauc_ndcg_at_1_max + value: 33.31270433423697 + - type: nauc_ndcg_at_1_std + value: -5.8264509798445925 + - type: nauc_ndcg_at_20_diff1 + value: 35.19768360191347 + - type: nauc_ndcg_at_20_max + value: 25.02001675750392 + - type: nauc_ndcg_at_20_std + value: -6.20782733166831 + - type: nauc_ndcg_at_3_diff1 + value: 40.154344522643925 + - type: nauc_ndcg_at_3_max + value: 27.955302837392672 + - type: nauc_ndcg_at_3_std + value: -7.6328532886404235 + - type: nauc_ndcg_at_5_diff1 + value: 38.743591122825606 + - type: nauc_ndcg_at_5_max + value: 27.72241812814964 + - type: nauc_ndcg_at_5_std + value: -6.257812072012101 + - type: nauc_precision_at_1000_diff1 + value: -3.9866748764702096 + - type: nauc_precision_at_1000_max + value: 14.72470736881832 + - type: nauc_precision_at_1000_std + value: 15.962534584653012 + - type: nauc_precision_at_100_diff1 + value: 14.40948301991166 + - type: nauc_precision_at_100_max + value: 16.61733733078467 + - type: nauc_precision_at_100_std + value: 6.847882296599798 + - type: nauc_precision_at_10_diff1 + value: 27.51873293006865 + - type: nauc_precision_at_10_max + value: 22.893866555907746 + - type: nauc_precision_at_10_std + value: -3.030805589162383 + - type: nauc_precision_at_1_diff1 + value: 48.39795549312159 + - type: nauc_precision_at_1_max + value: 33.31270433423697 + - type: nauc_precision_at_1_std + value: -5.8264509798445925 + - type: nauc_precision_at_20_diff1 + value: 22.56834807636722 + - type: nauc_precision_at_20_max + value: 20.490661671424448 + - type: nauc_precision_at_20_std + value: -0.660069645072748 + - type: nauc_precision_at_3_diff1 + value: 36.978184171791156 + - type: nauc_precision_at_3_max + value: 26.478381926029265 + - type: nauc_precision_at_3_std + value: -6.091960417034656 + - type: nauc_precision_at_5_diff1 + value: 33.58525371051779 + - type: nauc_precision_at_5_max + value: 26.334754741578593 + - type: nauc_precision_at_5_std + value: -3.154368502496007 + - type: nauc_recall_at_1000_diff1 + value: 5.958742292353638 + - type: nauc_recall_at_1000_max + value: 15.864543076240528 + - type: nauc_recall_at_1000_std + value: 21.86695402215286 + - type: nauc_recall_at_100_diff1 + value: 17.82865358233198 + - type: nauc_recall_at_100_max + value: 13.118309558968022 + - type: nauc_recall_at_100_std + value: 2.3032751559115114 + - type: nauc_recall_at_10_diff1 + value: 27.980644115353996 + - type: nauc_recall_at_10_max + value: 19.39950863468228 + - type: nauc_recall_at_10_std + value: -6.36618746193429 + - type: nauc_recall_at_1_diff1 + value: 47.64613699501138 + - type: nauc_recall_at_1_max + value: 31.632492599268748 + - type: nauc_recall_at_1_std + value: -7.883784832592304 + - type: nauc_recall_at_20_diff1 + value: 22.967595804626253 + - type: nauc_recall_at_20_max + value: 16.693327271336244 + - type: nauc_recall_at_20_std + value: -4.559238353011102 + - type: nauc_recall_at_3_diff1 + value: 35.41022087124811 + - type: nauc_recall_at_3_max + value: 24.543890488663166 + - type: nauc_recall_at_3_std + value: -8.200059552235023 + - type: nauc_recall_at_5_diff1 + value: 32.09822917090586 + - type: nauc_recall_at_5_max + value: 23.82588196783892 + - type: nauc_recall_at_5_std + value: -4.932704288647733 + - type: ndcg_at_1 + value: 14.179 + - type: ndcg_at_10 + value: 20.061999999999998 + - type: ndcg_at_100 + value: 24.149 + - type: ndcg_at_1000 + value: 27.644999999999996 + - type: ndcg_at_20 + value: 21.387999999999998 + - type: ndcg_at_3 + value: 16.794 + - type: ndcg_at_5 + value: 18.224 + - type: precision_at_1 + value: 14.179 + - type: precision_at_10 + value: 3.582 + - type: precision_at_100 + value: 0.623 + - type: precision_at_1000 + value: 0.105 + - type: precision_at_20 + value: 2.1319999999999997 + - type: precision_at_3 + value: 7.774 + - type: precision_at_5 + value: 5.5969999999999995 + - type: recall_at_1 + value: 11.856 + - type: recall_at_10 + value: 27.778999999999996 + - type: recall_at_100 + value: 46.733000000000004 + - type: recall_at_1000 + value: 72.481 + - type: recall_at_20 + value: 32.737 + - type: recall_at_3 + value: 18.859 + - type: recall_at_5 + value: 22.435 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackWebmastersRetrieval (default) + revision: 160c094312a0e1facb97e55eeddb698c0abe3571 + split: test + type: mteb/cqadupstack-webmasters + metrics: + - type: main_score + value: 23.735999999999997 + - type: map_at_1 + value: 13.164000000000001 + - type: map_at_10 + value: 19.317999999999998 + - type: map_at_100 + value: 20.463 + - type: map_at_1000 + value: 20.646 + - type: map_at_20 + value: 19.808 + - type: map_at_3 + value: 17.126 + - type: map_at_5 + value: 18.056 + - type: mrr_at_1 + value: 16.600790513833992 + - type: mrr_at_10 + value: 22.620067130936693 + - type: mrr_at_100 + value: 23.601448756772193 + - type: mrr_at_1000 + value: 23.675507750087586 + - type: mrr_at_20 + value: 23.09510872850641 + - type: mrr_at_3 + value: 20.685111989459816 + - type: mrr_at_5 + value: 21.46574440052701 + - type: nauc_map_at_1000_diff1 + value: 38.04966249247377 + - type: nauc_map_at_1000_max + value: 16.252263992463384 + - type: nauc_map_at_1000_std + value: -1.7460502582062356 + - type: nauc_map_at_100_diff1 + value: 38.014610979412474 + - type: nauc_map_at_100_max + value: 16.21534617931594 + - type: nauc_map_at_100_std + value: -1.862936037740923 + - type: nauc_map_at_10_diff1 + value: 37.85306201039408 + - type: nauc_map_at_10_max + value: 16.316152483605283 + - type: nauc_map_at_10_std + value: -1.9300768321014996 + - type: nauc_map_at_1_diff1 + value: 46.32670783118563 + - type: nauc_map_at_1_max + value: 19.162748070034993 + - type: nauc_map_at_1_std + value: -7.2143378209361435 + - type: nauc_map_at_20_diff1 + value: 37.76015277914087 + - type: nauc_map_at_20_max + value: 16.402558719060888 + - type: nauc_map_at_20_std + value: -2.065612538672495 + - type: nauc_map_at_3_diff1 + value: 39.76679931113434 + - type: nauc_map_at_3_max + value: 16.834290630961544 + - type: nauc_map_at_3_std + value: -3.9003170439130335 + - type: nauc_map_at_5_diff1 + value: 39.03208154755538 + - type: nauc_map_at_5_max + value: 16.225900244095133 + - type: nauc_map_at_5_std + value: -2.4557998742917273 + - type: nauc_mrr_at_1000_diff1 + value: 37.458213267102465 + - type: nauc_mrr_at_1000_max + value: 16.263132423271077 + - type: nauc_mrr_at_1000_std + value: -0.6455583895471498 + - type: nauc_mrr_at_100_diff1 + value: 37.45543984270519 + - type: nauc_mrr_at_100_max + value: 16.185738866185893 + - type: nauc_mrr_at_100_std + value: -0.6962640945779722 + - type: nauc_mrr_at_10_diff1 + value: 37.16827089026705 + - type: nauc_mrr_at_10_max + value: 15.901025716349201 + - type: nauc_mrr_at_10_std + value: -0.6599647334904797 + - type: nauc_mrr_at_1_diff1 + value: 44.322572770568456 + - type: nauc_mrr_at_1_max + value: 19.02126117731051 + - type: nauc_mrr_at_1_std + value: -5.8998188281784625 + - type: nauc_mrr_at_20_diff1 + value: 37.24551389599038 + - type: nauc_mrr_at_20_max + value: 16.113728443160127 + - type: nauc_mrr_at_20_std + value: -0.8856480048238807 + - type: nauc_mrr_at_3_diff1 + value: 38.800389636963004 + - type: nauc_mrr_at_3_max + value: 16.691447775512863 + - type: nauc_mrr_at_3_std + value: -2.2008701696190474 + - type: nauc_mrr_at_5_diff1 + value: 38.17066041754819 + - type: nauc_mrr_at_5_max + value: 15.854986493430074 + - type: nauc_mrr_at_5_std + value: -1.3419132385788708 + - type: nauc_ndcg_at_1000_diff1 + value: 36.500354605077305 + - type: nauc_ndcg_at_1000_max + value: 18.158853474546227 + - type: nauc_ndcg_at_1000_std + value: 3.7042707188045783 + - type: nauc_ndcg_at_100_diff1 + value: 35.68797486655767 + - type: nauc_ndcg_at_100_max + value: 15.949868116992763 + - type: nauc_ndcg_at_100_std + value: 1.8743757496922573 + - type: nauc_ndcg_at_10_diff1 + value: 34.44579459042251 + - type: nauc_ndcg_at_10_max + value: 14.976928472341097 + - type: nauc_ndcg_at_10_std + value: 0.668632426387858 + - type: nauc_ndcg_at_1_diff1 + value: 44.322572770568456 + - type: nauc_ndcg_at_1_max + value: 19.02126117731051 + - type: nauc_ndcg_at_1_std + value: -5.8998188281784625 + - type: nauc_ndcg_at_20_diff1 + value: 34.47554348325645 + - type: nauc_ndcg_at_20_max + value: 15.617518114283014 + - type: nauc_ndcg_at_20_std + value: 0.23279335295578624 + - type: nauc_ndcg_at_3_diff1 + value: 37.34865309502302 + - type: nauc_ndcg_at_3_max + value: 15.6035028610235 + - type: nauc_ndcg_at_3_std + value: -2.042290469888462 + - type: nauc_ndcg_at_5_diff1 + value: 36.710946337067 + - type: nauc_ndcg_at_5_max + value: 14.502265833101022 + - type: nauc_ndcg_at_5_std + value: -0.26386753108907807 + - type: nauc_precision_at_1000_diff1 + value: 3.5611970722748056 + - type: nauc_precision_at_1000_max + value: 6.9688736574296275 + - type: nauc_precision_at_1000_std + value: 7.291986774352235 + - type: nauc_precision_at_100_diff1 + value: 18.866491470530185 + - type: nauc_precision_at_100_max + value: 3.0721103361408497 + - type: nauc_precision_at_100_std + value: 4.384934503700695 + - type: nauc_precision_at_10_diff1 + value: 20.850504784204883 + - type: nauc_precision_at_10_max + value: 10.633189141801425 + - type: nauc_precision_at_10_std + value: 5.014926409884033 + - type: nauc_precision_at_1_diff1 + value: 44.322572770568456 + - type: nauc_precision_at_1_max + value: 19.02126117731051 + - type: nauc_precision_at_1_std + value: -5.8998188281784625 + - type: nauc_precision_at_20_diff1 + value: 20.309109922155518 + - type: nauc_precision_at_20_max + value: 9.029797084048417 + - type: nauc_precision_at_20_std + value: 2.758218391395686 + - type: nauc_precision_at_3_diff1 + value: 30.196789766812422 + - type: nauc_precision_at_3_max + value: 13.456577178909065 + - type: nauc_precision_at_3_std + value: 0.49917879030090373 + - type: nauc_precision_at_5_diff1 + value: 27.706537485425653 + - type: nauc_precision_at_5_max + value: 9.849229139569182 + - type: nauc_precision_at_5_std + value: 3.685125093555483 + - type: nauc_recall_at_1000_diff1 + value: 33.96229420221514 + - type: nauc_recall_at_1000_max + value: 37.16052892689619 + - type: nauc_recall_at_1000_std + value: 36.18222346361014 + - type: nauc_recall_at_100_diff1 + value: 27.657710979013174 + - type: nauc_recall_at_100_max + value: 15.352705013529967 + - type: nauc_recall_at_100_std + value: 11.850919034123116 + - type: nauc_recall_at_10_diff1 + value: 25.46843551212912 + - type: nauc_recall_at_10_max + value: 12.024769591895815 + - type: nauc_recall_at_10_std + value: 5.710557786436904 + - type: nauc_recall_at_1_diff1 + value: 46.32670783118563 + - type: nauc_recall_at_1_max + value: 19.162748070034993 + - type: nauc_recall_at_1_std + value: -7.2143378209361435 + - type: nauc_recall_at_20_diff1 + value: 24.950754303786603 + - type: nauc_recall_at_20_max + value: 13.779914894639022 + - type: nauc_recall_at_20_std + value: 4.337235880676669 + - type: nauc_recall_at_3_diff1 + value: 33.979943512337485 + - type: nauc_recall_at_3_max + value: 14.35407227008922 + - type: nauc_recall_at_3_std + value: -0.5408111812033761 + - type: nauc_recall_at_5_diff1 + value: 31.887819659716687 + - type: nauc_recall_at_5_max + value: 12.266354466300289 + - type: nauc_recall_at_5_std + value: 3.67855636796736 + - type: ndcg_at_1 + value: 16.601 + - type: ndcg_at_10 + value: 23.735999999999997 + - type: ndcg_at_100 + value: 29.047 + - type: ndcg_at_1000 + value: 32.323 + - type: ndcg_at_20 + value: 25.222 + - type: ndcg_at_3 + value: 20.013 + - type: ndcg_at_5 + value: 21.165 + - type: precision_at_1 + value: 16.601 + - type: precision_at_10 + value: 4.7829999999999995 + - type: precision_at_100 + value: 1.077 + - type: precision_at_1000 + value: 0.197 + - type: precision_at_20 + value: 3.0429999999999997 + - type: precision_at_3 + value: 9.881 + - type: precision_at_5 + value: 7.074999999999999 + - type: recall_at_1 + value: 13.164000000000001 + - type: recall_at_10 + value: 33.041 + - type: recall_at_100 + value: 57.907 + - type: recall_at_1000 + value: 79.887 + - type: recall_at_20 + value: 38.833 + - type: recall_at_3 + value: 21.397 + - type: recall_at_5 + value: 24.863 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackWordpressRetrieval (default) + revision: 4ffe81d471b1924886b33c7567bfb200e9eec5c4 + split: test + type: mteb/cqadupstack-wordpress + metrics: + - type: main_score + value: 16.794999999999998 + - type: map_at_1 + value: 10.08 + - type: map_at_10 + value: 14.069 + - type: map_at_100 + value: 14.860000000000001 + - type: map_at_1000 + value: 14.968 + - type: map_at_20 + value: 14.46 + - type: map_at_3 + value: 12.498 + - type: map_at_5 + value: 13.324 + - type: mrr_at_1 + value: 10.905730129390019 + - type: mrr_at_10 + value: 15.199146201918854 + - type: mrr_at_100 + value: 16.00264496872985 + - type: mrr_at_1000 + value: 16.09501918722929 + - type: mrr_at_20 + value: 15.633768523540942 + - type: mrr_at_3 + value: 13.493530499075785 + - type: mrr_at_5 + value: 14.36229205175601 + - type: nauc_map_at_1000_diff1 + value: 22.950167181074935 + - type: nauc_map_at_1000_max + value: 18.717980764527866 + - type: nauc_map_at_1000_std + value: -6.25267811740101 + - type: nauc_map_at_100_diff1 + value: 22.94728125565202 + - type: nauc_map_at_100_max + value: 18.719770177431155 + - type: nauc_map_at_100_std + value: -6.323089529332934 + - type: nauc_map_at_10_diff1 + value: 22.346430545898126 + - type: nauc_map_at_10_max + value: 18.80938448630523 + - type: nauc_map_at_10_std + value: -7.0008855212089065 + - type: nauc_map_at_1_diff1 + value: 31.95272198051361 + - type: nauc_map_at_1_max + value: 22.895259623649785 + - type: nauc_map_at_1_std + value: -9.582498979740272 + - type: nauc_map_at_20_diff1 + value: 22.86393142972787 + - type: nauc_map_at_20_max + value: 18.86264577450788 + - type: nauc_map_at_20_std + value: -6.45412214287895 + - type: nauc_map_at_3_diff1 + value: 24.099754234032194 + - type: nauc_map_at_3_max + value: 18.478412248275664 + - type: nauc_map_at_3_std + value: -7.165377931835313 + - type: nauc_map_at_5_diff1 + value: 23.19897817392842 + - type: nauc_map_at_5_max + value: 18.92826540423832 + - type: nauc_map_at_5_std + value: -6.707296227198584 + - type: nauc_mrr_at_1000_diff1 + value: 23.213771617115064 + - type: nauc_mrr_at_1000_max + value: 19.46803843401541 + - type: nauc_mrr_at_1000_std + value: -6.593116817917101 + - type: nauc_mrr_at_100_diff1 + value: 23.231343638867212 + - type: nauc_mrr_at_100_max + value: 19.452575181351783 + - type: nauc_mrr_at_100_std + value: -6.626683471900298 + - type: nauc_mrr_at_10_diff1 + value: 22.605547224050298 + - type: nauc_mrr_at_10_max + value: 19.467230968891098 + - type: nauc_mrr_at_10_std + value: -7.304335909859951 + - type: nauc_mrr_at_1_diff1 + value: 32.21591155654977 + - type: nauc_mrr_at_1_max + value: 23.898168032566968 + - type: nauc_mrr_at_1_std + value: -10.113298227732622 + - type: nauc_mrr_at_20_diff1 + value: 23.17788912060599 + - type: nauc_mrr_at_20_max + value: 19.681138842631395 + - type: nauc_mrr_at_20_std + value: -6.668117181278914 + - type: nauc_mrr_at_3_diff1 + value: 24.324685622276508 + - type: nauc_mrr_at_3_max + value: 19.28094175953585 + - type: nauc_mrr_at_3_std + value: -7.896612175052549 + - type: nauc_mrr_at_5_diff1 + value: 23.56101870977645 + - type: nauc_mrr_at_5_max + value: 19.830915115983956 + - type: nauc_mrr_at_5_std + value: -7.247689969483312 + - type: nauc_ndcg_at_1000_diff1 + value: 21.101486527699198 + - type: nauc_ndcg_at_1000_max + value: 17.661660378409593 + - type: nauc_ndcg_at_1000_std + value: -1.627651235714167 + - type: nauc_ndcg_at_100_diff1 + value: 21.24378422898819 + - type: nauc_ndcg_at_100_max + value: 17.493044854580774 + - type: nauc_ndcg_at_100_std + value: -3.419151472965354 + - type: nauc_ndcg_at_10_diff1 + value: 18.656346406751783 + - type: nauc_ndcg_at_10_max + value: 17.884063161669054 + - type: nauc_ndcg_at_10_std + value: -6.3304637473674985 + - type: nauc_ndcg_at_1_diff1 + value: 32.21591155654977 + - type: nauc_ndcg_at_1_max + value: 23.898168032566968 + - type: nauc_ndcg_at_1_std + value: -10.113298227732622 + - type: nauc_ndcg_at_20_diff1 + value: 20.517191848764295 + - type: nauc_ndcg_at_20_max + value: 18.302766567740257 + - type: nauc_ndcg_at_20_std + value: -4.676348966303663 + - type: nauc_ndcg_at_3_diff1 + value: 22.229860548618376 + - type: nauc_ndcg_at_3_max + value: 17.700425344082685 + - type: nauc_ndcg_at_3_std + value: -6.599851166419227 + - type: nauc_ndcg_at_5_diff1 + value: 20.760917715244236 + - type: nauc_ndcg_at_5_max + value: 18.320361121073617 + - type: nauc_ndcg_at_5_std + value: -5.968352306934327 + - type: nauc_precision_at_1000_diff1 + value: 6.111781725558282 + - type: nauc_precision_at_1000_max + value: 4.893420377600338 + - type: nauc_precision_at_1000_std + value: 13.552656007673166 + - type: nauc_precision_at_100_diff1 + value: 16.174564725391278 + - type: nauc_precision_at_100_max + value: 14.759102996929807 + - type: nauc_precision_at_100_std + value: 6.644858850147021 + - type: nauc_precision_at_10_diff1 + value: 8.889821893924042 + - type: nauc_precision_at_10_max + value: 15.574473888576575 + - type: nauc_precision_at_10_std + value: -2.6115731810417366 + - type: nauc_precision_at_1_diff1 + value: 32.21591155654977 + - type: nauc_precision_at_1_max + value: 23.898168032566968 + - type: nauc_precision_at_1_std + value: -10.113298227732622 + - type: nauc_precision_at_20_diff1 + value: 14.776717379922587 + - type: nauc_precision_at_20_max + value: 19.55219664568408 + - type: nauc_precision_at_20_std + value: 2.8624434397265373 + - type: nauc_precision_at_3_diff1 + value: 17.24181833195652 + - type: nauc_precision_at_3_max + value: 15.310985601785825 + - type: nauc_precision_at_3_std + value: -5.815145792096017 + - type: nauc_precision_at_5_diff1 + value: 14.568702652383378 + - type: nauc_precision_at_5_max + value: 16.90398092807837 + - type: nauc_precision_at_5_std + value: -4.884555559489991 + - type: nauc_recall_at_1000_diff1 + value: 17.718608305964434 + - type: nauc_recall_at_1000_max + value: 13.402668234081721 + - type: nauc_recall_at_1000_std + value: 21.623779371422756 + - type: nauc_recall_at_100_diff1 + value: 18.932841874380454 + - type: nauc_recall_at_100_max + value: 13.254799775623564 + - type: nauc_recall_at_100_std + value: 4.592397886568707 + - type: nauc_recall_at_10_diff1 + value: 10.256753131266485 + - type: nauc_recall_at_10_max + value: 15.34274332609289 + - type: nauc_recall_at_10_std + value: -5.019100394026518 + - type: nauc_recall_at_1_diff1 + value: 31.95272198051361 + - type: nauc_recall_at_1_max + value: 22.895259623649785 + - type: nauc_recall_at_1_std + value: -9.582498979740272 + - type: nauc_recall_at_20_diff1 + value: 16.098225999062155 + - type: nauc_recall_at_20_max + value: 16.11919310391389 + - type: nauc_recall_at_20_std + value: -0.981856820033547 + - type: nauc_recall_at_3_diff1 + value: 16.896414167717293 + - type: nauc_recall_at_3_max + value: 14.67655178851271 + - type: nauc_recall_at_3_std + value: -4.885403738918622 + - type: nauc_recall_at_5_diff1 + value: 15.074392597620905 + - type: nauc_recall_at_5_max + value: 16.457162195748644 + - type: nauc_recall_at_5_std + value: -3.6534367499331046 + - type: ndcg_at_1 + value: 10.906 + - type: ndcg_at_10 + value: 16.794999999999998 + - type: ndcg_at_100 + value: 21.434 + - type: ndcg_at_1000 + value: 24.743000000000002 + - type: ndcg_at_20 + value: 18.275 + - type: ndcg_at_3 + value: 13.507 + - type: ndcg_at_5 + value: 14.953 + - type: precision_at_1 + value: 10.906 + - type: precision_at_10 + value: 2.791 + - type: precision_at_100 + value: 0.5559999999999999 + - type: precision_at_1000 + value: 0.091 + - type: precision_at_20 + value: 1.738 + - type: precision_at_3 + value: 5.545 + - type: precision_at_5 + value: 4.14 + - type: recall_at_1 + value: 10.08 + - type: recall_at_10 + value: 24.184 + - type: recall_at_100 + value: 46.967999999999996 + - type: recall_at_1000 + value: 72.92999999999999 + - type: recall_at_20 + value: 29.852 + - type: recall_at_3 + value: 15.440999999999999 + - type: recall_at_5 + value: 18.829 + task: + type: Retrieval + - dataset: + config: default + name: MTEB ClimateFEVER (default) + revision: 47f2ac6acb640fc46020b02a5b59fdda04d39380 + split: test + type: mteb/climate-fever + metrics: + - type: main_score + value: 17.288999999999998 + - type: map_at_1 + value: 6.537 + - type: map_at_10 + value: 11.465 + - type: map_at_100 + value: 12.851 + - type: map_at_1000 + value: 13.045000000000002 + - type: map_at_20 + value: 12.174 + - type: map_at_3 + value: 9.369 + - type: map_at_5 + value: 10.331 + - type: mrr_at_1 + value: 15.2442996742671 + - type: mrr_at_10 + value: 23.59306654257793 + - type: mrr_at_100 + value: 24.771529453769823 + - type: mrr_at_1000 + value: 24.838895119526256 + - type: mrr_at_20 + value: 24.34915881726873 + - type: mrr_at_3 + value: 20.466883821932676 + - type: mrr_at_5 + value: 22.027144408251875 + - type: nauc_map_at_1000_diff1 + value: 21.34422077879759 + - type: nauc_map_at_1000_max + value: 22.628208123980382 + - type: nauc_map_at_1000_std + value: 15.80771024789922 + - type: nauc_map_at_100_diff1 + value: 21.373352148960333 + - type: nauc_map_at_100_max + value: 22.445247482460697 + - type: nauc_map_at_100_std + value: 15.551345921669244 + - type: nauc_map_at_10_diff1 + value: 22.093245216727393 + - type: nauc_map_at_10_max + value: 20.71848879842843 + - type: nauc_map_at_10_std + value: 13.073037988129768 + - type: nauc_map_at_1_diff1 + value: 32.56507685691908 + - type: nauc_map_at_1_max + value: 19.299512363814912 + - type: nauc_map_at_1_std + value: 7.980883065948159 + - type: nauc_map_at_20_diff1 + value: 21.612469499988222 + - type: nauc_map_at_20_max + value: 21.70315933461587 + - type: nauc_map_at_20_std + value: 14.51324386963804 + - type: nauc_map_at_3_diff1 + value: 22.671417020380986 + - type: nauc_map_at_3_max + value: 18.10374651349345 + - type: nauc_map_at_3_std + value: 9.73448791948781 + - type: nauc_map_at_5_diff1 + value: 22.034988196838064 + - type: nauc_map_at_5_max + value: 18.490696961140145 + - type: nauc_map_at_5_std + value: 11.001958112977931 + - type: nauc_mrr_at_1000_diff1 + value: 17.997877765827052 + - type: nauc_mrr_at_1000_max + value: 23.761191320854795 + - type: nauc_mrr_at_1000_std + value: 17.086288520129283 + - type: nauc_mrr_at_100_diff1 + value: 17.99589491236679 + - type: nauc_mrr_at_100_max + value: 23.76386777696214 + - type: nauc_mrr_at_100_std + value: 17.114923252433908 + - type: nauc_mrr_at_10_diff1 + value: 17.95028052166577 + - type: nauc_mrr_at_10_max + value: 23.313446785613046 + - type: nauc_mrr_at_10_std + value: 16.289313792057893 + - type: nauc_mrr_at_1_diff1 + value: 25.00794012521374 + - type: nauc_mrr_at_1_max + value: 20.934023514536086 + - type: nauc_mrr_at_1_std + value: 10.326842252115775 + - type: nauc_mrr_at_20_diff1 + value: 17.977173189525192 + - type: nauc_mrr_at_20_max + value: 23.858084437038833 + - type: nauc_mrr_at_20_std + value: 17.177629596269224 + - type: nauc_mrr_at_3_diff1 + value: 18.049118818264052 + - type: nauc_mrr_at_3_max + value: 21.812245650122605 + - type: nauc_mrr_at_3_std + value: 14.048078149579718 + - type: nauc_mrr_at_5_diff1 + value: 18.028877069283745 + - type: nauc_mrr_at_5_max + value: 21.88620019054395 + - type: nauc_mrr_at_5_std + value: 14.787661645971001 + - type: nauc_ndcg_at_1000_diff1 + value: 16.72726980659064 + - type: nauc_ndcg_at_1000_max + value: 30.043672363788087 + - type: nauc_ndcg_at_1000_std + value: 26.833584730455268 + - type: nauc_ndcg_at_100_diff1 + value: 17.16473243031922 + - type: nauc_ndcg_at_100_max + value: 28.239622016125566 + - type: nauc_ndcg_at_100_std + value: 24.469002695895977 + - type: nauc_ndcg_at_10_diff1 + value: 18.655890597433427 + - type: nauc_ndcg_at_10_max + value: 23.63136724071696 + - type: nauc_ndcg_at_10_std + value: 17.29295589103389 + - type: nauc_ndcg_at_1_diff1 + value: 25.00794012521374 + - type: nauc_ndcg_at_1_max + value: 20.934023514536086 + - type: nauc_ndcg_at_1_std + value: 10.326842252115775 + - type: nauc_ndcg_at_20_diff1 + value: 17.762757204969244 + - type: nauc_ndcg_at_20_max + value: 25.946755000541476 + - type: nauc_ndcg_at_20_std + value: 20.9523075152757 + - type: nauc_ndcg_at_3_diff1 + value: 18.258615831392746 + - type: nauc_ndcg_at_3_max + value: 20.21498568651181 + - type: nauc_ndcg_at_3_std + value: 12.588112301185989 + - type: nauc_ndcg_at_5_diff1 + value: 18.575198873459577 + - type: nauc_ndcg_at_5_max + value: 19.821485190942443 + - type: nauc_ndcg_at_5_std + value: 13.559611377687455 + - type: nauc_precision_at_1000_diff1 + value: -1.3591333339360123 + - type: nauc_precision_at_1000_max + value: 33.01866225202323 + - type: nauc_precision_at_1000_std + value: 38.26072433720804 + - type: nauc_precision_at_100_diff1 + value: 4.534183759090849 + - type: nauc_precision_at_100_max + value: 35.499433595656335 + - type: nauc_precision_at_100_std + value: 37.765227934597114 + - type: nauc_precision_at_10_diff1 + value: 11.369511250136568 + - type: nauc_precision_at_10_max + value: 30.281092515358527 + - type: nauc_precision_at_10_std + value: 26.690470077530847 + - type: nauc_precision_at_1_diff1 + value: 25.00794012521374 + - type: nauc_precision_at_1_max + value: 20.934023514536086 + - type: nauc_precision_at_1_std + value: 10.326842252115775 + - type: nauc_precision_at_20_diff1 + value: 8.133211694372351 + - type: nauc_precision_at_20_max + value: 34.161055315782775 + - type: nauc_precision_at_20_std + value: 33.33055010570849 + - type: nauc_precision_at_3_diff1 + value: 10.5682193001728 + - type: nauc_precision_at_3_max + value: 22.786982248944767 + - type: nauc_precision_at_3_std + value: 17.92766896610086 + - type: nauc_precision_at_5_diff1 + value: 10.940535871177055 + - type: nauc_precision_at_5_max + value: 23.197073410356037 + - type: nauc_precision_at_5_std + value: 20.612896217277573 + - type: nauc_recall_at_1000_diff1 + value: 5.540983045337761 + - type: nauc_recall_at_1000_max + value: 37.3394645787145 + - type: nauc_recall_at_1000_std + value: 43.905340993951555 + - type: nauc_recall_at_100_diff1 + value: 8.725053205627061 + - type: nauc_recall_at_100_max + value: 29.46589116376182 + - type: nauc_recall_at_100_std + value: 32.76739728784572 + - type: nauc_recall_at_10_diff1 + value: 13.519133005869758 + - type: nauc_recall_at_10_max + value: 23.66746585259265 + - type: nauc_recall_at_10_std + value: 19.744857128981092 + - type: nauc_recall_at_1_diff1 + value: 32.56507685691908 + - type: nauc_recall_at_1_max + value: 19.299512363814912 + - type: nauc_recall_at_1_std + value: 7.980883065948159 + - type: nauc_recall_at_20_diff1 + value: 10.866077600352101 + - type: nauc_recall_at_20_max + value: 26.726876720649262 + - type: nauc_recall_at_20_std + value: 26.28100368153264 + - type: nauc_recall_at_3_diff1 + value: 15.295338383488533 + - type: nauc_recall_at_3_max + value: 18.013167170259173 + - type: nauc_recall_at_3_std + value: 11.569701886642754 + - type: nauc_recall_at_5_diff1 + value: 14.214598780846863 + - type: nauc_recall_at_5_max + value: 17.96550333772466 + - type: nauc_recall_at_5_std + value: 13.720834673116972 + - type: ndcg_at_1 + value: 15.244 + - type: ndcg_at_10 + value: 17.288999999999998 + - type: ndcg_at_100 + value: 23.757 + - type: ndcg_at_1000 + value: 27.725 + - type: ndcg_at_20 + value: 19.686999999999998 + - type: ndcg_at_3 + value: 13.245000000000001 + - type: ndcg_at_5 + value: 14.485000000000001 + - type: precision_at_1 + value: 15.244 + - type: precision_at_10 + value: 5.733 + - type: precision_at_100 + value: 1.264 + - type: precision_at_1000 + value: 0.199 + - type: precision_at_20 + value: 3.85 + - type: precision_at_3 + value: 10.054 + - type: precision_at_5 + value: 7.9350000000000005 + - type: recall_at_1 + value: 6.537 + - type: recall_at_10 + value: 22.046 + - type: recall_at_100 + value: 44.818000000000005 + - type: recall_at_1000 + value: 67.676 + - type: recall_at_20 + value: 28.974 + - type: recall_at_3 + value: 12.232 + - type: recall_at_5 + value: 15.540999999999999 + task: + type: Retrieval + - dataset: + config: default + name: MTEB DBPedia (default) + revision: c0f706b76e590d620bd6618b3ca8efdd34e2d659 + split: test + type: mteb/dbpedia + metrics: + - type: main_score + value: 24.235 + - type: map_at_1 + value: 4.304 + - type: map_at_10 + value: 9.944 + - type: map_at_100 + value: 14.113000000000001 + - type: map_at_1000 + value: 15.085 + - type: map_at_20 + value: 11.594 + - type: map_at_3 + value: 7.228999999999999 + - type: map_at_5 + value: 8.368 + - type: mrr_at_1 + value: 43.0 + - type: mrr_at_10 + value: 53.30376984126983 + - type: mrr_at_100 + value: 53.97910163622114 + - type: mrr_at_1000 + value: 54.005267473599304 + - type: mrr_at_20 + value: 53.740161512249365 + - type: mrr_at_3 + value: 50.54166666666667 + - type: mrr_at_5 + value: 52.154166666666654 + - type: nauc_map_at_1000_diff1 + value: 26.809585057496545 + - type: nauc_map_at_1000_max + value: 27.599866660752987 + - type: nauc_map_at_1000_std + value: 31.459439584000094 + - type: nauc_map_at_100_diff1 + value: 27.049487336011836 + - type: nauc_map_at_100_max + value: 25.112936840752 + - type: nauc_map_at_100_std + value: 28.400137100413364 + - type: nauc_map_at_10_diff1 + value: 32.105246040146554 + - type: nauc_map_at_10_max + value: 9.658311385867774 + - type: nauc_map_at_10_std + value: 12.006591313970928 + - type: nauc_map_at_1_diff1 + value: 45.66826032911575 + - type: nauc_map_at_1_max + value: 1.1005171486965344 + - type: nauc_map_at_1_std + value: 3.2500050585955558 + - type: nauc_map_at_20_diff1 + value: 30.73734552740125 + - type: nauc_map_at_20_max + value: 14.994971393610829 + - type: nauc_map_at_20_std + value: 18.029603402042753 + - type: nauc_map_at_3_diff1 + value: 36.77585294977933 + - type: nauc_map_at_3_max + value: 2.0123666749907034 + - type: nauc_map_at_3_std + value: 3.1886056493854906 + - type: nauc_map_at_5_diff1 + value: 34.910885252980414 + - type: nauc_map_at_5_max + value: 4.606898880177816 + - type: nauc_map_at_5_std + value: 5.897472990222533 + - type: nauc_mrr_at_1000_diff1 + value: 32.8408203164654 + - type: nauc_mrr_at_1000_max + value: 44.57916824429895 + - type: nauc_mrr_at_1000_std + value: 25.76632603800019 + - type: nauc_mrr_at_100_diff1 + value: 32.83381181877902 + - type: nauc_mrr_at_100_max + value: 44.57742098993615 + - type: nauc_mrr_at_100_std + value: 25.763980866882193 + - type: nauc_mrr_at_10_diff1 + value: 32.85879447148161 + - type: nauc_mrr_at_10_max + value: 44.587973042043814 + - type: nauc_mrr_at_10_std + value: 25.548766798683893 + - type: nauc_mrr_at_1_diff1 + value: 36.064038704139605 + - type: nauc_mrr_at_1_max + value: 43.188409566789346 + - type: nauc_mrr_at_1_std + value: 24.26421817898062 + - type: nauc_mrr_at_20_diff1 + value: 32.752896264184685 + - type: nauc_mrr_at_20_max + value: 44.56787283356919 + - type: nauc_mrr_at_20_std + value: 25.763763879915313 + - type: nauc_mrr_at_3_diff1 + value: 33.265925003418126 + - type: nauc_mrr_at_3_max + value: 43.98236209085194 + - type: nauc_mrr_at_3_std + value: 24.811433062956347 + - type: nauc_mrr_at_5_diff1 + value: 33.02692454410134 + - type: nauc_mrr_at_5_max + value: 44.02150946107612 + - type: nauc_mrr_at_5_std + value: 24.414392179240878 + - type: nauc_ndcg_at_1000_diff1 + value: 29.071114816059023 + - type: nauc_ndcg_at_1000_max + value: 38.90222092060964 + - type: nauc_ndcg_at_1000_std + value: 44.44820451621514 + - type: nauc_ndcg_at_100_diff1 + value: 29.1316364198098 + - type: nauc_ndcg_at_100_max + value: 31.558894971415064 + - type: nauc_ndcg_at_100_std + value: 35.45395514581182 + - type: nauc_ndcg_at_10_diff1 + value: 29.303783217647744 + - type: nauc_ndcg_at_10_max + value: 31.009718153863414 + - type: nauc_ndcg_at_10_std + value: 27.49477754545124 + - type: nauc_ndcg_at_1_diff1 + value: 35.43480922848642 + - type: nauc_ndcg_at_1_max + value: 30.475722281046714 + - type: nauc_ndcg_at_1_std + value: 17.626646786380547 + - type: nauc_ndcg_at_20_diff1 + value: 29.30769894815147 + - type: nauc_ndcg_at_20_max + value: 27.870710525324107 + - type: nauc_ndcg_at_20_std + value: 28.334513734492532 + - type: nauc_ndcg_at_3_diff1 + value: 30.7536730308035 + - type: nauc_ndcg_at_3_max + value: 32.32457811814772 + - type: nauc_ndcg_at_3_std + value: 21.676427426548152 + - type: nauc_ndcg_at_5_diff1 + value: 29.96943892323901 + - type: nauc_ndcg_at_5_max + value: 31.493512707920964 + - type: nauc_ndcg_at_5_std + value: 24.0956693770445 + - type: nauc_precision_at_1000_diff1 + value: -5.720318672455256 + - type: nauc_precision_at_1000_max + value: 28.08646209634404 + - type: nauc_precision_at_1000_std + value: 29.34422238786186 + - type: nauc_precision_at_100_diff1 + value: 0.84607162708279 + - type: nauc_precision_at_100_max + value: 47.97391409332498 + - type: nauc_precision_at_100_std + value: 44.619521382937286 + - type: nauc_precision_at_10_diff1 + value: 9.622029967680373 + - type: nauc_precision_at_10_max + value: 45.89203900455004 + - type: nauc_precision_at_10_std + value: 38.276273021326745 + - type: nauc_precision_at_1_diff1 + value: 36.064038704139605 + - type: nauc_precision_at_1_max + value: 43.188409566789346 + - type: nauc_precision_at_1_std + value: 24.26421817898062 + - type: nauc_precision_at_20_diff1 + value: 6.709711811715244 + - type: nauc_precision_at_20_max + value: 47.47318907005896 + - type: nauc_precision_at_20_std + value: 42.595576770275095 + - type: nauc_precision_at_3_diff1 + value: 19.233575308317054 + - type: nauc_precision_at_3_max + value: 43.02563765159987 + - type: nauc_precision_at_3_std + value: 27.334254446564454 + - type: nauc_precision_at_5_diff1 + value: 14.298477498830673 + - type: nauc_precision_at_5_max + value: 42.72631241492758 + - type: nauc_precision_at_5_std + value: 32.14763584000337 + - type: nauc_recall_at_1000_diff1 + value: 18.551929022070503 + - type: nauc_recall_at_1000_max + value: 25.99572596347025 + - type: nauc_recall_at_1000_std + value: 49.479321187111644 + - type: nauc_recall_at_100_diff1 + value: 16.24655246342188 + - type: nauc_recall_at_100_max + value: 19.193014693852824 + - type: nauc_recall_at_100_std + value: 31.691642773148754 + - type: nauc_recall_at_10_diff1 + value: 21.181166055890365 + - type: nauc_recall_at_10_max + value: -0.020533885799737757 + - type: nauc_recall_at_10_std + value: 7.266191592314226 + - type: nauc_recall_at_1_diff1 + value: 45.66826032911575 + - type: nauc_recall_at_1_max + value: 1.1005171486965344 + - type: nauc_recall_at_1_std + value: 3.2500050585955558 + - type: nauc_recall_at_20_diff1 + value: 19.153797037751836 + - type: nauc_recall_at_20_max + value: 3.9385573002743057 + - type: nauc_recall_at_20_std + value: 14.048512138776442 + - type: nauc_recall_at_3_diff1 + value: 30.240078354763085 + - type: nauc_recall_at_3_max + value: -4.0841121814480195 + - type: nauc_recall_at_3_std + value: -2.3759344889809264 + - type: nauc_recall_at_5_diff1 + value: 26.22489817092464 + - type: nauc_recall_at_5_max + value: -3.2396073154699256 + - type: nauc_recall_at_5_std + value: -0.1327990827712389 + - type: ndcg_at_1 + value: 31.5 + - type: ndcg_at_10 + value: 24.235 + - type: ndcg_at_100 + value: 28.01 + - type: ndcg_at_1000 + value: 34.724 + - type: ndcg_at_20 + value: 24.265 + - type: ndcg_at_3 + value: 26.682 + - type: ndcg_at_5 + value: 25.249 + - type: precision_at_1 + value: 43.0 + - type: precision_at_10 + value: 21.65 + - type: precision_at_100 + value: 6.97 + - type: precision_at_1000 + value: 1.4449999999999998 + - type: precision_at_20 + value: 16.6 + - type: precision_at_3 + value: 32.25 + - type: precision_at_5 + value: 27.250000000000004 + - type: recall_at_1 + value: 4.304 + - type: recall_at_10 + value: 15.014 + - type: recall_at_100 + value: 35.115 + - type: recall_at_1000 + value: 58.52 + - type: recall_at_20 + value: 20.817 + - type: recall_at_3 + value: 8.698 + - type: recall_at_5 + value: 11.052 + task: + type: Retrieval + - dataset: + config: default + name: MTEB EmotionClassification (default) + revision: 4f58c6b202a23cf9a4da393831edf4f9183cad37 + split: test + type: mteb/emotion + metrics: + - type: accuracy + value: 45.09 + - type: f1 + value: 41.3731018097549 + - type: f1_weighted + value: 47.129694558751545 + - type: main_score + value: 45.09 + task: + type: Classification + - dataset: + config: default + name: MTEB FEVER (default) + revision: bea83ef9e8fb933d90a2f1d5515737465d613e12 + split: test + type: mteb/fever + metrics: + - type: main_score + value: 30.267 + - type: map_at_1 + value: 16.349 + - type: map_at_10 + value: 24.917 + - type: map_at_100 + value: 26.003 + - type: map_at_1000 + value: 26.072 + - type: map_at_20 + value: 25.558999999999997 + - type: map_at_3 + value: 22.067999999999998 + - type: map_at_5 + value: 23.610999999999997 + - type: mrr_at_1 + value: 17.416741674167415 + - type: mrr_at_10 + value: 26.439929707256365 + - type: mrr_at_100 + value: 27.508820939687954 + - type: mrr_at_1000 + value: 27.570352489203128 + - type: mrr_at_20 + value: 27.08319436248233 + - type: mrr_at_3 + value: 23.422342234223358 + - type: mrr_at_5 + value: 25.06350635063509 + - type: nauc_map_at_1000_diff1 + value: 21.773223671090857 + - type: nauc_map_at_1000_max + value: 6.412897130218669 + - type: nauc_map_at_1000_std + value: -6.3221009008493745 + - type: nauc_map_at_100_diff1 + value: 21.76483868507978 + - type: nauc_map_at_100_max + value: 6.404365200549758 + - type: nauc_map_at_100_std + value: -6.342840969370927 + - type: nauc_map_at_10_diff1 + value: 21.669481996014238 + - type: nauc_map_at_10_max + value: 6.019531738681224 + - type: nauc_map_at_10_std + value: -6.941777440293395 + - type: nauc_map_at_1_diff1 + value: 27.706382248361393 + - type: nauc_map_at_1_max + value: 4.030610814398596 + - type: nauc_map_at_1_std + value: -9.782554832619702 + - type: nauc_map_at_20_diff1 + value: 21.80535156700929 + - type: nauc_map_at_20_max + value: 6.361714278006344 + - type: nauc_map_at_20_std + value: -6.513790702798104 + - type: nauc_map_at_3_diff1 + value: 23.017059605983857 + - type: nauc_map_at_3_max + value: 5.110304244032051 + - type: nauc_map_at_3_std + value: -8.069547854658104 + - type: nauc_map_at_5_diff1 + value: 21.927491204194766 + - type: nauc_map_at_5_max + value: 5.462525780765053 + - type: nauc_map_at_5_std + value: -7.474340804858998 + - type: nauc_mrr_at_1000_diff1 + value: 21.61235920652557 + - type: nauc_mrr_at_1000_max + value: 6.6996553488043915 + - type: nauc_mrr_at_1000_std + value: -6.520954496784069 + - type: nauc_mrr_at_100_diff1 + value: 21.597831485534126 + - type: nauc_mrr_at_100_max + value: 6.705135295195408 + - type: nauc_mrr_at_100_std + value: -6.521597409657566 + - type: nauc_mrr_at_10_diff1 + value: 21.404259600861597 + - type: nauc_mrr_at_10_max + value: 6.348078634441438 + - type: nauc_mrr_at_10_std + value: -7.012906818443071 + - type: nauc_mrr_at_1_diff1 + value: 27.231264207663248 + - type: nauc_mrr_at_1_max + value: 4.04888129901842 + - type: nauc_mrr_at_1_std + value: -9.998368133129015 + - type: nauc_mrr_at_20_diff1 + value: 21.57543681953314 + - type: nauc_mrr_at_20_max + value: 6.670007051575425 + - type: nauc_mrr_at_20_std + value: -6.636382948186316 + - type: nauc_mrr_at_3_diff1 + value: 22.771758514181627 + - type: nauc_mrr_at_3_max + value: 5.389600538667887 + - type: nauc_mrr_at_3_std + value: -8.189661361743667 + - type: nauc_mrr_at_5_diff1 + value: 21.689397986510446 + - type: nauc_mrr_at_5_max + value: 5.765658649049543 + - type: nauc_mrr_at_5_std + value: -7.590205788150704 + - type: nauc_ndcg_at_1000_diff1 + value: 19.780729881850963 + - type: nauc_ndcg_at_1000_max + value: 8.968522119658385 + - type: nauc_ndcg_at_1000_std + value: -2.425269449284083 + - type: nauc_ndcg_at_100_diff1 + value: 19.46657224380776 + - type: nauc_ndcg_at_100_max + value: 9.05883201318058 + - type: nauc_ndcg_at_100_std + value: -2.5565659351523293 + - type: nauc_ndcg_at_10_diff1 + value: 19.29152253186839 + - type: nauc_ndcg_at_10_max + value: 7.499062048205841 + - type: nauc_ndcg_at_10_std + value: -5.2482566392088685 + - type: nauc_ndcg_at_1_diff1 + value: 27.231264207663248 + - type: nauc_ndcg_at_1_max + value: 4.04888129901842 + - type: nauc_ndcg_at_1_std + value: -9.998368133129015 + - type: nauc_ndcg_at_20_diff1 + value: 19.71545443537324 + - type: nauc_ndcg_at_20_max + value: 8.64504551388718 + - type: nauc_ndcg_at_20_std + value: -3.7667113417348976 + - type: nauc_ndcg_at_3_diff1 + value: 21.745216173844803 + - type: nauc_ndcg_at_3_max + value: 5.650727598972489 + - type: nauc_ndcg_at_3_std + value: -7.481336986244201 + - type: nauc_ndcg_at_5_diff1 + value: 19.936133837204203 + - type: nauc_ndcg_at_5_max + value: 6.259916537058443 + - type: nauc_ndcg_at_5_std + value: -6.484388158971839 + - type: nauc_precision_at_1000_diff1 + value: 1.471146535072958 + - type: nauc_precision_at_1000_max + value: 20.630906784097483 + - type: nauc_precision_at_1000_std + value: 21.9773366010731 + - type: nauc_precision_at_100_diff1 + value: 7.533964401054148 + - type: nauc_precision_at_100_max + value: 19.925643661900423 + - type: nauc_precision_at_100_std + value: 15.336729247195924 + - type: nauc_precision_at_10_diff1 + value: 12.150440335935734 + - type: nauc_precision_at_10_max + value: 11.983854268540387 + - type: nauc_precision_at_10_std + value: -0.37221151434129196 + - type: nauc_precision_at_1_diff1 + value: 27.231264207663248 + - type: nauc_precision_at_1_max + value: 4.04888129901842 + - type: nauc_precision_at_1_std + value: -9.998368133129015 + - type: nauc_precision_at_20_diff1 + value: 12.630450311503752 + - type: nauc_precision_at_20_max + value: 16.05605149278296 + - type: nauc_precision_at_20_std + value: 5.3999355877921165 + - type: nauc_precision_at_3_diff1 + value: 18.359563527526568 + - type: nauc_precision_at_3_max + value: 7.050702808245418 + - type: nauc_precision_at_3_std + value: -6.012052050420314 + - type: nauc_precision_at_5_diff1 + value: 14.398743831406193 + - type: nauc_precision_at_5_max + value: 8.47645601614165 + - type: nauc_precision_at_5_std + value: -4.017240645221931 + - type: nauc_recall_at_1000_diff1 + value: 7.839541590866944 + - type: nauc_recall_at_1000_max + value: 23.309619602703478 + - type: nauc_recall_at_1000_std + value: 27.804864458508405 + - type: nauc_recall_at_100_diff1 + value: 9.97691215791031 + - type: nauc_recall_at_100_max + value: 18.819153599870717 + - type: nauc_recall_at_100_std + value: 14.458117071228108 + - type: nauc_recall_at_10_diff1 + value: 12.810432997078946 + - type: nauc_recall_at_10_max + value: 10.766544057766287 + - type: nauc_recall_at_10_std + value: -0.5969028921503585 + - type: nauc_recall_at_1_diff1 + value: 27.706382248361393 + - type: nauc_recall_at_1_max + value: 4.030610814398596 + - type: nauc_recall_at_1_std + value: -9.782554832619702 + - type: nauc_recall_at_20_diff1 + value: 13.595110328407126 + - type: nauc_recall_at_20_max + value: 14.757809231376443 + - type: nauc_recall_at_20_std + value: 4.9020894617594575 + - type: nauc_recall_at_3_diff1 + value: 18.603105066886183 + - type: nauc_recall_at_3_max + value: 6.695351132956627 + - type: nauc_recall_at_3_std + value: -5.761401766506087 + - type: nauc_recall_at_5_diff1 + value: 14.770731919705574 + - type: nauc_recall_at_5_max + value: 7.754748009508286 + - type: nauc_recall_at_5_std + value: -3.7961358195332773 + - type: ndcg_at_1 + value: 17.416999999999998 + - type: ndcg_at_10 + value: 30.267 + - type: ndcg_at_100 + value: 35.650999999999996 + - type: ndcg_at_1000 + value: 37.57 + - type: ndcg_at_20 + value: 32.574 + - type: ndcg_at_3 + value: 24.303 + - type: ndcg_at_5 + value: 27.099 + - type: precision_at_1 + value: 17.416999999999998 + - type: precision_at_10 + value: 4.9590000000000005 + - type: precision_at_100 + value: 0.7799999999999999 + - type: precision_at_1000 + value: 0.096 + - type: precision_at_20 + value: 2.9819999999999998 + - type: precision_at_3 + value: 10.536 + - type: precision_at_5 + value: 7.807 + - type: recall_at_1 + value: 16.349 + - type: recall_at_10 + value: 45.678999999999995 + - type: recall_at_100 + value: 70.541 + - type: recall_at_1000 + value: 85.36500000000001 + - type: recall_at_20 + value: 54.541 + - type: recall_at_3 + value: 29.42 + - type: recall_at_5 + value: 36.112 + task: + type: Retrieval + - dataset: + config: default + name: MTEB FiQA2018 (default) + revision: 27a168819829fe9bcd655c2df245fb19452e8e06 + split: test + type: mteb/fiqa + metrics: + - type: main_score + value: 16.619 + - type: map_at_1 + value: 7.478999999999999 + - type: map_at_10 + value: 11.933 + - type: map_at_100 + value: 13.078000000000001 + - type: map_at_1000 + value: 13.267999999999999 + - type: map_at_20 + value: 12.465 + - type: map_at_3 + value: 9.975000000000001 + - type: map_at_5 + value: 10.928 + - type: mrr_at_1 + value: 14.660493827160495 + - type: mrr_at_10 + value: 20.737250146972368 + - type: mrr_at_100 + value: 21.718558761167632 + - type: mrr_at_1000 + value: 21.808600465854973 + - type: mrr_at_20 + value: 21.221196101889976 + - type: mrr_at_3 + value: 18.569958847736622 + - type: mrr_at_5 + value: 19.557613168724284 + - type: nauc_map_at_1000_diff1 + value: 21.51431734644358 + - type: nauc_map_at_1000_max + value: 4.931074809601008 + - type: nauc_map_at_1000_std + value: -3.3303160557020033 + - type: nauc_map_at_100_diff1 + value: 21.38249575770264 + - type: nauc_map_at_100_max + value: 4.725930298940441 + - type: nauc_map_at_100_std + value: -3.4448477852279473 + - type: nauc_map_at_10_diff1 + value: 21.195172969735484 + - type: nauc_map_at_10_max + value: 4.412691847045547 + - type: nauc_map_at_10_std + value: -4.350074377307911 + - type: nauc_map_at_1_diff1 + value: 28.103238263092063 + - type: nauc_map_at_1_max + value: 6.669837188399256 + - type: nauc_map_at_1_std + value: -4.3658897905036405 + - type: nauc_map_at_20_diff1 + value: 21.489132375885042 + - type: nauc_map_at_20_max + value: 4.303022314751493 + - type: nauc_map_at_20_std + value: -4.17992541434375 + - type: nauc_map_at_3_diff1 + value: 22.237087711122065 + - type: nauc_map_at_3_max + value: 4.533442194144081 + - type: nauc_map_at_3_std + value: -5.4916480142821635 + - type: nauc_map_at_5_diff1 + value: 21.876772694300065 + - type: nauc_map_at_5_max + value: 4.511112176374985 + - type: nauc_map_at_5_std + value: -5.176150118472554 + - type: nauc_mrr_at_1000_diff1 + value: 22.783625924297894 + - type: nauc_mrr_at_1000_max + value: 5.601679998803955 + - type: nauc_mrr_at_1000_std + value: -7.3878080622090865 + - type: nauc_mrr_at_100_diff1 + value: 22.729460521696915 + - type: nauc_mrr_at_100_max + value: 5.57805664833725 + - type: nauc_mrr_at_100_std + value: -7.3741470356357945 + - type: nauc_mrr_at_10_diff1 + value: 22.92977199129734 + - type: nauc_mrr_at_10_max + value: 5.36088601159652 + - type: nauc_mrr_at_10_std + value: -7.875413563795927 + - type: nauc_mrr_at_1_diff1 + value: 28.31095482042955 + - type: nauc_mrr_at_1_max + value: 7.815000197077026 + - type: nauc_mrr_at_1_std + value: -7.957538731368522 + - type: nauc_mrr_at_20_diff1 + value: 22.946584920142406 + - type: nauc_mrr_at_20_max + value: 5.384498887828733 + - type: nauc_mrr_at_20_std + value: -7.633579657779428 + - type: nauc_mrr_at_3_diff1 + value: 23.46361356498147 + - type: nauc_mrr_at_3_max + value: 4.50117125788086 + - type: nauc_mrr_at_3_std + value: -8.902224452227653 + - type: nauc_mrr_at_5_diff1 + value: 23.331352654582094 + - type: nauc_mrr_at_5_max + value: 4.978873752458006 + - type: nauc_mrr_at_5_std + value: -8.93749978655238 + - type: nauc_ndcg_at_1000_diff1 + value: 19.87039469365751 + - type: nauc_ndcg_at_1000_max + value: 8.696714614408632 + - type: nauc_ndcg_at_1000_std + value: 1.9681923697039077 + - type: nauc_ndcg_at_100_diff1 + value: 18.868322837780532 + - type: nauc_ndcg_at_100_max + value: 6.0333062132177675 + - type: nauc_ndcg_at_100_std + value: 0.44045929715801535 + - type: nauc_ndcg_at_10_diff1 + value: 19.727068370792786 + - type: nauc_ndcg_at_10_max + value: 4.277512828410901 + - type: nauc_ndcg_at_10_std + value: -4.086859790177703 + - type: nauc_ndcg_at_1_diff1 + value: 28.31095482042955 + - type: nauc_ndcg_at_1_max + value: 7.815000197077026 + - type: nauc_ndcg_at_1_std + value: -7.957538731368522 + - type: nauc_ndcg_at_20_diff1 + value: 20.29147215834196 + - type: nauc_ndcg_at_20_max + value: 4.095649235859702 + - type: nauc_ndcg_at_20_std + value: -3.35870597862009 + - type: nauc_ndcg_at_3_diff1 + value: 21.821928240162936 + - type: nauc_ndcg_at_3_max + value: 4.480256449572136 + - type: nauc_ndcg_at_3_std + value: -7.852741840584263 + - type: nauc_ndcg_at_5_diff1 + value: 21.15156996884851 + - type: nauc_ndcg_at_5_max + value: 4.290200639355712 + - type: nauc_ndcg_at_5_std + value: -6.820305338379054 + - type: nauc_precision_at_1000_diff1 + value: 8.075302805866599 + - type: nauc_precision_at_1000_max + value: 19.944406193476624 + - type: nauc_precision_at_1000_std + value: 7.381890177301082 + - type: nauc_precision_at_100_diff1 + value: 11.601078456057651 + - type: nauc_precision_at_100_max + value: 13.628171798745194 + - type: nauc_precision_at_100_std + value: 5.64401780985023 + - type: nauc_precision_at_10_diff1 + value: 16.653551040271243 + - type: nauc_precision_at_10_max + value: 6.546264597330201 + - type: nauc_precision_at_10_std + value: -4.71713361654603 + - type: nauc_precision_at_1_diff1 + value: 28.31095482042955 + - type: nauc_precision_at_1_max + value: 7.815000197077026 + - type: nauc_precision_at_1_std + value: -7.957538731368522 + - type: nauc_precision_at_20_diff1 + value: 17.066402720849883 + - type: nauc_precision_at_20_max + value: 6.178677607606832 + - type: nauc_precision_at_20_std + value: -3.987829586084965 + - type: nauc_precision_at_3_diff1 + value: 18.358060169256518 + - type: nauc_precision_at_3_max + value: 3.326657304001109 + - type: nauc_precision_at_3_std + value: -10.729398884603352 + - type: nauc_precision_at_5_diff1 + value: 19.41722339541596 + - type: nauc_precision_at_5_max + value: 5.714829813319856 + - type: nauc_precision_at_5_std + value: -8.915414021584194 + - type: nauc_recall_at_1000_diff1 + value: 9.365082280755011 + - type: nauc_recall_at_1000_max + value: 15.829818126823215 + - type: nauc_recall_at_1000_std + value: 27.360808820832666 + - type: nauc_recall_at_100_diff1 + value: 8.05391879951721 + - type: nauc_recall_at_100_max + value: 5.285477600522065 + - type: nauc_recall_at_100_std + value: 13.239431098719457 + - type: nauc_recall_at_10_diff1 + value: 13.288596558862537 + - type: nauc_recall_at_10_max + value: 1.9512189235666242 + - type: nauc_recall_at_10_std + value: 0.08420098367582614 + - type: nauc_recall_at_1_diff1 + value: 28.103238263092063 + - type: nauc_recall_at_1_max + value: 6.669837188399256 + - type: nauc_recall_at_1_std + value: -4.3658897905036405 + - type: nauc_recall_at_20_diff1 + value: 14.781087409113736 + - type: nauc_recall_at_20_max + value: 1.6715579437911525 + - type: nauc_recall_at_20_std + value: 1.4885011649849296 + - type: nauc_recall_at_3_diff1 + value: 16.904223069103445 + - type: nauc_recall_at_3_max + value: 1.2031021965601998 + - type: nauc_recall_at_3_std + value: -5.7358517453558395 + - type: nauc_recall_at_5_diff1 + value: 15.560583779980208 + - type: nauc_recall_at_5_max + value: 1.268944483676161 + - type: nauc_recall_at_5_std + value: -5.114882384179444 + - type: ndcg_at_1 + value: 14.66 + - type: ndcg_at_10 + value: 16.619 + - type: ndcg_at_100 + value: 22.467000000000002 + - type: ndcg_at_1000 + value: 26.745 + - type: ndcg_at_20 + value: 18.356 + - type: ndcg_at_3 + value: 13.547 + - type: ndcg_at_5 + value: 14.466999999999999 + - type: precision_at_1 + value: 14.66 + - type: precision_at_10 + value: 4.8149999999999995 + - type: precision_at_100 + value: 1.0619999999999998 + - type: precision_at_1000 + value: 0.182 + - type: precision_at_20 + value: 3.071 + - type: precision_at_3 + value: 9.002 + - type: precision_at_5 + value: 6.79 + - type: recall_at_1 + value: 7.478999999999999 + - type: recall_at_10 + value: 21.884 + - type: recall_at_100 + value: 45.545 + - type: recall_at_1000 + value: 71.887 + - type: recall_at_20 + value: 27.567999999999998 + - type: recall_at_3 + value: 12.485 + - type: recall_at_5 + value: 15.862000000000002 + task: + type: Retrieval + - dataset: + config: default + name: MTEB HotpotQA (default) + revision: ab518f4d6fcca38d87c25209f94beba119d02014 + split: test + type: mteb/hotpotqa + metrics: + - type: main_score + value: 36.217 + - type: map_at_1 + value: 20.628 + - type: map_at_10 + value: 28.559 + - type: map_at_100 + value: 29.5 + - type: map_at_1000 + value: 29.601 + - type: map_at_20 + value: 29.069 + - type: map_at_3 + value: 26.429000000000002 + - type: map_at_5 + value: 27.589000000000002 + - type: mrr_at_1 + value: 41.2559081701553 + - type: mrr_at_10 + value: 48.84337052399182 + - type: mrr_at_100 + value: 49.523346087979284 + - type: mrr_at_1000 + value: 49.56958885341236 + - type: mrr_at_20 + value: 49.24793448550151 + - type: mrr_at_3 + value: 46.893990546927924 + - type: mrr_at_5 + value: 48.02430790006756 + - type: nauc_map_at_1000_diff1 + value: 47.360168970984724 + - type: nauc_map_at_1000_max + value: 24.614881662381816 + - type: nauc_map_at_1000_std + value: 7.361001821254585 + - type: nauc_map_at_100_diff1 + value: 47.364333667549126 + - type: nauc_map_at_100_max + value: 24.59919582686935 + - type: nauc_map_at_100_std + value: 7.30629187742088 + - type: nauc_map_at_10_diff1 + value: 47.72981170600924 + - type: nauc_map_at_10_max + value: 24.438913671717863 + - type: nauc_map_at_10_std + value: 6.344771843030873 + - type: nauc_map_at_1_diff1 + value: 60.38112885477367 + - type: nauc_map_at_1_max + value: 25.9097175050165 + - type: nauc_map_at_1_std + value: 1.6564371988429167 + - type: nauc_map_at_20_diff1 + value: 47.57684884180127 + - type: nauc_map_at_20_max + value: 24.499763513475443 + - type: nauc_map_at_20_std + value: 6.846169751546589 + - type: nauc_map_at_3_diff1 + value: 49.86374782865936 + - type: nauc_map_at_3_max + value: 24.885292020762233 + - type: nauc_map_at_3_std + value: 4.8258321037343075 + - type: nauc_map_at_5_diff1 + value: 48.41433187485084 + - type: nauc_map_at_5_max + value: 24.439622781310288 + - type: nauc_map_at_5_std + value: 5.664110533938225 + - type: nauc_mrr_at_1000_diff1 + value: 56.730426912840926 + - type: nauc_mrr_at_1000_max + value: 25.303184184778832 + - type: nauc_mrr_at_1000_std + value: 4.096788282752593 + - type: nauc_mrr_at_100_diff1 + value: 56.72217642846328 + - type: nauc_mrr_at_100_max + value: 25.302090289174313 + - type: nauc_mrr_at_100_std + value: 4.108586907297719 + - type: nauc_mrr_at_10_diff1 + value: 56.738023427066885 + - type: nauc_mrr_at_10_max + value: 25.271616491844455 + - type: nauc_mrr_at_10_std + value: 3.824908381559653 + - type: nauc_mrr_at_1_diff1 + value: 60.38112885477367 + - type: nauc_mrr_at_1_max + value: 25.9097175050165 + - type: nauc_mrr_at_1_std + value: 1.6564371988429167 + - type: nauc_mrr_at_20_diff1 + value: 56.70644340159845 + - type: nauc_mrr_at_20_max + value: 25.27993872890672 + - type: nauc_mrr_at_20_std + value: 4.0064390570846875 + - type: nauc_mrr_at_3_diff1 + value: 57.245840183280194 + - type: nauc_mrr_at_3_max + value: 25.33525251108163 + - type: nauc_mrr_at_3_std + value: 2.9291934957523584 + - type: nauc_mrr_at_5_diff1 + value: 56.755596718387125 + - type: nauc_mrr_at_5_max + value: 25.22311364368114 + - type: nauc_mrr_at_5_std + value: 3.5613271952141865 + - type: nauc_ndcg_at_1000_diff1 + value: 46.553394894195456 + - type: nauc_ndcg_at_1000_max + value: 24.938550469205936 + - type: nauc_ndcg_at_1000_std + value: 11.539278224453703 + - type: nauc_ndcg_at_100_diff1 + value: 46.60518292153804 + - type: nauc_ndcg_at_100_max + value: 24.724969691359487 + - type: nauc_ndcg_at_100_std + value: 10.73834721703669 + - type: nauc_ndcg_at_10_diff1 + value: 48.12092181292035 + - type: nauc_ndcg_at_10_max + value: 24.2791002435645 + - type: nauc_ndcg_at_10_std + value: 7.153695707296072 + - type: nauc_ndcg_at_1_diff1 + value: 60.38112885477367 + - type: nauc_ndcg_at_1_max + value: 25.9097175050165 + - type: nauc_ndcg_at_1_std + value: 1.6564371988429167 + - type: nauc_ndcg_at_20_diff1 + value: 47.65117800859018 + - type: nauc_ndcg_at_20_max + value: 24.357451369693482 + - type: nauc_ndcg_at_20_std + value: 8.469581027730795 + - type: nauc_ndcg_at_3_diff1 + value: 51.08303103543016 + - type: nauc_ndcg_at_3_max + value: 24.799424583706255 + - type: nauc_ndcg_at_3_std + value: 4.63909501741516 + - type: nauc_ndcg_at_5_diff1 + value: 49.136821889915225 + - type: nauc_ndcg_at_5_max + value: 24.243099266851612 + - type: nauc_ndcg_at_5_std + value: 5.961841495442629 + - type: nauc_precision_at_1000_diff1 + value: 14.823992446535481 + - type: nauc_precision_at_1000_max + value: 17.957974549199044 + - type: nauc_precision_at_1000_std + value: 31.79928156519854 + - type: nauc_precision_at_100_diff1 + value: 23.121894912525356 + - type: nauc_precision_at_100_max + value: 19.166436915427486 + - type: nauc_precision_at_100_std + value: 23.79964191034748 + - type: nauc_precision_at_10_diff1 + value: 35.6440151764581 + - type: nauc_precision_at_10_max + value: 21.022400502868223 + - type: nauc_precision_at_10_std + value: 11.461152130387351 + - type: nauc_precision_at_1_diff1 + value: 60.38112885477367 + - type: nauc_precision_at_1_max + value: 25.9097175050165 + - type: nauc_precision_at_1_std + value: 1.6564371988429167 + - type: nauc_precision_at_20_diff1 + value: 31.893138428309527 + - type: nauc_precision_at_20_max + value: 19.961827091439737 + - type: nauc_precision_at_20_std + value: 15.056260461619232 + - type: nauc_precision_at_3_diff1 + value: 45.06971180999361 + - type: nauc_precision_at_3_max + value: 23.635891515921788 + - type: nauc_precision_at_3_std + value: 6.198234444102806 + - type: nauc_precision_at_5_diff1 + value: 39.43842818627394 + - type: nauc_precision_at_5_max + value: 21.623592109687603 + - type: nauc_precision_at_5_std + value: 8.718348302717638 + - type: nauc_recall_at_1000_diff1 + value: 14.823992446535502 + - type: nauc_recall_at_1000_max + value: 17.95797454919907 + - type: nauc_recall_at_1000_std + value: 31.799281565198577 + - type: nauc_recall_at_100_diff1 + value: 23.121894912525338 + - type: nauc_recall_at_100_max + value: 19.16643691542745 + - type: nauc_recall_at_100_std + value: 23.799641910347454 + - type: nauc_recall_at_10_diff1 + value: 35.64401517645808 + - type: nauc_recall_at_10_max + value: 21.022400502868223 + - type: nauc_recall_at_10_std + value: 11.461152130387346 + - type: nauc_recall_at_1_diff1 + value: 60.38112885477367 + - type: nauc_recall_at_1_max + value: 25.9097175050165 + - type: nauc_recall_at_1_std + value: 1.6564371988429167 + - type: nauc_recall_at_20_diff1 + value: 31.89313842830953 + - type: nauc_recall_at_20_max + value: 19.961827091439776 + - type: nauc_recall_at_20_std + value: 15.05626046161922 + - type: nauc_recall_at_3_diff1 + value: 45.06971180999365 + - type: nauc_recall_at_3_max + value: 23.6358915159218 + - type: nauc_recall_at_3_std + value: 6.198234444102802 + - type: nauc_recall_at_5_diff1 + value: 39.43842818627392 + - type: nauc_recall_at_5_max + value: 21.623592109687596 + - type: nauc_recall_at_5_std + value: 8.71834830271761 + - type: ndcg_at_1 + value: 41.256 + - type: ndcg_at_10 + value: 36.217 + - type: ndcg_at_100 + value: 40.422000000000004 + - type: ndcg_at_1000 + value: 42.762 + - type: ndcg_at_20 + value: 37.801 + - type: ndcg_at_3 + value: 32.275999999999996 + - type: ndcg_at_5 + value: 34.184 + - type: precision_at_1 + value: 41.256 + - type: precision_at_10 + value: 7.838000000000001 + - type: precision_at_100 + value: 1.119 + - type: precision_at_1000 + value: 0.14300000000000002 + - type: precision_at_20 + value: 4.429 + - type: precision_at_3 + value: 20.207 + - type: precision_at_5 + value: 13.636999999999999 + - type: recall_at_1 + value: 20.628 + - type: recall_at_10 + value: 39.190000000000005 + - type: recall_at_100 + value: 55.962 + - type: recall_at_1000 + value: 71.56700000000001 + - type: recall_at_20 + value: 44.288 + - type: recall_at_3 + value: 30.311 + - type: recall_at_5 + value: 34.092 + task: + type: Retrieval + - dataset: + config: default + name: MTEB ImdbClassification (default) + revision: 3d86128a09e091d6018b6d26cad27f2739fc2db7 + split: test + type: mteb/imdb + metrics: + - type: accuracy + value: 70.78 + - type: ap + value: 65.09281598781793 + - type: ap_weighted + value: 65.09281598781793 + - type: f1 + value: 70.56498155979408 + - type: f1_weighted + value: 70.56498155979408 + - type: main_score + value: 70.78 + task: + type: Classification + - dataset: + config: default + name: MTEB MSMARCO (default) + revision: c5a29a104738b98a9e76336939199e264163d4a0 + split: test + type: mteb/msmarco + metrics: + - type: main_score + value: 34.981 + - type: map_at_1 + value: 0.9369999999999999 + - type: map_at_10 + value: 6.105 + - type: map_at_100 + value: 16.573 + - type: map_at_1000 + value: 20.952 + - type: map_at_20 + value: 9.495000000000001 + - type: map_at_3 + value: 2.429 + - type: map_at_5 + value: 3.7199999999999998 + - type: mrr_at_1 + value: 55.81395348837209 + - type: mrr_at_10 + value: 68.06201550387597 + - type: mrr_at_100 + value: 68.1915571731129 + - type: mrr_at_1000 + value: 68.20171255038517 + - type: mrr_at_20 + value: 68.06201550387597 + - type: mrr_at_3 + value: 65.89147286821705 + - type: mrr_at_5 + value: 67.05426356589147 + - type: nauc_map_at_1000_diff1 + value: 18.395978949265306 + - type: nauc_map_at_1000_max + value: 65.4845955483722 + - type: nauc_map_at_1000_std + value: 60.01425674651855 + - type: nauc_map_at_100_diff1 + value: 17.66459171040137 + - type: nauc_map_at_100_max + value: 56.91214775388199 + - type: nauc_map_at_100_std + value: 51.26999006986676 + - type: nauc_map_at_10_diff1 + value: 16.954292128521953 + - type: nauc_map_at_10_max + value: 29.470502786246144 + - type: nauc_map_at_10_std + value: 26.609751637393327 + - type: nauc_map_at_1_diff1 + value: 10.947697022780028 + - type: nauc_map_at_1_max + value: 11.333211449460881 + - type: nauc_map_at_1_std + value: 19.475048420924633 + - type: nauc_map_at_20_diff1 + value: 13.788525799384063 + - type: nauc_map_at_20_max + value: 36.86668066777578 + - type: nauc_map_at_20_std + value: 31.64971965701265 + - type: nauc_map_at_3_diff1 + value: 17.859630126844696 + - type: nauc_map_at_3_max + value: 21.46834280704547 + - type: nauc_map_at_3_std + value: 21.076387895251823 + - type: nauc_map_at_5_diff1 + value: 20.17441650295119 + - type: nauc_map_at_5_max + value: 24.878188082696866 + - type: nauc_map_at_5_std + value: 25.307502719861176 + - type: nauc_mrr_at_1000_diff1 + value: 14.192749126463891 + - type: nauc_mrr_at_1000_max + value: 52.54526357757101 + - type: nauc_mrr_at_1000_std + value: 44.496694053499596 + - type: nauc_mrr_at_100_diff1 + value: 14.215939043892334 + - type: nauc_mrr_at_100_max + value: 52.564251294672225 + - type: nauc_mrr_at_100_std + value: 44.51890218594217 + - type: nauc_mrr_at_10_diff1 + value: 14.433120969285195 + - type: nauc_mrr_at_10_max + value: 52.78365722715205 + - type: nauc_mrr_at_10_std + value: 44.72011559301776 + - type: nauc_mrr_at_1_diff1 + value: 4.7355957804700415 + - type: nauc_mrr_at_1_max + value: 39.93352486009351 + - type: nauc_mrr_at_1_std + value: 39.55801119967461 + - type: nauc_mrr_at_20_diff1 + value: 14.433120969285195 + - type: nauc_mrr_at_20_max + value: 52.78365722715205 + - type: nauc_mrr_at_20_std + value: 44.72011559301776 + - type: nauc_mrr_at_3_diff1 + value: 13.11183382637074 + - type: nauc_mrr_at_3_max + value: 51.12370908328734 + - type: nauc_mrr_at_3_std + value: 40.238401804460075 + - type: nauc_mrr_at_5_diff1 + value: 13.179254658692855 + - type: nauc_mrr_at_5_max + value: 53.38265101836388 + - type: nauc_mrr_at_5_std + value: 44.541370972177624 + - type: nauc_ndcg_at_1000_diff1 + value: 21.69587945916941 + - type: nauc_ndcg_at_1000_max + value: 63.37066645313249 + - type: nauc_ndcg_at_1000_std + value: 62.97303091219909 + - type: nauc_ndcg_at_100_diff1 + value: 14.796314010328851 + - type: nauc_ndcg_at_100_max + value: 58.71101997436683 + - type: nauc_ndcg_at_100_std + value: 56.81420228421644 + - type: nauc_ndcg_at_10_diff1 + value: 3.194403093296008 + - type: nauc_ndcg_at_10_max + value: 48.55754387196878 + - type: nauc_ndcg_at_10_std + value: 47.48615570741263 + - type: nauc_ndcg_at_1_diff1 + value: -6.148169734658873 + - type: nauc_ndcg_at_1_max + value: 25.556355503841665 + - type: nauc_ndcg_at_1_std + value: 21.48805389151005 + - type: nauc_ndcg_at_20_diff1 + value: 4.461683170351035 + - type: nauc_ndcg_at_20_max + value: 56.88294190421313 + - type: nauc_ndcg_at_20_std + value: 51.93821404537562 + - type: nauc_ndcg_at_3_diff1 + value: -2.861880240597804 + - type: nauc_ndcg_at_3_max + value: 41.33450475096539 + - type: nauc_ndcg_at_3_std + value: 37.27470370159716 + - type: nauc_ndcg_at_5_diff1 + value: 0.08149020695323854 + - type: nauc_ndcg_at_5_max + value: 46.722954751612264 + - type: nauc_ndcg_at_5_std + value: 44.665247293303416 + - type: nauc_precision_at_1000_diff1 + value: 6.514642381748156 + - type: nauc_precision_at_1000_max + value: 54.61143553569596 + - type: nauc_precision_at_1000_std + value: 51.84636945565138 + - type: nauc_precision_at_100_diff1 + value: 9.181266993927007 + - type: nauc_precision_at_100_max + value: 63.29553111429812 + - type: nauc_precision_at_100_std + value: 59.013060721871035 + - type: nauc_precision_at_10_diff1 + value: 16.062673027273505 + - type: nauc_precision_at_10_max + value: 64.85826828536602 + - type: nauc_precision_at_10_std + value: 58.476222375984 + - type: nauc_precision_at_1_diff1 + value: 4.7355957804700415 + - type: nauc_precision_at_1_max + value: 39.93352486009351 + - type: nauc_precision_at_1_std + value: 39.55801119967461 + - type: nauc_precision_at_20_diff1 + value: 12.061096674017728 + - type: nauc_precision_at_20_max + value: 66.81322466200473 + - type: nauc_precision_at_20_std + value: 58.18606533749746 + - type: nauc_precision_at_3_diff1 + value: 9.10289433878097 + - type: nauc_precision_at_3_max + value: 61.00901833818042 + - type: nauc_precision_at_3_std + value: 52.94626237786338 + - type: nauc_precision_at_5_diff1 + value: 13.765083369324818 + - type: nauc_precision_at_5_max + value: 67.0735717931603 + - type: nauc_precision_at_5_std + value: 60.160759158192334 + - type: nauc_recall_at_1000_diff1 + value: 33.378885488094184 + - type: nauc_recall_at_1000_max + value: 58.97167459966026 + - type: nauc_recall_at_1000_std + value: 59.59218645358476 + - type: nauc_recall_at_100_diff1 + value: 25.1307767949282 + - type: nauc_recall_at_100_max + value: 48.29698220976826 + - type: nauc_recall_at_100_std + value: 44.76527467601765 + - type: nauc_recall_at_10_diff1 + value: 21.012536607264714 + - type: nauc_recall_at_10_max + value: 21.719714919287135 + - type: nauc_recall_at_10_std + value: 18.503987452436643 + - type: nauc_recall_at_1_diff1 + value: 10.947697022780028 + - type: nauc_recall_at_1_max + value: 11.333211449460881 + - type: nauc_recall_at_1_std + value: 19.475048420924633 + - type: nauc_recall_at_20_diff1 + value: 14.221666924930961 + - type: nauc_recall_at_20_max + value: 30.83326629354958 + - type: nauc_recall_at_20_std + value: 25.419400751031635 + - type: nauc_recall_at_3_diff1 + value: 19.488515137385438 + - type: nauc_recall_at_3_max + value: 18.682366339227507 + - type: nauc_recall_at_3_std + value: 14.801487977327957 + - type: nauc_recall_at_5_diff1 + value: 21.493404372645262 + - type: nauc_recall_at_5_max + value: 22.470910257369972 + - type: nauc_recall_at_5_std + value: 20.91789333035049 + - type: ndcg_at_1 + value: 36.047000000000004 + - type: ndcg_at_10 + value: 34.981 + - type: ndcg_at_100 + value: 33.928000000000004 + - type: ndcg_at_1000 + value: 42.553999999999995 + - type: ndcg_at_20 + value: 33.768 + - type: ndcg_at_3 + value: 35.477 + - type: ndcg_at_5 + value: 35.54 + - type: precision_at_1 + value: 55.814 + - type: precision_at_10 + value: 46.744 + - type: precision_at_100 + value: 22.721 + - type: precision_at_1000 + value: 4.781 + - type: precision_at_20 + value: 40.465 + - type: precision_at_3 + value: 52.713 + - type: precision_at_5 + value: 51.163000000000004 + - type: recall_at_1 + value: 0.9369999999999999 + - type: recall_at_10 + value: 7.921 + - type: recall_at_100 + value: 28.903000000000002 + - type: recall_at_1000 + value: 53.691 + - type: recall_at_20 + value: 12.745000000000001 + - type: recall_at_3 + value: 2.8240000000000003 + - type: recall_at_5 + value: 4.476999999999999 + task: + type: Retrieval + - dataset: + config: en + name: MTEB MTOPDomainClassification (en) + revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf + split: test + type: mteb/mtop_domain + metrics: + - type: accuracy + value: 88.95576835385319 + - type: f1 + value: 88.06364678376042 + - type: f1_weighted + value: 89.00721562093213 + - type: main_score + value: 88.95576835385319 + task: + type: Classification + - dataset: + config: en + name: MTEB MTOPIntentClassification (en) + revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba + split: test + type: mteb/mtop_intent + metrics: + - type: accuracy + value: 56.99726402188783 + - type: f1 + value: 38.19916053247397 + - type: f1_weighted + value: 59.96788951671549 + - type: main_score + value: 56.99726402188783 + task: + type: Classification + - dataset: + config: en + name: MTEB MassiveIntentClassification (en) + revision: 4672e20407010da34463acc759c162ca9734bca6 + split: test + type: mteb/amazon_massive_intent + metrics: + - type: accuracy + value: 63.79287155346336 + - type: f1 + value: 61.634629394462934 + - type: f1_weighted + value: 62.567311481126055 + - type: main_score + value: 63.79287155346336 + task: + type: Classification + - dataset: + config: en + name: MTEB MassiveScenarioClassification (en) + revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 + split: test + type: mteb/amazon_massive_scenario + metrics: + - type: accuracy + value: 70.30934767989241 + - type: f1 + value: 68.77914761769517 + - type: f1_weighted + value: 70.1128179307388 + - type: main_score + value: 70.30934767989241 + task: + type: Classification + - dataset: + config: default + name: MTEB MedrxivClusteringP2P (default) + revision: e7a26af6f3ae46b30dde8737f02c07b1505bcc73 + split: test + type: mteb/medrxiv-clustering-p2p + metrics: + - type: main_score + value: 27.61734940907637 + - type: v_measure + value: 27.61734940907637 + - type: v_measure_std + value: 1.2248100208316097 + task: + type: Clustering + - dataset: + config: default + name: MTEB MedrxivClusteringS2S (default) + revision: 35191c8c0dca72d8ff3efcd72aa802307d469663 + split: test + type: mteb/medrxiv-clustering-s2s + metrics: + - type: main_score + value: 23.802943866708308 + - type: v_measure + value: 23.802943866708308 + - type: v_measure_std + value: 1.4975518910969763 + task: + type: Clustering + - dataset: + config: default + name: MTEB MindSmallReranking (default) + revision: 59042f120c80e8afa9cdbb224f67076cec0fc9a7 + split: test + type: mteb/mind_small + metrics: + - type: main_score + value: 29.431722284942175 + - type: map + value: 29.431722284942175 + - type: mrr + value: 30.207239990924332 + - type: nAUC_map_diff1 + value: 8.996546748314882 + - type: nAUC_map_max + value: -23.177815249478726 + - type: nAUC_map_std + value: -8.953694065964015 + - type: nAUC_mrr_diff1 + value: 9.247690774332192 + - type: nAUC_mrr_max + value: -17.42779158552557 + - type: nAUC_mrr_std + value: -5.997215692334967 + task: + type: Reranking + - dataset: + config: default + name: MTEB NFCorpus (default) + revision: ec0fa4fe99da2ff19ca1214b7966684033a58814 + split: test + type: mteb/nfcorpus + metrics: + - type: main_score + value: 24.267 + - type: map_at_1 + value: 3.479 + - type: map_at_10 + value: 7.603 + - type: map_at_100 + value: 9.725999999999999 + - type: map_at_1000 + value: 10.84 + - type: map_at_20 + value: 8.458 + - type: map_at_3 + value: 5.844 + - type: map_at_5 + value: 6.732 + - type: mrr_at_1 + value: 33.746130030959755 + - type: mrr_at_10 + value: 43.515897587105016 + - type: mrr_at_100 + value: 44.1900925310943 + - type: mrr_at_1000 + value: 44.248355412773655 + - type: mrr_at_20 + value: 43.868459509915866 + - type: mrr_at_3 + value: 41.74406604747161 + - type: mrr_at_5 + value: 42.82765737874097 + - type: nauc_map_at_1000_diff1 + value: 34.88971488841416 + - type: nauc_map_at_1000_max + value: 31.233839968277195 + - type: nauc_map_at_1000_std + value: 17.992857492799814 + - type: nauc_map_at_100_diff1 + value: 36.76693324709909 + - type: nauc_map_at_100_max + value: 29.86086979425915 + - type: nauc_map_at_100_std + value: 13.839419605590217 + - type: nauc_map_at_10_diff1 + value: 41.84259867098214 + - type: nauc_map_at_10_max + value: 25.879197474145045 + - type: nauc_map_at_10_std + value: 5.172621372587683 + - type: nauc_map_at_1_diff1 + value: 59.30631217950276 + - type: nauc_map_at_1_max + value: 20.33548433428363 + - type: nauc_map_at_1_std + value: -1.8217254079917093 + - type: nauc_map_at_20_diff1 + value: 38.95414455683049 + - type: nauc_map_at_20_max + value: 26.987123257006363 + - type: nauc_map_at_20_std + value: 8.70109669516395 + - type: nauc_map_at_3_diff1 + value: 47.18504542973307 + - type: nauc_map_at_3_max + value: 21.706151469833202 + - type: nauc_map_at_3_std + value: 0.8205050181794802 + - type: nauc_map_at_5_diff1 + value: 45.415931092144476 + - type: nauc_map_at_5_max + value: 23.366427326413234 + - type: nauc_map_at_5_std + value: 2.036343948136038 + - type: nauc_mrr_at_1000_diff1 + value: 34.09352814360173 + - type: nauc_mrr_at_1000_max + value: 36.57744406738573 + - type: nauc_mrr_at_1000_std + value: 18.874642200828255 + - type: nauc_mrr_at_100_diff1 + value: 34.07606233752646 + - type: nauc_mrr_at_100_max + value: 36.570920987632604 + - type: nauc_mrr_at_100_std + value: 18.90704866545748 + - type: nauc_mrr_at_10_diff1 + value: 33.86749261732675 + - type: nauc_mrr_at_10_max + value: 36.53445713485045 + - type: nauc_mrr_at_10_std + value: 18.72635222657426 + - type: nauc_mrr_at_1_diff1 + value: 38.310753456104415 + - type: nauc_mrr_at_1_max + value: 32.080433604684444 + - type: nauc_mrr_at_1_std + value: 10.76705379557832 + - type: nauc_mrr_at_20_diff1 + value: 34.05889362360272 + - type: nauc_mrr_at_20_max + value: 36.539902847898894 + - type: nauc_mrr_at_20_std + value: 18.829170969376136 + - type: nauc_mrr_at_3_diff1 + value: 34.661230693226 + - type: nauc_mrr_at_3_max + value: 35.27494037957078 + - type: nauc_mrr_at_3_std + value: 16.799715396839538 + - type: nauc_mrr_at_5_diff1 + value: 34.30568391918026 + - type: nauc_mrr_at_5_max + value: 36.31513238612551 + - type: nauc_mrr_at_5_std + value: 18.248879043938977 + - type: nauc_ndcg_at_1000_diff1 + value: 28.625594076978317 + - type: nauc_ndcg_at_1000_max + value: 39.10317925519372 + - type: nauc_ndcg_at_1000_std + value: 28.285055860454257 + - type: nauc_ndcg_at_100_diff1 + value: 27.620568325357986 + - type: nauc_ndcg_at_100_max + value: 34.32867733567831 + - type: nauc_ndcg_at_100_std + value: 25.103257804738867 + - type: nauc_ndcg_at_10_diff1 + value: 24.527566945282576 + - type: nauc_ndcg_at_10_max + value: 32.19051221282665 + - type: nauc_ndcg_at_10_std + value: 25.403501921327432 + - type: nauc_ndcg_at_1_diff1 + value: 38.95386802348185 + - type: nauc_ndcg_at_1_max + value: 30.134605059752644 + - type: nauc_ndcg_at_1_std + value: 11.904644683131 + - type: nauc_ndcg_at_20_diff1 + value: 25.422544698266798 + - type: nauc_ndcg_at_20_max + value: 31.85394200124836 + - type: nauc_ndcg_at_20_std + value: 26.925279769256523 + - type: nauc_ndcg_at_3_diff1 + value: 27.968874988258573 + - type: nauc_ndcg_at_3_max + value: 30.93696431950224 + - type: nauc_ndcg_at_3_std + value: 18.551823245893114 + - type: nauc_ndcg_at_5_diff1 + value: 25.722349682774233 + - type: nauc_ndcg_at_5_max + value: 32.29294830500251 + - type: nauc_ndcg_at_5_std + value: 21.309663190563718 + - type: nauc_precision_at_1000_diff1 + value: -7.466934392543785 + - type: nauc_precision_at_1000_max + value: 17.534662065944236 + - type: nauc_precision_at_1000_std + value: 43.86335465977071 + - type: nauc_precision_at_100_diff1 + value: -2.073530455550674 + - type: nauc_precision_at_100_max + value: 26.51626141328235 + - type: nauc_precision_at_100_std + value: 47.02741717034574 + - type: nauc_precision_at_10_diff1 + value: 6.717006995188633 + - type: nauc_precision_at_10_max + value: 32.738691529253494 + - type: nauc_precision_at_10_std + value: 35.80103442917034 + - type: nauc_precision_at_1_diff1 + value: 38.310753456104415 + - type: nauc_precision_at_1_max + value: 32.080433604684444 + - type: nauc_precision_at_1_std + value: 10.76705379557832 + - type: nauc_precision_at_20_diff1 + value: 2.745832502363386 + - type: nauc_precision_at_20_max + value: 30.954145690157688 + - type: nauc_precision_at_20_std + value: 41.74795596694651 + - type: nauc_precision_at_3_diff1 + value: 20.04271494210498 + - type: nauc_precision_at_3_max + value: 32.49798591360355 + - type: nauc_precision_at_3_std + value: 22.433174666547337 + - type: nauc_precision_at_5_diff1 + value: 13.559244763754297 + - type: nauc_precision_at_5_max + value: 34.29174467545541 + - type: nauc_precision_at_5_std + value: 27.67088510253159 + - type: nauc_recall_at_1000_diff1 + value: 14.406899781864585 + - type: nauc_recall_at_1000_max + value: 18.63293041982341 + - type: nauc_recall_at_1000_std + value: 14.873113563587054 + - type: nauc_recall_at_100_diff1 + value: 20.276630820341023 + - type: nauc_recall_at_100_max + value: 20.74130868375551 + - type: nauc_recall_at_100_std + value: 14.253807947296465 + - type: nauc_recall_at_10_diff1 + value: 32.131322772361194 + - type: nauc_recall_at_10_max + value: 21.834619003317645 + - type: nauc_recall_at_10_std + value: 5.111047982154726 + - type: nauc_recall_at_1_diff1 + value: 59.30631217950276 + - type: nauc_recall_at_1_max + value: 20.33548433428363 + - type: nauc_recall_at_1_std + value: -1.8217254079917093 + - type: nauc_recall_at_20_diff1 + value: 29.009526186873646 + - type: nauc_recall_at_20_max + value: 19.222693262075214 + - type: nauc_recall_at_20_std + value: 8.263428180065297 + - type: nauc_recall_at_3_diff1 + value: 38.428506196942266 + - type: nauc_recall_at_3_max + value: 18.92885903756039 + - type: nauc_recall_at_3_std + value: 2.2767688747391106 + - type: nauc_recall_at_5_diff1 + value: 35.93597428489607 + - type: nauc_recall_at_5_max + value: 19.591607144107787 + - type: nauc_recall_at_5_std + value: 2.110828447844176 + - type: ndcg_at_1 + value: 31.424000000000003 + - type: ndcg_at_10 + value: 24.267 + - type: ndcg_at_100 + value: 22.416 + - type: ndcg_at_1000 + value: 31.165 + - type: ndcg_at_20 + value: 22.698 + - type: ndcg_at_3 + value: 28.349999999999998 + - type: ndcg_at_5 + value: 26.596999999999998 + - type: precision_at_1 + value: 33.745999999999995 + - type: precision_at_10 + value: 18.173000000000002 + - type: precision_at_100 + value: 6.142 + - type: precision_at_1000 + value: 1.856 + - type: precision_at_20 + value: 13.808000000000002 + - type: precision_at_3 + value: 27.141 + - type: precision_at_5 + value: 22.91 + - type: recall_at_1 + value: 3.479 + - type: recall_at_10 + value: 10.838000000000001 + - type: recall_at_100 + value: 23.817 + - type: recall_at_1000 + value: 54.910000000000004 + - type: recall_at_20 + value: 14.201 + - type: recall_at_3 + value: 7.236 + - type: recall_at_5 + value: 9.003 + task: + type: Retrieval + - dataset: + config: default + name: MTEB NQ (default) + revision: b774495ed302d8c44a3a7ea25c90dbce03968f31 + split: test + type: mteb/nq + metrics: + - type: main_score + value: 19.543 + - type: map_at_1 + value: 8.413 + - type: map_at_10 + value: 15.137 + - type: map_at_100 + value: 16.393 + - type: map_at_1000 + value: 16.492 + - type: map_at_20 + value: 15.827 + - type: map_at_3 + value: 12.584999999999999 + - type: map_at_5 + value: 13.963000000000001 + - type: mrr_at_1 + value: 9.73348783314021 + - type: mrr_at_10 + value: 16.79895712630359 + - type: mrr_at_100 + value: 17.96527488497497 + - type: mrr_at_1000 + value: 18.049284621380956 + - type: mrr_at_20 + value: 17.456541969883244 + - type: mrr_at_3 + value: 14.2429509463113 + - type: mrr_at_5 + value: 15.636346079567373 + - type: nauc_map_at_1000_diff1 + value: 18.819971639310904 + - type: nauc_map_at_1000_max + value: 13.814947350680912 + - type: nauc_map_at_1000_std + value: 2.521914759184715 + - type: nauc_map_at_100_diff1 + value: 18.814255883152295 + - type: nauc_map_at_100_max + value: 13.784098474987728 + - type: nauc_map_at_100_std + value: 2.463386644603925 + - type: nauc_map_at_10_diff1 + value: 18.859741700546 + - type: nauc_map_at_10_max + value: 13.200112454161522 + - type: nauc_map_at_10_std + value: 1.2838729142015952 + - type: nauc_map_at_1_diff1 + value: 22.792911666175435 + - type: nauc_map_at_1_max + value: 9.420966909430586 + - type: nauc_map_at_1_std + value: -2.177707391834426 + - type: nauc_map_at_20_diff1 + value: 18.857585870077603 + - type: nauc_map_at_20_max + value: 13.494371000020585 + - type: nauc_map_at_20_std + value: 1.7987081767888724 + - type: nauc_map_at_3_diff1 + value: 20.3919043114244 + - type: nauc_map_at_3_max + value: 11.229233328712159 + - type: nauc_map_at_3_std + value: -0.38627708043707826 + - type: nauc_map_at_5_diff1 + value: 19.354241266183816 + - type: nauc_map_at_5_max + value: 12.050995012138287 + - type: nauc_map_at_5_std + value: 0.4619900683963445 + - type: nauc_mrr_at_1000_diff1 + value: 17.44597143162577 + - type: nauc_mrr_at_1000_max + value: 12.99325734801233 + - type: nauc_mrr_at_1000_std + value: 3.843471729334042 + - type: nauc_mrr_at_100_diff1 + value: 17.435646674940784 + - type: nauc_mrr_at_100_max + value: 12.977733602157626 + - type: nauc_mrr_at_100_std + value: 3.819688827654704 + - type: nauc_mrr_at_10_diff1 + value: 17.366258247556274 + - type: nauc_mrr_at_10_max + value: 12.525863095955028 + - type: nauc_mrr_at_10_std + value: 2.9586217333067033 + - type: nauc_mrr_at_1_diff1 + value: 21.181200992092933 + - type: nauc_mrr_at_1_max + value: 9.071174422547715 + - type: nauc_mrr_at_1_std + value: 0.37666341313223156 + - type: nauc_mrr_at_20_diff1 + value: 17.47842029246494 + - type: nauc_mrr_at_20_max + value: 12.782728137865854 + - type: nauc_mrr_at_20_std + value: 3.335207400639897 + - type: nauc_mrr_at_3_diff1 + value: 18.51145002403263 + - type: nauc_mrr_at_3_max + value: 10.835289485126742 + - type: nauc_mrr_at_3_std + value: 1.9317890085586098 + - type: nauc_mrr_at_5_diff1 + value: 17.85072852768249 + - type: nauc_mrr_at_5_max + value: 11.48513938150474 + - type: nauc_mrr_at_5_std + value: 2.42459300983239 + - type: nauc_ndcg_at_1000_diff1 + value: 16.90906471124972 + - type: nauc_ndcg_at_1000_max + value: 18.10309890125217 + - type: nauc_ndcg_at_1000_std + value: 9.531587494208333 + - type: nauc_ndcg_at_100_diff1 + value: 16.794610031459452 + - type: nauc_ndcg_at_100_max + value: 17.320423121617587 + - type: nauc_ndcg_at_100_std + value: 8.36089871892644 + - type: nauc_ndcg_at_10_diff1 + value: 16.9238328483549 + - type: nauc_ndcg_at_10_max + value: 15.003898384476175 + - type: nauc_ndcg_at_10_std + value: 3.220068514580869 + - type: nauc_ndcg_at_1_diff1 + value: 21.181200992092933 + - type: nauc_ndcg_at_1_max + value: 9.071174422547715 + - type: nauc_ndcg_at_1_std + value: 0.37666341313223156 + - type: nauc_ndcg_at_20_diff1 + value: 17.122783032672636 + - type: nauc_ndcg_at_20_max + value: 15.811529036192868 + - type: nauc_ndcg_at_20_std + value: 4.638881062044276 + - type: nauc_ndcg_at_3_diff1 + value: 19.397651629456085 + - type: nauc_ndcg_at_3_max + value: 11.519185092964664 + - type: nauc_ndcg_at_3_std + value: 0.5852664941054009 + - type: nauc_ndcg_at_5_diff1 + value: 17.836092374281833 + - type: nauc_ndcg_at_5_max + value: 12.692159310256345 + - type: nauc_ndcg_at_5_std + value: 1.7356004993081944 + - type: nauc_precision_at_1000_diff1 + value: 3.073453832047264 + - type: nauc_precision_at_1000_max + value: 23.790855697865958 + - type: nauc_precision_at_1000_std + value: 32.57511127212919 + - type: nauc_precision_at_100_diff1 + value: 9.127444700503846 + - type: nauc_precision_at_100_max + value: 22.71156118580008 + - type: nauc_precision_at_100_std + value: 24.63648530454141 + - type: nauc_precision_at_10_diff1 + value: 13.02401021030829 + - type: nauc_precision_at_10_max + value: 18.85263386483255 + - type: nauc_precision_at_10_std + value: 8.373513612599647 + - type: nauc_precision_at_1_diff1 + value: 21.181200992092933 + - type: nauc_precision_at_1_max + value: 9.071174422547715 + - type: nauc_precision_at_1_std + value: 0.37666341313223156 + - type: nauc_precision_at_20_diff1 + value: 12.975989332948448 + - type: nauc_precision_at_20_max + value: 20.296858370304385 + - type: nauc_precision_at_20_std + value: 12.119876359299383 + - type: nauc_precision_at_3_diff1 + value: 17.130641156396027 + - type: nauc_precision_at_3_max + value: 12.010571872098485 + - type: nauc_precision_at_3_std + value: 2.637465881798806 + - type: nauc_precision_at_5_diff1 + value: 14.960326184287629 + - type: nauc_precision_at_5_max + value: 14.264819044499205 + - type: nauc_precision_at_5_std + value: 4.5445140864787215 + - type: nauc_recall_at_1000_diff1 + value: 11.322486975456016 + - type: nauc_recall_at_1000_max + value: 42.74305283200241 + - type: nauc_recall_at_1000_std + value: 47.78794764298061 + - type: nauc_recall_at_100_diff1 + value: 12.242221079259041 + - type: nauc_recall_at_100_max + value: 26.918744103646013 + - type: nauc_recall_at_100_std + value: 24.541980019505186 + - type: nauc_recall_at_10_diff1 + value: 13.38045827515169 + - type: nauc_recall_at_10_max + value: 18.545456163809533 + - type: nauc_recall_at_10_std + value: 5.734945625849404 + - type: nauc_recall_at_1_diff1 + value: 22.792911666175435 + - type: nauc_recall_at_1_max + value: 9.420966909430586 + - type: nauc_recall_at_1_std + value: -2.177707391834426 + - type: nauc_recall_at_20_diff1 + value: 14.133329746281683 + - type: nauc_recall_at_20_max + value: 20.394153554260118 + - type: nauc_recall_at_20_std + value: 9.229321407977622 + - type: nauc_recall_at_3_diff1 + value: 18.230047011254864 + - type: nauc_recall_at_3_max + value: 12.217461047044784 + - type: nauc_recall_at_3_std + value: 1.0395060720237228 + - type: nauc_recall_at_5_diff1 + value: 14.947190921163273 + - type: nauc_recall_at_5_max + value: 13.844816353548604 + - type: nauc_recall_at_5_std + value: 2.9621844586841086 + - type: ndcg_at_1 + value: 9.733 + - type: ndcg_at_10 + value: 19.543 + - type: ndcg_at_100 + value: 25.965 + - type: ndcg_at_1000 + value: 28.663 + - type: ndcg_at_20 + value: 21.985 + - type: ndcg_at_3 + value: 14.308000000000002 + - type: ndcg_at_5 + value: 16.771 + - type: precision_at_1 + value: 9.733 + - type: precision_at_10 + value: 3.7249999999999996 + - type: precision_at_100 + value: 0.739 + - type: precision_at_1000 + value: 0.1 + - type: precision_at_20 + value: 2.4330000000000003 + - type: precision_at_3 + value: 6.856 + - type: precision_at_5 + value: 5.475 + - type: recall_at_1 + value: 8.413 + - type: recall_at_10 + value: 31.668000000000003 + - type: recall_at_100 + value: 61.551 + - type: recall_at_1000 + value: 82.228 + - type: recall_at_20 + value: 40.888999999999996 + - type: recall_at_3 + value: 17.669 + - type: recall_at_5 + value: 23.488999999999997 + task: + type: Retrieval + - dataset: + config: default + name: MTEB QuoraRetrieval (default) + revision: e4e08e0b7dbe3c8700f0daef558ff32256715259 + split: test + type: mteb/quora + metrics: + - type: main_score + value: 80.598 + - type: map_at_1 + value: 63.532 + - type: map_at_10 + value: 76.07300000000001 + - type: map_at_100 + value: 76.863 + - type: map_at_1000 + value: 76.896 + - type: map_at_20 + value: 76.575 + - type: map_at_3 + value: 73.075 + - type: map_at_5 + value: 74.888 + - type: mrr_at_1 + value: 73.11 + - type: mrr_at_10 + value: 80.13760714285678 + - type: mrr_at_100 + value: 80.40676931635143 + - type: mrr_at_1000 + value: 80.413857041773 + - type: mrr_at_20 + value: 80.33569450368124 + - type: mrr_at_3 + value: 78.73166666666627 + - type: mrr_at_5 + value: 79.60316666666607 + - type: nauc_map_at_1000_diff1 + value: 71.76748518946404 + - type: nauc_map_at_1000_max + value: 37.52091562623074 + - type: nauc_map_at_1000_std + value: -19.886772833711106 + - type: nauc_map_at_100_diff1 + value: 71.77392469494623 + - type: nauc_map_at_100_max + value: 37.51305402355471 + - type: nauc_map_at_100_std + value: -19.90950133564633 + - type: nauc_map_at_10_diff1 + value: 71.78435718469383 + - type: nauc_map_at_10_max + value: 37.12859151143304 + - type: nauc_map_at_10_std + value: -20.6727975668906 + - type: nauc_map_at_1_diff1 + value: 74.16329762399023 + - type: nauc_map_at_1_max + value: 30.710315707498864 + - type: nauc_map_at_1_std + value: -19.3193474040897 + - type: nauc_map_at_20_diff1 + value: 71.8048608565351 + - type: nauc_map_at_20_max + value: 37.437936254957336 + - type: nauc_map_at_20_std + value: -20.256332267213164 + - type: nauc_map_at_3_diff1 + value: 72.15934361454754 + - type: nauc_map_at_3_max + value: 35.34630080626579 + - type: nauc_map_at_3_std + value: -22.03571060362441 + - type: nauc_map_at_5_diff1 + value: 71.83699898564598 + - type: nauc_map_at_5_max + value: 36.479498983192975 + - type: nauc_map_at_5_std + value: -21.231304270451062 + - type: nauc_mrr_at_1000_diff1 + value: 72.88897169606878 + - type: nauc_mrr_at_1000_max + value: 40.200221349285634 + - type: nauc_mrr_at_1000_std + value: -17.633375591506123 + - type: nauc_mrr_at_100_diff1 + value: 72.88918562563104 + - type: nauc_mrr_at_100_max + value: 40.20508375617468 + - type: nauc_mrr_at_100_std + value: -17.62754237516005 + - type: nauc_mrr_at_10_diff1 + value: 72.78722143722388 + - type: nauc_mrr_at_10_max + value: 40.26493516347653 + - type: nauc_mrr_at_10_std + value: -17.591516046092213 + - type: nauc_mrr_at_1_diff1 + value: 74.20323111992924 + - type: nauc_mrr_at_1_max + value: 39.1888925247388 + - type: nauc_mrr_at_1_std + value: -17.041083591080856 + - type: nauc_mrr_at_20_diff1 + value: 72.87614719969847 + - type: nauc_mrr_at_20_max + value: 40.25187245577547 + - type: nauc_mrr_at_20_std + value: -17.623643078270213 + - type: nauc_mrr_at_3_diff1 + value: 72.70424133205663 + - type: nauc_mrr_at_3_max + value: 40.015103745774944 + - type: nauc_mrr_at_3_std + value: -18.296912082298693 + - type: nauc_mrr_at_5_diff1 + value: 72.6695462203408 + - type: nauc_mrr_at_5_max + value: 40.166677547198724 + - type: nauc_mrr_at_5_std + value: -17.836669429879553 + - type: nauc_ndcg_at_1000_diff1 + value: 71.7014600627096 + - type: nauc_ndcg_at_1000_max + value: 39.17528447849729 + - type: nauc_ndcg_at_1000_std + value: -18.169144412803025 + - type: nauc_ndcg_at_100_diff1 + value: 71.72812292491562 + - type: nauc_ndcg_at_100_max + value: 39.178065817466866 + - type: nauc_ndcg_at_100_std + value: -17.98857148420824 + - type: nauc_ndcg_at_10_diff1 + value: 71.22490342106018 + - type: nauc_ndcg_at_10_max + value: 38.58976910658222 + - type: nauc_ndcg_at_10_std + value: -19.3807889122846 + - type: nauc_ndcg_at_1_diff1 + value: 74.20323111992924 + - type: nauc_ndcg_at_1_max + value: 39.18366557965937 + - type: nauc_ndcg_at_1_std + value: -16.979563433712343 + - type: nauc_ndcg_at_20_diff1 + value: 71.59416957115776 + - type: nauc_ndcg_at_20_max + value: 39.11048553178983 + - type: nauc_ndcg_at_20_std + value: -18.913452979338476 + - type: nauc_ndcg_at_3_diff1 + value: 71.15596154191027 + - type: nauc_ndcg_at_3_max + value: 37.36564154714553 + - type: nauc_ndcg_at_3_std + value: -20.721815190390565 + - type: nauc_ndcg_at_5_diff1 + value: 71.0047395584928 + - type: nauc_ndcg_at_5_max + value: 37.95479899642812 + - type: nauc_ndcg_at_5_std + value: -20.008045920279887 + - type: nauc_precision_at_1000_diff1 + value: -36.79287717727177 + - type: nauc_precision_at_1000_max + value: -4.853042765778535 + - type: nauc_precision_at_1000_std + value: 21.89700327903914 + - type: nauc_precision_at_100_diff1 + value: -33.803566917391024 + - type: nauc_precision_at_100_max + value: -2.343501157957199 + - type: nauc_precision_at_100_std + value: 21.03134251148425 + - type: nauc_precision_at_10_diff1 + value: -19.647078935128047 + - type: nauc_precision_at_10_max + value: 7.646163968592671 + - type: nauc_precision_at_10_std + value: 11.425640109742039 + - type: nauc_precision_at_1_diff1 + value: 74.20323111992924 + - type: nauc_precision_at_1_max + value: 39.18366557965937 + - type: nauc_precision_at_1_std + value: -16.979563433712343 + - type: nauc_precision_at_20_diff1 + value: -26.95360783576433 + - type: nauc_precision_at_20_max + value: 3.534889652498316 + - type: nauc_precision_at_20_std + value: 16.011941126119197 + - type: nauc_precision_at_3_diff1 + value: 7.80806721613657 + - type: nauc_precision_at_3_max + value: 18.93471456458755 + - type: nauc_precision_at_3_std + value: -2.3471793824170493 + - type: nauc_precision_at_5_diff1 + value: -7.187077136844068 + - type: nauc_precision_at_5_max + value: 13.710196203710806 + - type: nauc_precision_at_5_std + value: 5.029517000064198 + - type: nauc_recall_at_1000_diff1 + value: 55.29138658386572 + - type: nauc_recall_at_1000_max + value: 57.58368141138265 + - type: nauc_recall_at_1000_std + value: 33.353499745829765 + - type: nauc_recall_at_100_diff1 + value: 65.98407378542676 + - type: nauc_recall_at_100_max + value: 43.3437006049648 + - type: nauc_recall_at_100_std + value: 3.7556643837275345 + - type: nauc_recall_at_10_diff1 + value: 64.73552843826317 + - type: nauc_recall_at_10_max + value: 37.93061567923699 + - type: nauc_recall_at_10_std + value: -19.1098323242707 + - type: nauc_recall_at_1_diff1 + value: 74.16329762399023 + - type: nauc_recall_at_1_max + value: 30.710315707498864 + - type: nauc_recall_at_1_std + value: -19.3193474040897 + - type: nauc_recall_at_20_diff1 + value: 64.4507396763554 + - type: nauc_recall_at_20_max + value: 40.62914458603293 + - type: nauc_recall_at_20_std + value: -15.040711675139082 + - type: nauc_recall_at_3_diff1 + value: 67.8143518137102 + - type: nauc_recall_at_3_max + value: 33.649275891159945 + - type: nauc_recall_at_3_std + value: -24.400275123272163 + - type: nauc_recall_at_5_diff1 + value: 65.9405683463817 + - type: nauc_recall_at_5_max + value: 35.64051201738537 + - type: nauc_recall_at_5_std + value: -22.06335424061329 + - type: ndcg_at_1 + value: 73.11 + - type: ndcg_at_10 + value: 80.598 + - type: ndcg_at_100 + value: 82.75200000000001 + - type: ndcg_at_1000 + value: 83.145 + - type: ndcg_at_20 + value: 81.71300000000001 + - type: ndcg_at_3 + value: 77.025 + - type: ndcg_at_5 + value: 78.85 + - type: precision_at_1 + value: 73.11 + - type: precision_at_10 + value: 12.206999999999999 + - type: precision_at_100 + value: 1.459 + - type: precision_at_1000 + value: 0.155 + - type: precision_at_20 + value: 6.579 + - type: precision_at_3 + value: 33.36 + - type: precision_at_5 + value: 22.09 + - type: recall_at_1 + value: 63.532 + - type: recall_at_10 + value: 89.32600000000001 + - type: recall_at_100 + value: 97.35000000000001 + - type: recall_at_1000 + value: 99.613 + - type: recall_at_20 + value: 93.151 + - type: recall_at_3 + value: 79.074 + - type: recall_at_5 + value: 84.143 + task: + type: Retrieval + - dataset: + config: default + name: MTEB RedditClustering (default) + revision: 24640382cdbf8abc73003fb0fa6d111a705499eb + split: test + type: mteb/reddit-clustering + metrics: + - type: main_score + value: 39.5465127563479 + - type: v_measure + value: 39.5465127563479 + - type: v_measure_std + value: 5.038703300031419 + task: + type: Clustering + - dataset: + config: default + name: MTEB RedditClusteringP2P (default) + revision: 385e3cb46b4cfa89021f56c4380204149d0efe33 + split: test + type: mteb/reddit-clustering-p2p + metrics: + - type: main_score + value: 47.07911795189491 + - type: v_measure + value: 47.07911795189491 + - type: v_measure_std + value: 11.546436135362846 + task: + type: Clustering + - dataset: + config: default + name: MTEB SCIDOCS (default) + revision: f8c2fcf00f625baaa80f62ec5bd9e1fff3b8ae88 + split: test + type: mteb/scidocs + metrics: + - type: main_score + value: 12.386999999999999 + - type: map_at_1 + value: 3.053 + - type: map_at_10 + value: 6.912999999999999 + - type: map_at_100 + value: 8.261000000000001 + - type: map_at_1000 + value: 8.530999999999999 + - type: map_at_20 + value: 7.566000000000001 + - type: map_at_3 + value: 5.094 + - type: map_at_5 + value: 5.997 + - type: mrr_at_1 + value: 15.0 + - type: mrr_at_10 + value: 22.795357142857135 + - type: mrr_at_100 + value: 24.007787966055577 + - type: mrr_at_1000 + value: 24.09964360060081 + - type: mrr_at_20 + value: 23.466190383404 + - type: mrr_at_3 + value: 20.100000000000012 + - type: mrr_at_5 + value: 21.685000000000006 + - type: nauc_map_at_1000_diff1 + value: 11.73412101608325 + - type: nauc_map_at_1000_max + value: 14.330449150895694 + - type: nauc_map_at_1000_std + value: 15.742095990011743 + - type: nauc_map_at_100_diff1 + value: 11.777038848684697 + - type: nauc_map_at_100_max + value: 14.104140826193404 + - type: nauc_map_at_100_std + value: 15.155771699462264 + - type: nauc_map_at_10_diff1 + value: 12.374060330916672 + - type: nauc_map_at_10_max + value: 11.856630361520313 + - type: nauc_map_at_10_std + value: 11.753665232073269 + - type: nauc_map_at_1_diff1 + value: 16.986085327339335 + - type: nauc_map_at_1_max + value: 12.246255844992572 + - type: nauc_map_at_1_std + value: 7.863450169503143 + - type: nauc_map_at_20_diff1 + value: 11.634858111388464 + - type: nauc_map_at_20_max + value: 13.108008262696513 + - type: nauc_map_at_20_std + value: 13.423455469499999 + - type: nauc_map_at_3_diff1 + value: 14.889445454705324 + - type: nauc_map_at_3_max + value: 11.572110481390013 + - type: nauc_map_at_3_std + value: 8.556136010622351 + - type: nauc_map_at_5_diff1 + value: 12.907309838627985 + - type: nauc_map_at_5_max + value: 11.000220583694968 + - type: nauc_map_at_5_std + value: 10.111376166991917 + - type: nauc_mrr_at_1000_diff1 + value: 14.963874100415397 + - type: nauc_mrr_at_1000_max + value: 13.495160823256164 + - type: nauc_mrr_at_1000_std + value: 11.28815345444998 + - type: nauc_mrr_at_100_diff1 + value: 14.97621893176082 + - type: nauc_mrr_at_100_max + value: 13.464936280105155 + - type: nauc_mrr_at_100_std + value: 11.305521958378108 + - type: nauc_mrr_at_10_diff1 + value: 14.956869421525884 + - type: nauc_mrr_at_10_max + value: 13.425685629657924 + - type: nauc_mrr_at_10_std + value: 10.767260180262618 + - type: nauc_mrr_at_1_diff1 + value: 16.83378691664147 + - type: nauc_mrr_at_1_max + value: 12.112287067835906 + - type: nauc_mrr_at_1_std + value: 8.418304606390475 + - type: nauc_mrr_at_20_diff1 + value: 14.917032940839656 + - type: nauc_mrr_at_20_max + value: 13.41755983642966 + - type: nauc_mrr_at_20_std + value: 11.11458079038555 + - type: nauc_mrr_at_3_diff1 + value: 15.214496970273089 + - type: nauc_mrr_at_3_max + value: 12.165871395179483 + - type: nauc_mrr_at_3_std + value: 9.980162064503286 + - type: nauc_mrr_at_5_diff1 + value: 14.835204244776087 + - type: nauc_mrr_at_5_max + value: 12.524956858818742 + - type: nauc_mrr_at_5_std + value: 10.099655249800849 + - type: nauc_ndcg_at_1000_diff1 + value: 10.764737128236437 + - type: nauc_ndcg_at_1000_max + value: 18.3469700109834 + - type: nauc_ndcg_at_1000_std + value: 23.22837765426608 + - type: nauc_ndcg_at_100_diff1 + value: 11.606245579895573 + - type: nauc_ndcg_at_100_max + value: 17.167157579603412 + - type: nauc_ndcg_at_100_std + value: 20.347909657378473 + - type: nauc_ndcg_at_10_diff1 + value: 12.394040285590439 + - type: nauc_ndcg_at_10_max + value: 13.388439287974505 + - type: nauc_ndcg_at_10_std + value: 13.188024533529397 + - type: nauc_ndcg_at_1_diff1 + value: 16.83378691664147 + - type: nauc_ndcg_at_1_max + value: 12.112287067835906 + - type: nauc_ndcg_at_1_std + value: 8.418304606390475 + - type: nauc_ndcg_at_20_diff1 + value: 11.212784095325706 + - type: nauc_ndcg_at_20_max + value: 15.185332617097233 + - type: nauc_ndcg_at_20_std + value: 16.087050160363443 + - type: nauc_ndcg_at_3_diff1 + value: 14.708471591387005 + - type: nauc_ndcg_at_3_max + value: 11.70756510699363 + - type: nauc_ndcg_at_3_std + value: 9.658612404132116 + - type: nauc_ndcg_at_5_diff1 + value: 13.123868466784149 + - type: nauc_ndcg_at_5_max + value: 11.60382600862464 + - type: nauc_ndcg_at_5_std + value: 10.625775061954277 + - type: nauc_precision_at_1000_diff1 + value: 3.608251418490512 + - type: nauc_precision_at_1000_max + value: 20.501537930519582 + - type: nauc_precision_at_1000_std + value: 34.4770607840569 + - type: nauc_precision_at_100_diff1 + value: 7.864853652134883 + - type: nauc_precision_at_100_max + value: 19.894334894038547 + - type: nauc_precision_at_100_std + value: 28.711783183330663 + - type: nauc_precision_at_10_diff1 + value: 9.605214553552692 + - type: nauc_precision_at_10_max + value: 14.347596155123817 + - type: nauc_precision_at_10_std + value: 16.242794843380032 + - type: nauc_precision_at_1_diff1 + value: 16.83378691664147 + - type: nauc_precision_at_1_max + value: 12.112287067835906 + - type: nauc_precision_at_1_std + value: 8.418304606390475 + - type: nauc_precision_at_20_diff1 + value: 6.9964985542924545 + - type: nauc_precision_at_20_max + value: 17.275243538199216 + - type: nauc_precision_at_20_std + value: 20.986245055691036 + - type: nauc_precision_at_3_diff1 + value: 13.995705983866177 + - type: nauc_precision_at_3_max + value: 11.391320470301181 + - type: nauc_precision_at_3_std + value: 10.151716783634907 + - type: nauc_precision_at_5_diff1 + value: 11.064867165700008 + - type: nauc_precision_at_5_max + value: 10.965289810519257 + - type: nauc_precision_at_5_std + value: 11.837752544253021 + - type: nauc_recall_at_1000_diff1 + value: 3.4118402840027118 + - type: nauc_recall_at_1000_max + value: 21.505334337938027 + - type: nauc_recall_at_1000_std + value: 34.87205826061254 + - type: nauc_recall_at_100_diff1 + value: 7.793188645900735 + - type: nauc_recall_at_100_max + value: 20.09269964020807 + - type: nauc_recall_at_100_std + value: 28.838050639358375 + - type: nauc_recall_at_10_diff1 + value: 10.010288074812564 + - type: nauc_recall_at_10_max + value: 14.470333599080465 + - type: nauc_recall_at_10_std + value: 16.106977670704044 + - type: nauc_recall_at_1_diff1 + value: 16.986085327339335 + - type: nauc_recall_at_1_max + value: 12.246255844992572 + - type: nauc_recall_at_1_std + value: 7.863450169503143 + - type: nauc_recall_at_20_diff1 + value: 7.248991485381231 + - type: nauc_recall_at_20_max + value: 17.357162157871585 + - type: nauc_recall_at_20_std + value: 20.916649810908385 + - type: nauc_recall_at_3_diff1 + value: 14.190312777099356 + - type: nauc_recall_at_3_max + value: 11.494013846579504 + - type: nauc_recall_at_3_std + value: 9.871734511413411 + - type: nauc_recall_at_5_diff1 + value: 11.369318015463497 + - type: nauc_recall_at_5_max + value: 11.0867249382338 + - type: nauc_recall_at_5_std + value: 11.565786080587733 + - type: ndcg_at_1 + value: 15.0 + - type: ndcg_at_10 + value: 12.386999999999999 + - type: ndcg_at_100 + value: 18.533 + - type: ndcg_at_1000 + value: 23.955000000000002 + - type: ndcg_at_20 + value: 14.459 + - type: ndcg_at_3 + value: 11.75 + - type: ndcg_at_5 + value: 10.285 + - type: precision_at_1 + value: 15.0 + - type: precision_at_10 + value: 6.36 + - type: precision_at_100 + value: 1.528 + - type: precision_at_1000 + value: 0.28300000000000003 + - type: precision_at_20 + value: 4.375 + - type: precision_at_3 + value: 10.767 + - type: precision_at_5 + value: 8.9 + - type: recall_at_1 + value: 3.053 + - type: recall_at_10 + value: 12.873000000000001 + - type: recall_at_100 + value: 30.982 + - type: recall_at_1000 + value: 57.489999999999995 + - type: recall_at_20 + value: 17.718 + - type: recall_at_3 + value: 6.553000000000001 + - type: recall_at_5 + value: 9.013 + task: + type: Retrieval + - dataset: + config: default + name: MTEB SICK-R (default) + revision: 20a6d6f312dd54037fe07a32d58e5e168867909d + split: test + type: mteb/sickr-sts + metrics: + - type: cosine_pearson + value: 75.67336823619708 + - type: cosine_spearman + value: 64.6753400763881 + - type: euclidean_pearson + value: 69.13481550039579 + - type: euclidean_spearman + value: 64.6752133161514 + - type: main_score + value: 64.6753400763881 + - type: manhattan_pearson + value: 69.01619023671678 + - type: manhattan_spearman + value: 64.8728231074179 + - type: pearson + value: 75.67336823619708 + - type: spearman + value: 64.6753400763881 + task: + type: STS + - dataset: + config: default + name: MTEB STS12 (default) + revision: a0d554a64d88156834ff5ae9920b964011b16384 + split: test + type: mteb/sts12-sts + metrics: + - type: cosine_pearson + value: 72.06681927996405 + - type: cosine_spearman + value: 62.248985055530525 + - type: euclidean_pearson + value: 68.05815981894538 + - type: euclidean_spearman + value: 62.248985055530525 + - type: main_score + value: 62.248985055530525 + - type: manhattan_pearson + value: 66.68543185400786 + - type: manhattan_spearman + value: 61.43850654925033 + - type: pearson + value: 72.06681927996405 + - type: spearman + value: 62.248985055530525 + task: + type: STS + - dataset: + config: default + name: MTEB STS13 (default) + revision: 7e90230a92c190f1bf69ae9002b8cea547a64cca + split: test + type: mteb/sts13-sts + metrics: + - type: cosine_pearson + value: 76.53983680018591 + - type: cosine_spearman + value: 77.27600787572996 + - type: euclidean_pearson + value: 76.77960647262235 + - type: euclidean_spearman + value: 77.27600787572996 + - type: main_score + value: 77.27600787572996 + - type: manhattan_pearson + value: 76.37651436440808 + - type: manhattan_spearman + value: 76.85568457177312 + - type: pearson + value: 76.53983680018591 + - type: spearman + value: 77.27600787572996 + task: + type: STS + - dataset: + config: default + name: MTEB STS14 (default) + revision: 6031580fec1f6af667f0bd2da0a551cf4f0b2375 + split: test + type: mteb/sts14-sts + metrics: + - type: cosine_pearson + value: 76.20854411766629 + - type: cosine_spearman + value: 71.914099628002 + - type: euclidean_pearson + value: 74.5273047891339 + - type: euclidean_spearman + value: 71.914099628002 + - type: main_score + value: 71.914099628002 + - type: manhattan_pearson + value: 74.53275458017302 + - type: manhattan_spearman + value: 71.9720930787841 + - type: pearson + value: 76.20854411766629 + - type: spearman + value: 71.914099628002 + task: + type: STS + - dataset: + config: default + name: MTEB STS15 (default) + revision: ae752c7c21bf194d8b67fd573edf7ae58183cbe3 + split: test + type: mteb/sts15-sts + metrics: + - type: cosine_pearson + value: 79.24273419832653 + - type: cosine_spearman + value: 79.75345871163103 + - type: euclidean_pearson + value: 79.31395801169265 + - type: euclidean_spearman + value: 79.75345871163103 + - type: main_score + value: 79.75345871163103 + - type: manhattan_pearson + value: 79.24199238927697 + - type: manhattan_spearman + value: 79.64058599210834 + - type: pearson + value: 79.24273419832653 + - type: spearman + value: 79.75345871163103 + task: + type: STS + - dataset: + config: default + name: MTEB STS16 (default) + revision: 4d8694f8f0e0100860b497b999b3dbed754a0513 + split: test + type: mteb/sts16-sts + metrics: + - type: cosine_pearson + value: 75.64452330127995 + - type: cosine_spearman + value: 76.26343823222666 + - type: euclidean_pearson + value: 75.64112047932008 + - type: euclidean_spearman + value: 76.26343823222666 + - type: main_score + value: 76.26343823222666 + - type: manhattan_pearson + value: 75.32718809126764 + - type: manhattan_spearman + value: 75.9420892784719 + - type: pearson + value: 75.64452330127995 + - type: spearman + value: 76.26343823222666 + task: + type: STS + - dataset: + config: es-en + name: MTEB STS17 (es-en) + revision: faeb762787bd10488a50c8b5be4a3b82e411949c + split: test + type: mteb/sts17-crosslingual-sts + metrics: + - type: cosine_pearson + value: 17.52217310066287 + - type: cosine_spearman + value: 14.729958484232528 + - type: euclidean_pearson + value: 17.507234354096582 + - type: euclidean_spearman + value: 14.729958484232528 + - type: main_score + value: 14.729958484232528 + - type: manhattan_pearson + value: 15.286020788097272 + - type: manhattan_spearman + value: 11.320242312589713 + - type: pearson + value: 17.52217310066287 + - type: spearman + value: 14.729958484232528 + task: + type: STS + - dataset: + config: en-en + name: MTEB STS17 (en-en) + revision: faeb762787bd10488a50c8b5be4a3b82e411949c + split: test + type: mteb/sts17-crosslingual-sts + metrics: + - type: cosine_pearson + value: 84.67406984717113 + - type: cosine_spearman + value: 85.96709815630739 + - type: euclidean_pearson + value: 84.7186375682207 + - type: euclidean_spearman + value: 85.96709815630739 + - type: main_score + value: 85.96709815630739 + - type: manhattan_pearson + value: 85.07894758059129 + - type: manhattan_spearman + value: 86.57110045700985 + - type: pearson + value: 84.67406984717113 + - type: spearman + value: 85.96709815630739 + task: + type: STS + - dataset: + config: fr-en + name: MTEB STS17 (fr-en) + revision: faeb762787bd10488a50c8b5be4a3b82e411949c + split: test + type: mteb/sts17-crosslingual-sts + metrics: + - type: cosine_pearson + value: 36.02331692863771 + - type: cosine_spearman + value: 34.28540470062557 + - type: euclidean_pearson + value: 35.996881386631514 + - type: euclidean_spearman + value: 34.28540470062557 + - type: main_score + value: 34.28540470062557 + - type: manhattan_pearson + value: 35.47246063445784 + - type: manhattan_spearman + value: 34.83247787211397 + - type: pearson + value: 36.02331692863771 + - type: spearman + value: 34.28540470062557 + task: + type: STS + - dataset: + config: en-tr + name: MTEB STS17 (en-tr) + revision: faeb762787bd10488a50c8b5be4a3b82e411949c + split: test + type: mteb/sts17-crosslingual-sts + metrics: + - type: cosine_pearson + value: 13.925983981770388 + - type: cosine_spearman + value: 11.193291331109325 + - type: euclidean_pearson + value: 13.9151651239108 + - type: euclidean_spearman + value: 11.193291331109325 + - type: main_score + value: 11.193291331109325 + - type: manhattan_pearson + value: 12.652407957594654 + - type: manhattan_spearman + value: 9.888358907769014 + - type: pearson + value: 13.925983981770388 + - type: spearman + value: 11.193291331109325 + task: + type: STS + - dataset: + config: en-de + name: MTEB STS17 (en-de) + revision: faeb762787bd10488a50c8b5be4a3b82e411949c + split: test + type: mteb/sts17-crosslingual-sts + metrics: + - type: cosine_pearson + value: 26.77839285232968 + - type: cosine_spearman + value: 23.010015986939717 + - type: euclidean_pearson + value: 27.13668235790385 + - type: euclidean_spearman + value: 23.010015986939717 + - type: main_score + value: 23.010015986939717 + - type: manhattan_pearson + value: 27.02698710744775 + - type: manhattan_spearman + value: 23.038730409304936 + - type: pearson + value: 26.77839285232968 + - type: spearman + value: 23.010015986939717 + task: + type: STS + - dataset: + config: it-en + name: MTEB STS17 (it-en) + revision: faeb762787bd10488a50c8b5be4a3b82e411949c + split: test + type: mteb/sts17-crosslingual-sts + metrics: + - type: cosine_pearson + value: 25.330935194314364 + - type: cosine_spearman + value: 23.143555348782797 + - type: euclidean_pearson + value: 24.670147594978143 + - type: euclidean_spearman + value: 23.143555348782797 + - type: main_score + value: 23.143555348782797 + - type: manhattan_pearson + value: 24.879695698914418 + - type: manhattan_spearman + value: 25.916630507885134 + - type: pearson + value: 25.330935194314364 + - type: spearman + value: 23.143555348782797 + task: + type: STS + - dataset: + config: en-ar + name: MTEB STS17 (en-ar) + revision: faeb762787bd10488a50c8b5be4a3b82e411949c + split: test + type: mteb/sts17-crosslingual-sts + metrics: + - type: cosine_pearson + value: 6.61651078645899 + - type: cosine_spearman + value: 5.415104433010482 + - type: euclidean_pearson + value: 6.791575957480809 + - type: euclidean_spearman + value: 5.415104433010482 + - type: main_score + value: 5.415104433010482 + - type: manhattan_pearson + value: 3.6585407382250987 + - type: manhattan_spearman + value: 4.566044103659472 + - type: pearson + value: 6.61651078645899 + - type: spearman + value: 5.415104433010482 + task: + type: STS + - dataset: + config: nl-en + name: MTEB STS17 (nl-en) + revision: faeb762787bd10488a50c8b5be4a3b82e411949c + split: test + type: mteb/sts17-crosslingual-sts + metrics: + - type: cosine_pearson + value: 32.718045784523184 + - type: cosine_spearman + value: 27.52844368619317 + - type: euclidean_pearson + value: 32.98978359596458 + - type: euclidean_spearman + value: 27.52844368619317 + - type: main_score + value: 27.52844368619317 + - type: manhattan_pearson + value: 35.57923949366344 + - type: manhattan_spearman + value: 34.27137422651138 + - type: pearson + value: 32.718045784523184 + - type: spearman + value: 27.52844368619317 + task: + type: STS + - dataset: + config: es-en + name: MTEB STS22 (es-en) + revision: de9d86b3b84231dc21f76c7b7af1f28e2f57f6e3 + split: test + type: mteb/sts22-crosslingual-sts + metrics: + - type: cosine_pearson + value: 9.98410299881163 + - type: cosine_spearman + value: 10.98684405086525 + - type: euclidean_pearson + value: 9.461680781495218 + - type: euclidean_spearman + value: 10.9925413190658 + - type: main_score + value: 10.98684405086525 + - type: manhattan_pearson + value: 9.442055271895944 + - type: manhattan_spearman + value: 11.226101908391069 + - type: pearson + value: 9.98410299881163 + - type: spearman + value: 10.98684405086525 + task: + type: STS + - dataset: + config: en + name: MTEB STS22 (en) + revision: de9d86b3b84231dc21f76c7b7af1f28e2f57f6e3 + split: test + type: mteb/sts22-crosslingual-sts + metrics: + - type: cosine_pearson + value: 59.3180680265132 + - type: cosine_spearman + value: 63.07956002739231 + - type: euclidean_pearson + value: 62.46424835000928 + - type: euclidean_spearman + value: 63.07956002739231 + - type: main_score + value: 63.07956002739231 + - type: manhattan_pearson + value: 62.048137683643766 + - type: manhattan_spearman + value: 61.83898606879604 + - type: pearson + value: 59.3180680265132 + - type: spearman + value: 63.07956002739231 + task: + type: STS + - dataset: + config: de-en + name: MTEB STS22 (de-en) + revision: de9d86b3b84231dc21f76c7b7af1f28e2f57f6e3 + split: test + type: mteb/sts22-crosslingual-sts + metrics: + - type: cosine_pearson + value: 29.061215770374826 + - type: cosine_spearman + value: 36.21441725938738 + - type: euclidean_pearson + value: 28.44045530150387 + - type: euclidean_spearman + value: 36.21441725938738 + - type: main_score + value: 36.21441725938738 + - type: manhattan_pearson + value: 29.32403221599612 + - type: manhattan_spearman + value: 38.914481153396494 + - type: pearson + value: 29.061215770374826 + - type: spearman + value: 36.21441725938738 + task: + type: STS + - dataset: + config: zh-en + name: MTEB STS22 (zh-en) + revision: de9d86b3b84231dc21f76c7b7af1f28e2f57f6e3 + split: test + type: mteb/sts22-crosslingual-sts + metrics: + - type: cosine_pearson + value: 11.266385865086239 + - type: cosine_spearman + value: 17.291293843893733 + - type: euclidean_pearson + value: 10.045897285683115 + - type: euclidean_spearman + value: 17.321323804048646 + - type: main_score + value: 17.291293843893733 + - type: manhattan_pearson + value: 15.333482209624194 + - type: manhattan_spearman + value: 20.399166731513915 + - type: pearson + value: 11.266385865086239 + - type: spearman + value: 17.291293843893733 + task: + type: STS + - dataset: + config: pl-en + name: MTEB STS22 (pl-en) + revision: de9d86b3b84231dc21f76c7b7af1f28e2f57f6e3 + split: test + type: mteb/sts22-crosslingual-sts + metrics: + - type: cosine_pearson + value: 9.647587208410648 + - type: cosine_spearman + value: 21.33739699413266 + - type: euclidean_pearson + value: 7.451981822243237 + - type: euclidean_spearman + value: 21.33739699413266 + - type: main_score + value: 21.33739699413266 + - type: manhattan_pearson + value: 10.05280275870948 + - type: manhattan_spearman + value: 22.233400969472218 + - type: pearson + value: 9.647587208410648 + - type: spearman + value: 21.33739699413266 + task: + type: STS + - dataset: + config: default + name: MTEB STSBenchmark (default) + revision: b0fddb56ed78048fa8b90373c8a3cfc37b684831 + split: test + type: mteb/stsbenchmark-sts + metrics: + - type: cosine_pearson + value: 77.2598255013409 + - type: cosine_spearman + value: 75.40519061413276 + - type: euclidean_pearson + value: 77.19878276657876 + - type: euclidean_spearman + value: 75.40519061413276 + - type: main_score + value: 75.40519061413276 + - type: manhattan_pearson + value: 77.04099640594512 + - type: manhattan_spearman + value: 75.32219501493076 + - type: pearson + value: 77.2598255013409 + - type: spearman + value: 75.40519061413276 + task: + type: STS + - dataset: + config: default + name: MTEB SciDocsRR (default) + revision: d3c5e1fc0b855ab6097bf1cda04dd73947d7caab + split: test + type: mteb/scidocs-reranking + metrics: + - type: main_score + value: 72.10127087089839 + - type: map + value: 72.10127087089839 + - type: mrr + value: 90.62288020621355 + - type: nAUC_map_diff1 + value: 8.726677558277695 + - type: nAUC_map_max + value: 54.59636736704295 + - type: nAUC_map_std + value: 67.36367052533402 + - type: nAUC_mrr_diff1 + value: 47.77588337162405 + - type: nAUC_mrr_max + value: 74.90946175462605 + - type: nAUC_mrr_std + value: 71.81332269641806 + task: + type: Reranking + - dataset: + config: default + name: MTEB SciFact (default) + revision: 0228b52cf27578f30900b9e5271d331663a030d7 + split: test + type: mteb/scifact + metrics: + - type: main_score + value: 50.63999999999999 + - type: map_at_1 + value: 35.5 + - type: map_at_10 + value: 45.238 + - type: map_at_100 + value: 46.135999999999996 + - type: map_at_1000 + value: 46.181 + - type: map_at_20 + value: 45.767 + - type: map_at_3 + value: 42.329 + - type: map_at_5 + value: 44.054 + - type: mrr_at_1 + value: 37.666666666666664 + - type: mrr_at_10 + value: 46.6611111111111 + - type: mrr_at_100 + value: 47.37819687814183 + - type: mrr_at_1000 + value: 47.417644921595766 + - type: mrr_at_20 + value: 47.06856780130773 + - type: mrr_at_3 + value: 43.94444444444443 + - type: mrr_at_5 + value: 45.52777777777777 + - type: nauc_map_at_1000_diff1 + value: 52.83081390161976 + - type: nauc_map_at_1000_max + value: 37.21621852995913 + - type: nauc_map_at_1000_std + value: -3.416369626271914 + - type: nauc_map_at_100_diff1 + value: 52.823502489139884 + - type: nauc_map_at_100_max + value: 37.2435733087758 + - type: nauc_map_at_100_std + value: -3.376708460074628 + - type: nauc_map_at_10_diff1 + value: 52.495695868970785 + - type: nauc_map_at_10_max + value: 36.79244353087952 + - type: nauc_map_at_10_std + value: -3.998841918813238 + - type: nauc_map_at_1_diff1 + value: 55.20714819661926 + - type: nauc_map_at_1_max + value: 33.68583272500883 + - type: nauc_map_at_1_std + value: -7.806502386166579 + - type: nauc_map_at_20_diff1 + value: 52.82557233788675 + - type: nauc_map_at_20_max + value: 37.02532534485883 + - type: nauc_map_at_20_std + value: -3.6962702134516126 + - type: nauc_map_at_3_diff1 + value: 53.005833884053054 + - type: nauc_map_at_3_max + value: 35.102473883265056 + - type: nauc_map_at_3_std + value: -6.237364868462919 + - type: nauc_map_at_5_diff1 + value: 52.67151253564545 + - type: nauc_map_at_5_max + value: 36.083416260083574 + - type: nauc_map_at_5_std + value: -4.7023113318143785 + - type: nauc_mrr_at_1000_diff1 + value: 52.938698102997094 + - type: nauc_mrr_at_1000_max + value: 39.46705187537523 + - type: nauc_mrr_at_1000_std + value: 0.6163818152860598 + - type: nauc_mrr_at_100_diff1 + value: 52.93491193041612 + - type: nauc_mrr_at_100_max + value: 39.490426719059165 + - type: nauc_mrr_at_100_std + value: 0.6662007971949842 + - type: nauc_mrr_at_10_diff1 + value: 52.70216069864656 + - type: nauc_mrr_at_10_max + value: 39.52193808791504 + - type: nauc_mrr_at_10_std + value: 0.536595037291294 + - type: nauc_mrr_at_1_diff1 + value: 55.77100806609076 + - type: nauc_mrr_at_1_max + value: 37.966164940491446 + - type: nauc_mrr_at_1_std + value: -2.1074234936282537 + - type: nauc_mrr_at_20_diff1 + value: 52.942136130524986 + - type: nauc_mrr_at_20_max + value: 39.42716448302782 + - type: nauc_mrr_at_20_std + value: 0.5472281187662744 + - type: nauc_mrr_at_3_diff1 + value: 53.144295072591206 + - type: nauc_mrr_at_3_max + value: 38.05294316134295 + - type: nauc_mrr_at_3_std + value: -1.2360608664776096 + - type: nauc_mrr_at_5_diff1 + value: 52.789220500594205 + - type: nauc_mrr_at_5_max + value: 38.83395427252616 + - type: nauc_mrr_at_5_std + value: -0.09099470685601964 + - type: nauc_ndcg_at_1000_diff1 + value: 52.16867590195915 + - type: nauc_ndcg_at_1000_max + value: 39.70115643730131 + - type: nauc_ndcg_at_1000_std + value: 0.904258507053096 + - type: nauc_ndcg_at_100_diff1 + value: 51.87328245345757 + - type: nauc_ndcg_at_100_max + value: 40.59055338026654 + - type: nauc_ndcg_at_100_std + value: 2.554356951645788 + - type: nauc_ndcg_at_10_diff1 + value: 50.809281234563805 + - type: nauc_ndcg_at_10_max + value: 39.085094925973245 + - type: nauc_ndcg_at_10_std + value: -0.23387754671232033 + - type: nauc_ndcg_at_1_diff1 + value: 55.77100806609076 + - type: nauc_ndcg_at_1_max + value: 37.966164940491446 + - type: nauc_ndcg_at_1_std + value: -2.1074234936282537 + - type: nauc_ndcg_at_20_diff1 + value: 51.74864887078553 + - type: nauc_ndcg_at_20_max + value: 39.32033115509482 + - type: nauc_ndcg_at_20_std + value: 0.4346356935494506 + - type: nauc_ndcg_at_3_diff1 + value: 51.9909705702443 + - type: nauc_ndcg_at_3_max + value: 36.078476037019094 + - type: nauc_ndcg_at_3_std + value: -4.014502363911228 + - type: nauc_ndcg_at_5_diff1 + value: 51.312788955634325 + - type: nauc_ndcg_at_5_max + value: 37.54290824294073 + - type: nauc_ndcg_at_5_std + value: -1.8169251273098448 + - type: nauc_precision_at_1000_diff1 + value: 1.4596703970072096 + - type: nauc_precision_at_1000_max + value: 36.408552907408 + - type: nauc_precision_at_1000_std + value: 53.892991905053776 + - type: nauc_precision_at_100_diff1 + value: 17.90829681479967 + - type: nauc_precision_at_100_max + value: 50.02058762977557 + - type: nauc_precision_at_100_std + value: 50.95242296795188 + - type: nauc_precision_at_10_diff1 + value: 33.69533492770854 + - type: nauc_precision_at_10_max + value: 47.554637845938025 + - type: nauc_precision_at_10_std + value: 21.812883074791838 + - type: nauc_precision_at_1_diff1 + value: 55.77100806609076 + - type: nauc_precision_at_1_max + value: 37.966164940491446 + - type: nauc_precision_at_1_std + value: -2.1074234936282537 + - type: nauc_precision_at_20_diff1 + value: 31.797703948512723 + - type: nauc_precision_at_20_max + value: 46.94077230822751 + - type: nauc_precision_at_20_std + value: 29.525569664289396 + - type: nauc_precision_at_3_diff1 + value: 41.753151429999456 + - type: nauc_precision_at_3_max + value: 38.30163209243931 + - type: nauc_precision_at_3_std + value: 6.19935377482869 + - type: nauc_precision_at_5_diff1 + value: 38.479320931912575 + - type: nauc_precision_at_5_max + value: 41.576866734894516 + - type: nauc_precision_at_5_std + value: 13.327714566652604 + - type: nauc_recall_at_1000_diff1 + value: 50.28923446773287 + - type: nauc_recall_at_1000_max + value: 68.29528746364413 + - type: nauc_recall_at_1000_std + value: 48.2313231806132 + - type: nauc_recall_at_100_diff1 + value: 46.22085619290839 + - type: nauc_recall_at_100_max + value: 61.60933703216747 + - type: nauc_recall_at_100_std + value: 42.210649980610896 + - type: nauc_recall_at_10_diff1 + value: 43.10485234893865 + - type: nauc_recall_at_10_max + value: 43.06779802776641 + - type: nauc_recall_at_10_std + value: 8.272818985431385 + - type: nauc_recall_at_1_diff1 + value: 55.20714819661926 + - type: nauc_recall_at_1_max + value: 33.68583272500883 + - type: nauc_recall_at_1_std + value: -7.806502386166579 + - type: nauc_recall_at_20_diff1 + value: 46.850902149595036 + - type: nauc_recall_at_20_max + value: 44.58623368637416 + - type: nauc_recall_at_20_std + value: 11.890054420031708 + - type: nauc_recall_at_3_diff1 + value: 48.80301236823221 + - type: nauc_recall_at_3_max + value: 34.177890037375 + - type: nauc_recall_at_3_std + value: -3.852215004054359 + - type: nauc_recall_at_5_diff1 + value: 46.206941308622056 + - type: nauc_recall_at_5_max + value: 38.61994260176494 + - type: nauc_recall_at_5_std + value: 2.735469769782116 + - type: ndcg_at_1 + value: 37.667 + - type: ndcg_at_10 + value: 50.63999999999999 + - type: ndcg_at_100 + value: 54.885 + - type: ndcg_at_1000 + value: 56.274 + - type: ndcg_at_20 + value: 52.349000000000004 + - type: ndcg_at_3 + value: 44.891999999999996 + - type: ndcg_at_5 + value: 47.788000000000004 + - type: precision_at_1 + value: 37.667 + - type: precision_at_10 + value: 7.3 + - type: precision_at_100 + value: 0.97 + - type: precision_at_1000 + value: 0.11 + - type: precision_at_20 + value: 4.067 + - type: precision_at_3 + value: 18.333 + - type: precision_at_5 + value: 12.6 + - type: recall_at_1 + value: 35.5 + - type: recall_at_10 + value: 66.178 + - type: recall_at_100 + value: 85.9 + - type: recall_at_1000 + value: 97.1 + - type: recall_at_20 + value: 72.60600000000001 + - type: recall_at_3 + value: 50.306 + - type: recall_at_5 + value: 57.443999999999996 + task: + type: Retrieval + - dataset: + config: default + name: MTEB SprintDuplicateQuestions (default) + revision: d66bd1f72af766a5cc4b0ca5e00c162f89e8cc46 + split: test + type: mteb/sprintduplicatequestions-pairclassification + metrics: + - type: cosine_accuracy + value: 99.71386138613862 + - type: cosine_accuracy_threshold + value: 78.56961662426235 + - type: cosine_ap + value: 90.20131927652946 + - type: cosine_f1 + value: 84.7749114820435 + - type: cosine_f1_threshold + value: 75.7768544371973 + - type: cosine_precision + value: 85.7727737973388 + - type: cosine_recall + value: 83.8 + - type: dot_accuracy + value: 99.71386138613862 + - type: dot_accuracy_threshold + value: 78.56961780669964 + - type: dot_ap + value: 90.20131927652946 + - type: dot_f1 + value: 84.7749114820435 + - type: dot_f1_threshold + value: 75.77685228378391 + - type: dot_precision + value: 85.7727737973388 + - type: dot_recall + value: 83.8 + - type: euclidean_accuracy + value: 99.71386138613862 + - type: euclidean_accuracy_threshold + value: 65.46813529720524 + - type: euclidean_ap + value: 90.20131927652946 + - type: euclidean_f1 + value: 84.7749114820435 + - type: euclidean_f1_threshold + value: 69.60336608830053 + - type: euclidean_precision + value: 85.7727737973388 + - type: euclidean_recall + value: 83.8 + - type: main_score + value: 90.20131927652946 + - type: manhattan_accuracy + value: 99.7059405940594 + - type: manhattan_accuracy_threshold + value: 804.8100425289704 + - type: manhattan_ap + value: 90.00682250828237 + - type: manhattan_f1 + value: 84.44211629125196 + - type: manhattan_f1_threshold + value: 828.8486447498144 + - type: manhattan_precision + value: 88.66886688668868 + - type: manhattan_recall + value: 80.60000000000001 + - type: max_accuracy + value: 99.71386138613862 + - type: max_ap + value: 90.20131927652946 + - type: max_f1 + value: 84.7749114820435 + - type: max_precision + value: 88.66886688668868 + - type: max_recall + value: 83.8 + - type: similarity_accuracy + value: 99.71386138613862 + - type: similarity_accuracy_threshold + value: 78.56961662426235 + - type: similarity_ap + value: 90.20131927652946 + - type: similarity_f1 + value: 84.7749114820435 + - type: similarity_f1_threshold + value: 75.7768544371973 + - type: similarity_precision + value: 85.7727737973388 + - type: similarity_recall + value: 83.8 + task: + type: PairClassification + - dataset: + config: default + name: MTEB StackExchangeClustering (default) + revision: 6cbc1f7b2bc0622f2e39d2c77fa502909748c259 + split: test + type: mteb/stackexchange-clustering + metrics: + - type: main_score + value: 48.18939518021159 + - type: v_measure + value: 48.18939518021159 + - type: v_measure_std + value: 4.6189444340187995 + task: + type: Clustering + - dataset: + config: default + name: MTEB StackExchangeClusteringP2P (default) + revision: 815ca46b2622cec33ccafc3735d572c266efdb44 + split: test + type: mteb/stackexchange-clustering-p2p + metrics: + - type: main_score + value: 30.743938802421265 + - type: v_measure + value: 30.743938802421265 + - type: v_measure_std + value: 1.4645401677053824 + task: + type: Clustering + - dataset: + config: default + name: MTEB StackOverflowDupQuestions (default) + revision: e185fbe320c72810689fc5848eb6114e1ef5ec69 + split: test + type: mteb/stackoverflowdupquestions-reranking + metrics: + - type: main_score + value: 43.254152892780986 + - type: map + value: 43.254152892780986 + - type: mrr + value: 43.70483989050165 + - type: nAUC_map_diff1 + value: 33.22453777168869 + - type: nAUC_map_max + value: 13.175366935671228 + - type: nAUC_map_std + value: 3.718253924398536 + - type: nAUC_mrr_diff1 + value: 32.58818809467491 + - type: nAUC_mrr_max + value: 14.093758435205075 + - type: nAUC_mrr_std + value: 4.198791420159734 + task: + type: Reranking + - dataset: + config: default + name: MTEB SummEval (default) + revision: cda12ad7615edc362dbf25a00fdd61d3b1eaf93c + split: test + type: mteb/summeval + metrics: + - type: cosine_pearson + value: 29.88360050203766 + - type: cosine_spearman + value: 29.275185932109494 + - type: dot_pearson + value: 29.883597746108975 + - type: dot_spearman + value: 29.28377974870949 + - type: main_score + value: 29.275185932109494 + - type: pearson + value: 29.88360050203766 + - type: spearman + value: 29.275185932109494 + task: + type: Summarization + - dataset: + config: default + name: MTEB TRECCOVID (default) + revision: bb9466bac8153a0349341eb1b22e06409e78ef4e + split: test + type: mteb/trec-covid + metrics: + - type: main_score + value: 45.747 + - type: map_at_1 + value: 0.148 + - type: map_at_10 + value: 0.972 + - type: map_at_100 + value: 4.652 + - type: map_at_1000 + value: 11.511000000000001 + - type: map_at_20 + value: 1.643 + - type: map_at_3 + value: 0.369 + - type: map_at_5 + value: 0.561 + - type: mrr_at_1 + value: 62.0 + - type: mrr_at_10 + value: 70.06904761904761 + - type: mrr_at_100 + value: 70.45500059672992 + - type: mrr_at_1000 + value: 70.45500059672992 + - type: mrr_at_20 + value: 70.31716791979949 + - type: mrr_at_3 + value: 68.0 + - type: mrr_at_5 + value: 69.19999999999999 + - type: nauc_map_at_1000_diff1 + value: -0.8266899821302324 + - type: nauc_map_at_1000_max + value: 34.62914536640893 + - type: nauc_map_at_1000_std + value: 57.177693387251615 + - type: nauc_map_at_100_diff1 + value: -3.3097934383165613 + - type: nauc_map_at_100_max + value: 22.052336613600293 + - type: nauc_map_at_100_std + value: 29.905360060478188 + - type: nauc_map_at_10_diff1 + value: 6.057035481050755 + - type: nauc_map_at_10_max + value: 22.742824418774667 + - type: nauc_map_at_10_std + value: 5.649441588476496 + - type: nauc_map_at_1_diff1 + value: 10.469485578180873 + - type: nauc_map_at_1_max + value: 4.582098501050435 + - type: nauc_map_at_1_std + value: -10.47482550446343 + - type: nauc_map_at_20_diff1 + value: 1.5813367839245727 + - type: nauc_map_at_20_max + value: 25.09380802651507 + - type: nauc_map_at_20_std + value: 11.733045886140895 + - type: nauc_map_at_3_diff1 + value: -0.4174848325628528 + - type: nauc_map_at_3_max + value: 16.54291715633098 + - type: nauc_map_at_3_std + value: -6.315368365719176 + - type: nauc_map_at_5_diff1 + value: 1.6439114449809122 + - type: nauc_map_at_5_max + value: 18.119472468345634 + - type: nauc_map_at_5_std + value: -1.4642215840068935 + - type: nauc_mrr_at_1000_diff1 + value: 19.962304210632194 + - type: nauc_mrr_at_1000_max + value: 28.66281052259736 + - type: nauc_mrr_at_1000_std + value: 14.4833499197582 + - type: nauc_mrr_at_100_diff1 + value: 19.962304210632194 + - type: nauc_mrr_at_100_max + value: 28.66281052259736 + - type: nauc_mrr_at_100_std + value: 14.4833499197582 + - type: nauc_mrr_at_10_diff1 + value: 19.79498540271038 + - type: nauc_mrr_at_10_max + value: 28.07551011390951 + - type: nauc_mrr_at_10_std + value: 13.820791565247939 + - type: nauc_mrr_at_1_diff1 + value: 23.72088730271045 + - type: nauc_mrr_at_1_max + value: 29.338830261821947 + - type: nauc_mrr_at_1_std + value: 10.463649509276033 + - type: nauc_mrr_at_20_diff1 + value: 20.06776286940325 + - type: nauc_mrr_at_20_max + value: 28.69272909781133 + - type: nauc_mrr_at_20_std + value: 14.560673636667628 + - type: nauc_mrr_at_3_diff1 + value: 18.71166001912622 + - type: nauc_mrr_at_3_max + value: 30.645161290322555 + - type: nauc_mrr_at_3_std + value: 16.37394164159257 + - type: nauc_mrr_at_5_diff1 + value: 15.791374902745353 + - type: nauc_mrr_at_5_max + value: 28.51602708149093 + - type: nauc_mrr_at_5_std + value: 15.246386476651619 + - type: nauc_ndcg_at_1000_diff1 + value: -5.179304837164554 + - type: nauc_ndcg_at_1000_max + value: 27.27301986190763 + - type: nauc_ndcg_at_1000_std + value: 49.239144813886654 + - type: nauc_ndcg_at_100_diff1 + value: 7.283019925558149 + - type: nauc_ndcg_at_100_max + value: 29.80340187562149 + - type: nauc_ndcg_at_100_std + value: 47.60799676958296 + - type: nauc_ndcg_at_10_diff1 + value: 11.621471677557253 + - type: nauc_ndcg_at_10_max + value: 31.78727749460396 + - type: nauc_ndcg_at_10_std + value: 26.339328462146177 + - type: nauc_ndcg_at_1_diff1 + value: 26.896384303421446 + - type: nauc_ndcg_at_1_max + value: 28.727080596332872 + - type: nauc_ndcg_at_1_std + value: 12.10515793682523 + - type: nauc_ndcg_at_20_diff1 + value: 7.253524538786647 + - type: nauc_ndcg_at_20_max + value: 33.412855576178295 + - type: nauc_ndcg_at_20_std + value: 34.10895211064073 + - type: nauc_ndcg_at_3_diff1 + value: 11.303112239393863 + - type: nauc_ndcg_at_3_max + value: 35.0880605283756 + - type: nauc_ndcg_at_3_std + value: 18.514877130637803 + - type: nauc_ndcg_at_5_diff1 + value: 8.537541001217583 + - type: nauc_ndcg_at_5_max + value: 32.24796400964019 + - type: nauc_ndcg_at_5_std + value: 21.65596013895985 + - type: nauc_precision_at_1000_diff1 + value: 5.217123572202896 + - type: nauc_precision_at_1000_max + value: 31.954154167309177 + - type: nauc_precision_at_1000_std + value: 60.51613061301686 + - type: nauc_precision_at_100_diff1 + value: 5.748688865778208 + - type: nauc_precision_at_100_max + value: 28.503515028630567 + - type: nauc_precision_at_100_std + value: 52.8175811950368 + - type: nauc_precision_at_10_diff1 + value: 9.634424129349284 + - type: nauc_precision_at_10_max + value: 33.90210630229416 + - type: nauc_precision_at_10_std + value: 30.197787312348073 + - type: nauc_precision_at_1_diff1 + value: 23.72088730271045 + - type: nauc_precision_at_1_max + value: 29.338830261821947 + - type: nauc_precision_at_1_std + value: 10.463649509276033 + - type: nauc_precision_at_20_diff1 + value: 2.6440820197838923 + - type: nauc_precision_at_20_max + value: 36.6927642980172 + - type: nauc_precision_at_20_std + value: 40.53918258763216 + - type: nauc_precision_at_3_diff1 + value: 2.9773659425793695 + - type: nauc_precision_at_3_max + value: 35.63522203655881 + - type: nauc_precision_at_3_std + value: 17.365942579371055 + - type: nauc_precision_at_5_diff1 + value: 3.883249981522982 + - type: nauc_precision_at_5_max + value: 34.19785174053362 + - type: nauc_precision_at_5_std + value: 25.391096548495977 + - type: nauc_recall_at_1000_diff1 + value: -10.977265624215267 + - type: nauc_recall_at_1000_max + value: 22.349720150932985 + - type: nauc_recall_at_1000_std + value: 47.14118127199015 + - type: nauc_recall_at_100_diff1 + value: -10.566105105889243 + - type: nauc_recall_at_100_max + value: 13.59897332326766 + - type: nauc_recall_at_100_std + value: 25.1260269383207 + - type: nauc_recall_at_10_diff1 + value: 3.9418824014124514 + - type: nauc_recall_at_10_max + value: 18.87305117920693 + - type: nauc_recall_at_10_std + value: 4.227456274746917 + - type: nauc_recall_at_1_diff1 + value: 10.469485578180873 + - type: nauc_recall_at_1_max + value: 4.582098501050435 + - type: nauc_recall_at_1_std + value: -10.47482550446343 + - type: nauc_recall_at_20_diff1 + value: -3.663384950691917 + - type: nauc_recall_at_20_max + value: 20.838703493064635 + - type: nauc_recall_at_20_std + value: 10.729793670370862 + - type: nauc_recall_at_3_diff1 + value: -1.1850402683856456 + - type: nauc_recall_at_3_max + value: 16.033671610288522 + - type: nauc_recall_at_3_std + value: -6.953520529126048 + - type: nauc_recall_at_5_diff1 + value: -0.5156927662191768 + - type: nauc_recall_at_5_max + value: 15.556954479927315 + - type: nauc_recall_at_5_std + value: -2.965229848389009 + - type: ndcg_at_1 + value: 56.00000000000001 + - type: ndcg_at_10 + value: 45.747 + - type: ndcg_at_100 + value: 32.761 + - type: ndcg_at_1000 + value: 29.633 + - type: ndcg_at_20 + value: 42.905 + - type: ndcg_at_3 + value: 50.641999999999996 + - type: ndcg_at_5 + value: 48.231 + - type: precision_at_1 + value: 62.0 + - type: precision_at_10 + value: 47.8 + - type: precision_at_100 + value: 33.72 + - type: precision_at_1000 + value: 14.238000000000001 + - type: precision_at_20 + value: 45.2 + - type: precision_at_3 + value: 54.0 + - type: precision_at_5 + value: 50.8 + - type: recall_at_1 + value: 0.148 + - type: recall_at_10 + value: 1.143 + - type: recall_at_100 + value: 7.219 + - type: recall_at_1000 + value: 28.294999999999998 + - type: recall_at_20 + value: 2.083 + - type: recall_at_3 + value: 0.395 + - type: recall_at_5 + value: 0.628 + task: + type: Retrieval + - dataset: + config: default + name: MTEB Touche2020 (default) + revision: a34f9a33db75fa0cbb21bb5cfc3dae8dc8bec93f + split: test + type: mteb/touche2020 + metrics: + - type: main_score + value: 18.618000000000002 + - type: map_at_1 + value: 1.22 + - type: map_at_10 + value: 6.635000000000001 + - type: map_at_100 + value: 10.873 + - type: map_at_1000 + value: 12.415 + - type: map_at_20 + value: 8.334 + - type: map_at_3 + value: 2.8240000000000003 + - type: map_at_5 + value: 4.111 + - type: mrr_at_1 + value: 14.285714285714285 + - type: mrr_at_10 + value: 31.959831551668284 + - type: mrr_at_100 + value: 33.15059576942869 + - type: mrr_at_1000 + value: 33.15059576942869 + - type: mrr_at_20 + value: 32.685999641281754 + - type: mrr_at_3 + value: 25.850340136054424 + - type: mrr_at_5 + value: 29.31972789115646 + - type: nauc_map_at_1000_diff1 + value: 8.820920087157313 + - type: nauc_map_at_1000_max + value: -33.58280072902863 + - type: nauc_map_at_1000_std + value: -22.730292551065183 + - type: nauc_map_at_100_diff1 + value: 9.741008911531535 + - type: nauc_map_at_100_max + value: -33.6532837418042 + - type: nauc_map_at_100_std + value: -28.3444309192652 + - type: nauc_map_at_10_diff1 + value: 7.657150877271815 + - type: nauc_map_at_10_max + value: -41.7412362957407 + - type: nauc_map_at_10_std + value: -35.66062824513052 + - type: nauc_map_at_1_diff1 + value: 7.593190069621649 + - type: nauc_map_at_1_max + value: -39.58442010649443 + - type: nauc_map_at_1_std + value: -22.564719811889777 + - type: nauc_map_at_20_diff1 + value: 7.245303325270055 + - type: nauc_map_at_20_max + value: -37.804327180430946 + - type: nauc_map_at_20_std + value: -32.702756826489846 + - type: nauc_map_at_3_diff1 + value: 6.742365189818029 + - type: nauc_map_at_3_max + value: -41.7228290771728 + - type: nauc_map_at_3_std + value: -30.230168338925107 + - type: nauc_map_at_5_diff1 + value: 11.935913888588882 + - type: nauc_map_at_5_max + value: -41.39335754887243 + - type: nauc_map_at_5_std + value: -33.780157609546535 + - type: nauc_mrr_at_1000_diff1 + value: -1.6708159098532442 + - type: nauc_mrr_at_1000_max + value: -36.55890935351506 + - type: nauc_mrr_at_1000_std + value: -24.27343264470873 + - type: nauc_mrr_at_100_diff1 + value: -1.6708159098532442 + - type: nauc_mrr_at_100_max + value: -36.55890935351506 + - type: nauc_mrr_at_100_std + value: -24.27343264470873 + - type: nauc_mrr_at_10_diff1 + value: -0.42650070974468685 + - type: nauc_mrr_at_10_max + value: -37.09244916127389 + - type: nauc_mrr_at_10_std + value: -24.66093983608399 + - type: nauc_mrr_at_1_diff1 + value: -5.630573652147252 + - type: nauc_mrr_at_1_max + value: -33.616658797870684 + - type: nauc_mrr_at_1_std + value: -23.601564115907 + - type: nauc_mrr_at_20_diff1 + value: -1.832519847770416 + - type: nauc_mrr_at_20_max + value: -37.12461848720876 + - type: nauc_mrr_at_20_std + value: -24.697864546344437 + - type: nauc_mrr_at_3_diff1 + value: -0.005683436651441496 + - type: nauc_mrr_at_3_max + value: -32.50516010446863 + - type: nauc_mrr_at_3_std + value: -21.544877233050823 + - type: nauc_mrr_at_5_diff1 + value: -2.354001730958692 + - type: nauc_mrr_at_5_max + value: -32.51899298268129 + - type: nauc_mrr_at_5_std + value: -23.68035252143919 + - type: nauc_ndcg_at_1000_diff1 + value: 14.007950932108976 + - type: nauc_ndcg_at_1000_max + value: -31.274257790464837 + - type: nauc_ndcg_at_1000_std + value: 3.658749568249879 + - type: nauc_ndcg_at_100_diff1 + value: 13.626007116136158 + - type: nauc_ndcg_at_100_max + value: -35.59107319590088 + - type: nauc_ndcg_at_100_std + value: -18.874707006492024 + - type: nauc_ndcg_at_10_diff1 + value: 9.82558048538336 + - type: nauc_ndcg_at_10_max + value: -39.51461465840459 + - type: nauc_ndcg_at_10_std + value: -30.33405672804229 + - type: nauc_ndcg_at_1_diff1 + value: -1.598770159246464 + - type: nauc_ndcg_at_1_max + value: -31.975857803575675 + - type: nauc_ndcg_at_1_std + value: -18.993368614347663 + - type: nauc_ndcg_at_20_diff1 + value: 11.616460882964375 + - type: nauc_ndcg_at_20_max + value: -36.68867443298684 + - type: nauc_ndcg_at_20_std + value: -27.831158282067598 + - type: nauc_ndcg_at_3_diff1 + value: 3.6760483719742556 + - type: nauc_ndcg_at_3_max + value: -30.935030030092992 + - type: nauc_ndcg_at_3_std + value: -18.717891674270643 + - type: nauc_ndcg_at_5_diff1 + value: 10.773599917143413 + - type: nauc_ndcg_at_5_max + value: -31.08451038101287 + - type: nauc_ndcg_at_5_std + value: -25.478457258577336 + - type: nauc_precision_at_1000_diff1 + value: -6.780225586359699 + - type: nauc_precision_at_1000_max + value: 38.71975790762798 + - type: nauc_precision_at_1000_std + value: 57.8083677042306 + - type: nauc_precision_at_100_diff1 + value: 2.959136061872892 + - type: nauc_precision_at_100_max + value: -8.27764507575222 + - type: nauc_precision_at_100_std + value: 5.742410187313611 + - type: nauc_precision_at_10_diff1 + value: 9.882789695687109 + - type: nauc_precision_at_10_max + value: -31.486245698037102 + - type: nauc_precision_at_10_std + value: -29.081919554833874 + - type: nauc_precision_at_1_diff1 + value: -5.630573652147252 + - type: nauc_precision_at_1_max + value: -33.616658797870684 + - type: nauc_precision_at_1_std + value: -23.601564115907 + - type: nauc_precision_at_20_diff1 + value: 5.165999913921455 + - type: nauc_precision_at_20_max + value: -19.322665087378923 + - type: nauc_precision_at_20_std + value: -19.841805142598865 + - type: nauc_precision_at_3_diff1 + value: 2.846740832419061 + - type: nauc_precision_at_3_max + value: -30.76562032864513 + - type: nauc_precision_at_3_std + value: -23.610192672373636 + - type: nauc_precision_at_5_diff1 + value: 13.83881140180208 + - type: nauc_precision_at_5_max + value: -23.40672207825652 + - type: nauc_precision_at_5_std + value: -26.803291207458884 + - type: nauc_recall_at_1000_diff1 + value: 5.989093134294799 + - type: nauc_recall_at_1000_max + value: -23.01810906637643 + - type: nauc_recall_at_1000_std + value: 51.72967782759332 + - type: nauc_recall_at_100_diff1 + value: 9.279568158025599 + - type: nauc_recall_at_100_max + value: -32.49225165397591 + - type: nauc_recall_at_100_std + value: -14.266931753931292 + - type: nauc_recall_at_10_diff1 + value: 8.789441102892894 + - type: nauc_recall_at_10_max + value: -41.575759675933185 + - type: nauc_recall_at_10_std + value: -36.066608504981836 + - type: nauc_recall_at_1_diff1 + value: 7.593190069621649 + - type: nauc_recall_at_1_max + value: -39.58442010649443 + - type: nauc_recall_at_1_std + value: -22.564719811889777 + - type: nauc_recall_at_20_diff1 + value: 7.288095720364289 + - type: nauc_recall_at_20_max + value: -34.19747470428325 + - type: nauc_recall_at_20_std + value: -29.334755464530023 + - type: nauc_recall_at_3_diff1 + value: 7.541743741210702 + - type: nauc_recall_at_3_max + value: -38.357726279072416 + - type: nauc_recall_at_3_std + value: -29.877869977138204 + - type: nauc_recall_at_5_diff1 + value: 11.512545675995455 + - type: nauc_recall_at_5_max + value: -37.366204857623586 + - type: nauc_recall_at_5_std + value: -33.58926486109219 + - type: ndcg_at_1 + value: 12.245000000000001 + - type: ndcg_at_10 + value: 18.618000000000002 + - type: ndcg_at_100 + value: 28.488000000000003 + - type: ndcg_at_1000 + value: 41.208 + - type: ndcg_at_20 + value: 19.536 + - type: ndcg_at_3 + value: 15.045 + - type: ndcg_at_5 + value: 16.359 + - type: precision_at_1 + value: 14.285999999999998 + - type: precision_at_10 + value: 19.796 + - type: precision_at_100 + value: 6.5920000000000005 + - type: precision_at_1000 + value: 1.471 + - type: precision_at_20 + value: 15.204 + - type: precision_at_3 + value: 18.367 + - type: precision_at_5 + value: 18.776 + - type: recall_at_1 + value: 1.22 + - type: recall_at_10 + value: 13.763 + - type: recall_at_100 + value: 40.107 + - type: recall_at_1000 + value: 79.06800000000001 + - type: recall_at_20 + value: 20.049 + - type: recall_at_3 + value: 4.2540000000000004 + - type: recall_at_5 + value: 7.142999999999999 + task: + type: Retrieval + - dataset: + config: default + name: MTEB ToxicConversationsClassification (default) + revision: edfaf9da55d3dd50d43143d90c1ac476895ae6de + split: test + type: mteb/toxic_conversations_50k + metrics: + - type: accuracy + value: 69.0625 + - type: ap + value: 12.429057046174089 + - type: ap_weighted + value: 12.429057046174089 + - type: f1 + value: 52.366056859622454 + - type: f1_weighted + value: 75.91632061778698 + - type: main_score + value: 69.0625 + task: + type: Classification + - dataset: + config: default + name: MTEB TweetSentimentExtractionClassification (default) + revision: d604517c81ca91fe16a244d1248fc021f9ecee7a + split: test + type: mteb/tweet_sentiment_extraction + metrics: + - type: accuracy + value: 55.387662705149964 + - type: f1 + value: 55.62292803889264 + - type: f1_weighted + value: 55.01561915660653 + - type: main_score + value: 55.387662705149964 + task: + type: Classification + - dataset: + config: default + name: MTEB TwentyNewsgroupsClustering (default) + revision: 6125ec4e24fa026cec8a478383ee943acfbd5449 + split: test + type: mteb/twentynewsgroups-clustering + metrics: + - type: main_score + value: 33.535908963951435 + - type: v_measure + value: 33.535908963951435 + - type: v_measure_std + value: 1.8862804680454297 + task: + type: Clustering + - dataset: + config: default + name: MTEB TwitterSemEval2015 (default) + revision: 70970daeab8776df92f5ea462b6173c0b46fd2d1 + split: test + type: mteb/twittersemeval2015-pairclassification + metrics: + - type: cosine_accuracy + value: 81.57000655659535 + - type: cosine_accuracy_threshold + value: 76.01186428039885 + - type: cosine_ap + value: 57.187252502171674 + - type: cosine_f1 + value: 54.94480738905159 + - type: cosine_f1_threshold + value: 63.27845286960887 + - type: cosine_precision + value: 47.93632075471698 + - type: cosine_recall + value: 64.35356200527704 + - type: dot_accuracy + value: 81.57000655659535 + - type: dot_accuracy_threshold + value: 76.01186510638954 + - type: dot_ap + value: 57.1872568788409 + - type: dot_f1 + value: 54.94480738905159 + - type: dot_f1_threshold + value: 63.27845437266042 + - type: dot_precision + value: 47.93632075471698 + - type: dot_recall + value: 64.35356200527704 + - type: euclidean_accuracy + value: 81.57000655659535 + - type: euclidean_accuracy_threshold + value: 69.2649048666448 + - type: euclidean_ap + value: 57.18724194735979 + - type: euclidean_f1 + value: 54.94480738905159 + - type: euclidean_f1_threshold + value: 85.69894748780587 + - type: euclidean_precision + value: 47.93632075471698 + - type: euclidean_recall + value: 64.35356200527704 + - type: main_score + value: 57.516050924090266 + - type: manhattan_accuracy + value: 81.71902008702389 + - type: manhattan_accuracy_threshold + value: 856.8997862166725 + - type: manhattan_ap + value: 57.516050924090266 + - type: manhattan_f1 + value: 55.16339869281046 + - type: manhattan_f1_threshold + value: 1035.858379830097 + - type: manhattan_precision + value: 50.18378378378379 + - type: manhattan_recall + value: 61.24010554089709 + - type: max_accuracy + value: 81.71902008702389 + - type: max_ap + value: 57.516050924090266 + - type: max_f1 + value: 55.16339869281046 + - type: max_precision + value: 50.18378378378379 + - type: max_recall + value: 64.35356200527704 + - type: similarity_accuracy + value: 81.57000655659535 + - type: similarity_accuracy_threshold + value: 76.01186428039885 + - type: similarity_ap + value: 57.187252502171674 + - type: similarity_f1 + value: 54.94480738905159 + - type: similarity_f1_threshold + value: 63.27845286960887 + - type: similarity_precision + value: 47.93632075471698 + - type: similarity_recall + value: 64.35356200527704 + task: + type: PairClassification + - dataset: + config: default + name: MTEB TwitterURLCorpus (default) + revision: 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf + split: test + type: mteb/twitterurlcorpus-pairclassification + metrics: + - type: cosine_accuracy + value: 87.09977878682035 + - type: cosine_accuracy_threshold + value: 63.00089389314832 + - type: cosine_ap + value: 81.9487582699938 + - type: cosine_f1 + value: 74.04089724292375 + - type: cosine_f1_threshold + value: 56.35024835869245 + - type: cosine_precision + value: 70.7599466704091 + - type: cosine_recall + value: 77.64089929165382 + - type: dot_accuracy + value: 87.09977878682035 + - type: dot_accuracy_threshold + value: 63.00089560728222 + - type: dot_ap + value: 81.94879514546079 + - type: dot_f1 + value: 74.04089724292375 + - type: dot_f1_threshold + value: 56.350250341728405 + - type: dot_precision + value: 70.7599466704091 + - type: dot_recall + value: 77.64089929165382 + - type: euclidean_accuracy + value: 87.09977878682035 + - type: euclidean_accuracy_threshold + value: 86.02221469735642 + - type: euclidean_ap + value: 81.94875892553148 + - type: euclidean_f1 + value: 74.04089724292375 + - type: euclidean_f1_threshold + value: 93.43420484744681 + - type: euclidean_precision + value: 70.7599466704091 + - type: euclidean_recall + value: 77.64089929165382 + - type: main_score + value: 82.13756947863085 + - type: manhattan_accuracy + value: 87.19292117825125 + - type: manhattan_accuracy_threshold + value: 1076.0586285257887 + - type: manhattan_ap + value: 82.13756947863085 + - type: manhattan_f1 + value: 74.36426623424485 + - type: manhattan_f1_threshold + value: 1148.366796662276 + - type: manhattan_precision + value: 71.32051463311183 + - type: manhattan_recall + value: 77.6793963658762 + - type: max_accuracy + value: 87.19292117825125 + - type: max_ap + value: 82.13756947863085 + - type: max_f1 + value: 74.36426623424485 + - type: max_precision + value: 71.32051463311183 + - type: max_recall + value: 77.6793963658762 + - type: similarity_accuracy + value: 87.09977878682035 + - type: similarity_accuracy_threshold + value: 63.00089389314832 + - type: similarity_ap + value: 81.9487582699938 + - type: similarity_f1 + value: 74.04089724292375 + - type: similarity_f1_threshold + value: 56.35024835869245 + - type: similarity_precision + value: 70.7599466704091 + - type: similarity_recall + value: 77.64089929165382 + task: + type: PairClassification +model_name: potion-base-8M +tags: +- embeddings +- static-embeddings +- mteb +--- + +# potion-base-8M Model Card + +This [Model2Vec](https://github.com/MinishLab/model2vec) model is pre-trained using [Tokenlearn](https://github.com/MinishLab/tokenlearn). It is a distilled version of the [baai/bge-base-en-v1.5](https://huggingface.co/baai/bge-base-en-v1.5) Sentence Transformer. It uses static embeddings, allowing text embeddings to be computed orders of magnitude faster on both GPU and CPU. It is designed for applications where computational resources are limited or where real-time performance is critical. + + + +## Installation + +Install model2vec using pip: +``` +pip install model2vec +``` + +## Usage +Load this model using the `from_pretrained` method: +```python +from model2vec import StaticModel + +# Load a pretrained Model2Vec model +model = StaticModel.from_pretrained("minishlab/potion-base-8M") + +# Compute text embeddings +embeddings = model.encode(["Example sentence"]) +``` + + +## How it works + +Model2vec creates a small, static model that outperforms other static embedding models by a large margin on all tasks on [MTEB](https://huggingface.co/spaces/mteb/leaderboard). This model is pre-trained using [Tokenlearn](https://github.com/MinishLab/tokenlearn). It's created using the following steps: +- Distillation: first, a model is distilled from a sentence transformer model using Model2Vec. +- Training data creation: the sentence transformer model is used to create training data by creating mean output embeddings on a large corpus. +- Training: the distilled model is trained on the training data using Tokenlearn. +- Post-training re-regularization: after training, the model is re-regularized by weighting the tokens based on their frequency, applying PCA, and finally applying [SIF weighting](https://openreview.net/pdf?id=SyK00v5xx). + +The results for this model can be found on the [Model2Vec results page](https://github.com/MinishLab/model2vec/blob/main/results/README.md). + +## Additional Resources + +- [All Model2Vec models on the hub](https://huggingface.co/models?library=model2vec) +- [Model2Vec Repo](https://github.com/MinishLab/model2vec) +- [Tokenlearn repo](https://github.com/MinishLab/tokenlearn) +- [Model2Vec Results](https://github.com/MinishLab/model2vec/blob/main/results/README.md) +- [Model2Vec Tutorials](https://github.com/MinishLab/model2vec/tree/main/tutorials) + +## Library Authors + +Model2Vec was developed by the [Minish Lab](https://github.com/MinishLab) team consisting of [Stephan Tulkens](https://github.com/stephantul) and [Thomas van Dongen](https://github.com/Pringled). + +## Citation + +Please cite the [Model2Vec repository](https://github.com/MinishLab/model2vec) if you use this model in your work. +``` +@software{minishlab2024model2vec, + authors = {Stephan Tulkens, Thomas van Dongen}, + title = {Model2Vec: Turn any Sentence Transformer into a Small Fast Model}, + year = {2024}, + url = {https://github.com/MinishLab/model2vec}, +} +``` \ No newline at end of file