metadata

base_model: BAAI/bge-base-en-v1.5
datasets: []
language:
  - en
library_name: sentence-transformers
license: apache-2.0
metrics:
  - cosine_accuracy@1
  - cosine_accuracy@3
  - cosine_accuracy@5
  - cosine_accuracy@10
  - cosine_precision@1
  - cosine_precision@3
  - cosine_precision@5
  - cosine_precision@10
  - cosine_recall@1
  - cosine_recall@3
  - cosine_recall@5
  - cosine_recall@10
  - cosine_ndcg@10
  - cosine_mrr@10
  - cosine_map@100
pipeline_tag: sentence-similarity
tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - generated_from_trainer
  - dataset_size:1000
  - loss:MatryoshkaLoss
  - loss:MultipleNegativesRankingLoss
widget:
  - source_sentence: >-
      Revision stage: Edit the output to correct content unsupported by evidence
      while preserving the original content as much as possible. Initialize the
      revised text $y=x$.


      (1) Per $(q_i, e_{ij})$, an agreement model (via few-shot prompting + CoT,
      $(y, q, e) \to {0,1}$) checks whether the evidence $e_i$ disagrees with
      the current revised text $y$.

      (2) Only if a disagreement is detect, the edit model (via few-shot
      prompting + CoT, $(y, q, e) \to \text{ new }y$) outputs a new version of
      $y$ that aims to agree with evidence $e_{ij}$ while otherwise minimally
      altering $y$.

      (3) Finally only a limited number $M=5$ of evidence goes into the
      attribution report $A$.





      Fig. 12. Illustration of RARR (Retrofit Attribution using Research and
      Revision). (Image source: Gao et al. 2022)

      When evaluating the revised text $y$, both attribution and preservation
      metrics matter.
    sentences:
      - >-
        What is the impact of claim extraction on the efficiency of query
        generation within various tool querying methodologies?
      - >-
        What are the implications of integrating both attribution and
        preservation metrics in the assessment of a revised text for an
        attribution report?
      - >-
        What impact does the calibration of large language models, as discussed
        in the research by Kadavath et al. (2022), have on the consistency and
        accuracy of their responses, particularly in the context of multiple
        choice questions?
  - source_sentence: >-
      Fig. 1. Knowledge categorization of close-book QA examples based on how
      likely the model outputs correct answers. (Image source: Gekhman et al.
      2024)

      Some interesting observations of the experiments, where dev set accuracy
      is considered a proxy for hallucinations.


      Unknown examples are fitted substantially slower than Known.

      The best dev performance is obtained when the LLM fits the majority of the
      Known training examples but only a few of the Unknown ones. The model
      starts to hallucinate when it learns most of the Unknown examples.

      Among Known examples, MaybeKnown cases result in better overall
      performance, more essential than HighlyKnown ones.
    sentences:
      - >-
        What are the implications of a language model's performance when it is
        primarily trained on familiar examples compared to a diverse set of
        unfamiliar examples, and how does this relate to the phenomenon of
        hallucinations in language models?
      - >-
        How can the insights gained from the evaluation framework inform the
        future enhancements of AI models, particularly in terms of improving
        factual accuracy and entity recognition?
      - >-
        What role does the MPNet model play in evaluating the faithfulness of
        reasoning paths, particularly in relation to scores of entailment and
        contradiction?
  - source_sentence: >-
      Non-context LLM: Prompt LLM directly with <atomic-fact> True or False?
      without additional context.

      Retrieval→LLM: Prompt with $k$ related passages retrieved from the
      knowledge source as context.

      Nonparametric probability (NP)): Compute the average likelihood of tokens
      in the atomic fact by a masked LM and use that to make a prediction.

      Retrieval→LLM + NP: Ensemble of two methods.


      Some interesting observations on model hallucination behavior:


      Error rates are higher for rarer entities in the task of biography
      generation.

      Error rates are higher for facts mentioned later in the generation.

      Using retrieval to ground the model generation significantly helps reduce
      hallucination.
    sentences:
      - >-
        What methods does the model employ to generate impactful, non-standard
        verification questions that enhance the fact-checking process?
      - >-
        What impact does the timing of fact presentation in AI outputs have on
        the likelihood of generating inaccuracies?
      - >-
        What are the benefits of using the 'Factor+revise' strategy in enhancing
        the reliability of verification processes in few-shot learning,
        particularly when it comes to identifying inconsistencies?
  - source_sentence: >-
      Research stage: Find related documents as evidence.


      (1) First use a query generation model (via few-shot prompting, $x \to
      {q_1, \dots, q_N}$) to construct a set of search queries ${q_1, \dots,
      q_N}$ to verify all aspects of each sentence.

      (2) Run Google search, $K=5$ results per query $q_i$.

      (3) Utilize a pretrained query-document relevance model to assign
      relevance scores and only retain one most relevant $J=1$ document $e_{i1},
      \dots, e_{iJ}$ per query $q_i$.



      Revision stage: Edit the output to correct content unsupported by evidence
      while preserving the original content as much as possible. Initialize the
      revised text $y=x$.
    sentences:
      - >-
        In what ways does the process of generating queries facilitate the
        verification of content accuracy, particularly through the lens of
        evidence-based editing methodologies?
      - >-
        What role do attribution and preservation metrics play in assessing the
        quality of revised texts, and how might these factors influence the
        success of the Evidence Disagreement Detection process?
      - >-
        What are the practical ways to utilize the F1 @ K metric for assessing
        how well FacTool identifies factual inaccuracies in various fields?
  - source_sentence: >-
      (1) Joint: join with step 2, where the few-shot examples are structured as
      (response, verification questions, verification answers); The drawback is
      that the original response is in the context, so the model may repeat
      similar hallucination.

      (2) 2-step: separate the verification planning and execution steps, such
      as the original response doesn’t impact

      (3) Factored: each verification question is answered separately. Say, if a
      long-form base generation results in multiple verification questions, we
      would answer each question one-by-one.

      (4) Factor+revise: adding a “cross-checking” step after factored
      verification execution, conditioned on both the baseline response and the
      verification question and answer. It detects inconsistency.



      Final output: Generate the final, refined output. The output gets revised
      at this step if any inconsistency is discovered.
    sentences:
      - >-
        What are the key challenges associated with using a pre-training dataset
        for world knowledge, particularly in maintaining the factual accuracy of
        the outputs generated by the model?
      - >-
        What obstacles arise when depending on the pre-training dataset in the
        context of extrinsic hallucination affecting model outputs?
      - >-
        In what ways does the 'Factor+revise' method enhance the reliability of
        responses when compared to the 'Joint' and '2-step' methods used for
        cross-checking?
model-index:
  - name: BGE base Financial Matryoshka
    results:
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 768
          type: dim_768
        metrics:
          - type: cosine_accuracy@1
            value: 0.8802083333333334
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.984375
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.9947916666666666
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.9947916666666666
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.8802083333333334
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.328125
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.19895833333333335
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.09947916666666667
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.8802083333333334
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.984375
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.9947916666666666
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.9947916666666666
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.9495062223081544
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.9337673611111109
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.934240845959596
            name: Cosine Map@100
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 512
          type: dim_512
        metrics:
          - type: cosine_accuracy@1
            value: 0.8854166666666666
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.984375
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.9947916666666666
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 1
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.8854166666666666
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.328125
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.19895833333333335
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.09999999999999999
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.8854166666666666
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.984375
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.9947916666666666
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 1
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.9536782535355709
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.937818287037037
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.937818287037037
            name: Cosine Map@100
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 256
          type: dim_256
        metrics:
          - type: cosine_accuracy@1
            value: 0.9010416666666666
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.984375
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 1
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 1
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.9010416666666666
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.328125
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.19999999999999998
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.09999999999999999
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.9010416666666666
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.984375
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 1
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 1
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.9587563670488631
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.9446180555555554
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.9446180555555556
            name: Cosine Map@100
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 128
          type: dim_128
        metrics:
          - type: cosine_accuracy@1
            value: 0.90625
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.984375
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 1
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 1
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.90625
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.328125
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.19999999999999998
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.09999999999999999
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.90625
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.984375
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 1
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 1
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.9609068566179642
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.9474826388888888
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.947482638888889
            name: Cosine Map@100
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 64
          type: dim_64
        metrics:
          - type: cosine_accuracy@1
            value: 0.890625
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.984375
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 1
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 1
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.890625
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.328125
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.19999999999999998
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.09999999999999999
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.890625
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.984375
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 1
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 1
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.9551401340175182
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.9396701388888888
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.939670138888889
            name: Cosine Map@100

BGE base Financial Matryoshka

This is a sentence-transformers model finetuned from BAAI/bge-base-en-v1.5. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: BAAI/bge-base-en-v1.5
Maximum Sequence Length: 512 tokens
Output Dimensionality: 768 tokens
Similarity Function: Cosine Similarity
Language: en
License: apache-2.0

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("joshuapb/fine-tuned-matryoshka-1000")
# Run inference
sentences = [
    '(1) Joint: join with step 2, where the few-shot examples are structured as (response, verification questions, verification answers); The drawback is that the original response is in the context, so the model may repeat similar hallucination.\n(2) 2-step: separate the verification planning and execution steps, such as the original response doesn’t impact\n(3) Factored: each verification question is answered separately. Say, if a long-form base generation results in multiple verification questions, we would answer each question one-by-one.\n(4) Factor+revise: adding a “cross-checking” step after factored verification execution, conditioned on both the baseline response and the verification question and answer. It detects inconsistency.\n\n\nFinal output: Generate the final, refined output. The output gets revised at this step if any inconsistency is discovered.',
    "In what ways does the 'Factor+revise' method enhance the reliability of responses when compared to the 'Joint' and '2-step' methods used for cross-checking?",
    'What obstacles arise when depending on the pre-training dataset in the context of extrinsic hallucination affecting model outputs?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

Dataset: dim_768
Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.8802
cosine_accuracy@3	0.9844
cosine_accuracy@5	0.9948
cosine_accuracy@10	0.9948
cosine_precision@1	0.8802
cosine_precision@3	0.3281
cosine_precision@5	0.199
cosine_precision@10	0.0995
cosine_recall@1	0.8802
cosine_recall@3	0.9844
cosine_recall@5	0.9948
cosine_recall@10	0.9948
cosine_ndcg@10	0.9495
cosine_mrr@10	0.9338
cosine_map@100	0.9342

Information Retrieval

Dataset: dim_512
Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.8854
cosine_accuracy@3	0.9844
cosine_accuracy@5	0.9948
cosine_accuracy@10	1.0
cosine_precision@1	0.8854
cosine_precision@3	0.3281
cosine_precision@5	0.199
cosine_precision@10	0.1
cosine_recall@1	0.8854
cosine_recall@3	0.9844
cosine_recall@5	0.9948
cosine_recall@10	1.0
cosine_ndcg@10	0.9537
cosine_mrr@10	0.9378
cosine_map@100	0.9378

Information Retrieval

Dataset: dim_256
Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.901
cosine_accuracy@3	0.9844
cosine_accuracy@5	1.0
cosine_accuracy@10	1.0
cosine_precision@1	0.901
cosine_precision@3	0.3281
cosine_precision@5	0.2
cosine_precision@10	0.1
cosine_recall@1	0.901
cosine_recall@3	0.9844
cosine_recall@5	1.0
cosine_recall@10	1.0
cosine_ndcg@10	0.9588
cosine_mrr@10	0.9446
cosine_map@100	0.9446

Information Retrieval

Dataset: dim_128
Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.9062
cosine_accuracy@3	0.9844
cosine_accuracy@5	1.0
cosine_accuracy@10	1.0
cosine_precision@1	0.9062
cosine_precision@3	0.3281
cosine_precision@5	0.2
cosine_precision@10	0.1
cosine_recall@1	0.9062
cosine_recall@3	0.9844
cosine_recall@5	1.0
cosine_recall@10	1.0
cosine_ndcg@10	0.9609
cosine_mrr@10	0.9475
cosine_map@100	0.9475

Information Retrieval

Dataset: dim_64
Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.8906
cosine_accuracy@3	0.9844
cosine_accuracy@5	1.0
cosine_accuracy@10	1.0
cosine_precision@1	0.8906
cosine_precision@3	0.3281
cosine_precision@5	0.2
cosine_precision@10	0.1
cosine_recall@1	0.8906
cosine_recall@3	0.9844
cosine_recall@5	1.0
cosine_recall@10	1.0
cosine_ndcg@10	0.9551
cosine_mrr@10	0.9397
cosine_map@100	0.9397

Training Details

Training Hyperparameters

Non-Default Hyperparameters

eval_strategy: epoch
per_device_eval_batch_size: 16
learning_rate: 2e-05
num_train_epochs: 5
lr_scheduler_type: cosine
warmup_ratio: 0.1
load_best_model_at_end: True

All Hyperparameters

Click to expand

overwrite_output_dir: False
do_predict: False
eval_strategy: epoch
prediction_loss_only: True
per_device_train_batch_size: 8
per_device_eval_batch_size: 16
per_gpu_train_batch_size: None
per_gpu_eval_batch_size: None
gradient_accumulation_steps: 1
eval_accumulation_steps: None
learning_rate: 2e-05
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1.0
num_train_epochs: 5
max_steps: -1
lr_scheduler_type: cosine
lr_scheduler_kwargs: {}
warmup_ratio: 0.1
warmup_steps: 0
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
save_safetensors: True
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
no_cuda: False
use_cpu: False
use_mps_device: False
seed: 42
data_seed: None
jit_mode_eval: False
use_ipex: False
bf16: False
fp16: False
fp16_opt_level: O1
half_precision_backend: auto
bf16_full_eval: False
fp16_full_eval: False
tf32: None
local_rank: 0
ddp_backend: None
tpu_num_cores: None
tpu_metrics_debug: False
debug: []
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_prefetch_factor: None
past_index: -1
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: True
ignore_data_skip: False
fsdp: []
fsdp_min_num_params: 0
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
fsdp_transformer_layer_cls_to_wrap: None
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch
optim_args: None
adafactor: False
group_by_length: False
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
use_legacy_prediction_loop: False
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: False
hub_always_push: False
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_inputs_for_metrics: False
eval_do_concat_batches: True
fp16_backend: auto
push_to_hub_model_id: None
push_to_hub_organization: None
mp_parameters:
auto_find_batch_size: False
full_determinism: False
torchdynamo: None
ray_scope: last
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
dispatch_batches: None
split_batches: None
include_tokens_per_second: False
include_num_input_tokens_seen: False
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
eval_on_start: False
batch_sampler: batch_sampler
multi_dataset_batch_sampler: proportional

Training Logs

Click to expand

Epoch	Step	Training Loss	dim_128_cosine_map@100	dim_256_cosine_map@100	dim_512_cosine_map@100	dim_64_cosine_map@100	dim_768_cosine_map@100
0.04	5	4.9678	-	-	-	-	-
0.08	10	4.6482	-	-	-	-	-
0.12	15	5.0735	-	-	-	-	-
0.16	20	4.0336	-	-	-	-	-
0.2	25	3.7572	-	-	-	-	-
0.24	30	4.3054	-	-	-	-	-
0.28	35	2.6705	-	-	-	-	-
0.32	40	3.1929	-	-	-	-	-
0.36	45	3.1139	-	-	-	-	-
0.4	50	2.5219	-	-	-	-	-
0.44	55	3.1847	-	-	-	-	-
0.48	60	2.2306	-	-	-	-	-
0.52	65	2.251	-	-	-	-	-
0.56	70	2.2432	-	-	-	-	-
0.6	75	2.7462	-	-	-	-	-
0.64	80	2.9992	-	-	-	-	-
0.68	85	2.338	-	-	-	-	-
0.72	90	2.0169	-	-	-	-	-
0.76	95	1.257	-	-	-	-	-
0.8	100	1.5015	-	-	-	-	-
0.84	105	1.9198	-	-	-	-	-
0.88	110	2.2154	-	-	-	-	-
0.92	115	2.4026	-	-	-	-	-
0.96	120	1.911	-	-	-	-	-
1.0	125	2.079	0.9151	0.9098	0.9220	0.8788	0.9251
1.04	130	1.4704	-	-	-	-	-
1.08	135	0.7323	-	-	-	-	-
1.12	140	0.6308	-	-	-	-	-
1.16	145	0.4655	-	-	-	-	-
1.2	150	1.0186	-	-	-	-	-
1.24	155	1.1408	-	-	-	-	-
1.28	160	1.965	-	-	-	-	-
1.32	165	1.5987	-	-	-	-	-
1.3600	170	3.288	-	-	-	-	-
1.4	175	1.632	-	-	-	-	-
1.44	180	1.0376	-	-	-	-	-
1.48	185	0.9466	-	-	-	-	-
1.52	190	1.0106	-	-	-	-	-
1.56	195	1.4875	-	-	-	-	-
1.6	200	1.314	-	-	-	-	-
1.6400	205	1.3022	-	-	-	-	-
1.6800	210	1.5312	-	-	-	-	-
1.72	215	1.7982	-	-	-	-	-
1.76	220	1.7962	-	-	-	-	-
1.8	225	1.5788	-	-	-	-	-
1.8400	230	1.152	-	-	-	-	-
1.88	235	2.0556	-	-	-	-	-
1.92	240	1.3165	-	-	-	-	-
1.96	245	0.6941	-	-	-	-	-
2.0	250	1.2239	0.9404	0.944	0.9427	0.9327	0.9424
2.04	255	1.0423	-	-	-	-	-
2.08	260	0.8893	-	-	-	-	-
2.12	265	1.2859	-	-	-	-	-
2.16	270	1.4505	-	-	-	-	-
2.2	275	0.2728	-	-	-	-	-
2.24	280	0.6588	-	-	-	-	-
2.2800	285	0.8014	-	-	-	-	-
2.32	290	0.3053	-	-	-	-	-
2.36	295	1.4289	-	-	-	-	-
2.4	300	1.1458	-	-	-	-	-
2.44	305	0.6987	-	-	-	-	-
2.48	310	1.3389	-	-	-	-	-
2.52	315	1.2991	-	-	-	-	-
2.56	320	1.8088	-	-	-	-	-
2.6	325	0.4242	-	-	-	-	-
2.64	330	1.5873	-	-	-	-	-
2.68	335	1.3873	-	-	-	-	-
2.7200	340	1.4297	-	-	-	-	-
2.76	345	2.0637	-	-	-	-	-
2.8	350	1.1252	-	-	-	-	-
2.84	355	0.367	-	-	-	-	-
2.88	360	1.7606	-	-	-	-	-
2.92	365	1.196	-	-	-	-	-
2.96	370	1.8827	-	-	-	-	-
3.0	375	0.6822	0.9494	0.9479	0.9336	0.9414	0.9405
3.04	380	0.4954	-	-	-	-	-
3.08	385	0.1717	-	-	-	-	-
3.12	390	0.7435	-	-	-	-	-
3.16	395	1.4323	-	-	-	-	-
3.2	400	1.1207	-	-	-	-	-
3.24	405	1.9009	-	-	-	-	-
3.2800	410	1.6706	-	-	-	-	-
3.32	415	0.8378	-	-	-	-	-
3.36	420	1.0911	-	-	-	-	-
3.4	425	0.6565	-	-	-	-	-
3.44	430	1.0302	-	-	-	-	-
3.48	435	0.6425	-	-	-	-	-
3.52	440	1.1472	-	-	-	-	-
3.56	445	1.996	-	-	-	-	-
3.6	450	1.5308	-	-	-	-	-
3.64	455	0.7427	-	-	-	-	-
3.68	460	1.4596	-	-	-	-	-
3.7200	465	1.1984	-	-	-	-	-
3.76	470	0.7601	-	-	-	-	-
3.8	475	1.3544	-	-	-	-	-
3.84	480	1.6655	-	-	-	-	-
3.88	485	1.2596	-	-	-	-	-
3.92	490	0.9451	-	-	-	-	-
3.96	495	0.7079	-	-	-	-	-
4.0	500	1.3471	0.9453	0.9446	0.9404	0.9371	0.9335
4.04	505	0.4583	-	-	-	-	-
4.08	510	1.288	-	-	-	-	-
4.12	515	1.6946	-	-	-	-	-
4.16	520	1.1239	-	-	-	-	-
4.2	525	1.1026	-	-	-	-	-
4.24	530	1.4121	-	-	-	-	-
4.28	535	1.7113	-	-	-	-	-
4.32	540	0.8389	-	-	-	-	-
4.36	545	0.3117	-	-	-	-	-
4.4	550	0.3144	-	-	-	-	-
4.44	555	1.4694	-	-	-	-	-
4.48	560	1.3233	-	-	-	-	-
4.52	565	0.792	-	-	-	-	-
4.5600	570	0.4881	-	-	-	-	-
4.6	575	0.5097	-	-	-	-	-
4.64	580	1.6377	-	-	-	-	-
4.68	585	0.7273	-	-	-	-	-
4.72	590	1.5464	-	-	-	-	-
4.76	595	1.4392	-	-	-	-	-
4.8	600	1.4384	-	-	-	-	-
4.84	605	0.6375	-	-	-	-	-
4.88	610	1.0528	-	-	-	-	-
4.92	615	0.0276	-	-	-	-	-
4.96	620	0.9604	-	-	-	-	-
5.0	625	0.7219	0.9475	0.9446	0.9378	0.9397	0.9342

The bold row denotes the saved checkpoint.

Framework Versions

Python: 3.10.12
Sentence Transformers: 3.0.1
Transformers: 4.42.4
PyTorch: 2.3.1+cu121
Accelerate: 0.32.1
Datasets: 2.21.0
Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning}, 
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply}, 
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}