README.md

metadata

base_model: BAAI/bge-base-en-v1.5
datasets: []
language:
  - en
library_name: sentence-transformers
license: apache-2.0
metrics:
  - cosine_accuracy@1
  - cosine_accuracy@3
  - cosine_accuracy@5
  - cosine_accuracy@10
  - cosine_precision@1
  - cosine_precision@3
  - cosine_precision@5
  - cosine_precision@10
  - cosine_recall@1
  - cosine_recall@3
  - cosine_recall@5
  - cosine_recall@10
  - cosine_ndcg@10
  - cosine_mrr@10
  - cosine_map@100
pipeline_tag: sentence-similarity
tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - generated_from_trainer
  - dataset_size:200
  - loss:MatryoshkaLoss
  - loss:MultipleNegativesRankingLoss
widget:
  - source_sentence: >-
      Verbalized number or word (e.g. “lowest”, “low”, “medium”, “high”,
      “highest”), such as "Confidence: 60% / Medium".

      Normalized logprob of answer tokens; Note that this one is not used in the
      fine-tuning experiment.

      Logprob of an indirect "True/False" token after the raw answer.

      Their experiments focused on how well calibration generalizes under
      distribution shifts in task difficulty or content. Each fine-tuning
      datapoint is a question, the model’s answer (possibly incorrect), and a
      calibrated confidence. Verbalized probability generalizes well to both
      cases, while all setups are doing well on multiply-divide task shift. 
      Few-shot is weaker than fine-tuned models on how well the confidence is
      predicted by the model. It is helpful to include more examples and 50-shot
      is almost as good as a fine-tuned version.
    sentences:
      - >-
        What is the relationship between the calibration of AI models and the
        effectiveness of verbalized probabilities when applied to tasks of
        varying difficulty levels?
      - >-
        In what ways does the F1 @ K metric contribute to evaluating the factual
        accuracy and comprehensiveness of outputs generated by long-form
        language models?
      - >-
        What impact does the implementation of a pretrained query-document
        relevance model have on the process of document selection in research
        methodologies?
  - source_sentence: >-
      Fig. 4. Overview of SAFE for factuality evaluation of long-form LLM
      generation. (Image source: Wei et al. 2024)

      The SAFE evaluation metric is F1 @ K. The motivation is that model
      response for long-form factuality should ideally hit both precision and
      recall, as the response should be both


      factual : measured by precision, the percentage of supported facts among
      all facts in the entire response.

      long : measured by recall, the percentage of provided facts among all
      relevant facts that should appear in the response. Therefore we want to
      consider the number of supported facts up to $K$.


      Given the model response $y$, the metric F1 @ K is defined as:
    sentences:
      - >-
        What methodologies does the agreement model employ to identify
        discrepancies between the original and revised text, and how do these
        methodologies impact the overall editing workflow?
      - >-
        In what ways does the SAFE evaluation metric achieve a harmonious
        equilibrium between precision and recall in the context of evaluating
        the factual accuracy of long-form outputs generated by large language
        models?
      - >-
        In what ways does the inherently adversarial structure of TruthfulQA
        inquiries facilitate the detection of prevalent fallacies in human
        cognitive processes, and what implications does this have for
        understanding the constraints of expansive language models?
  - source_sentence: >-
      Non-context LLM: Prompt LLM directly with <atomic-fact> True or False?
      without additional context.

      Retrieval→LLM: Prompt with $k$ related passages retrieved from the
      knowledge source as context.

      Nonparametric probability (NP)): Compute the average likelihood of tokens
      in the atomic fact by a masked LM and use that to make a prediction.

      Retrieval→LLM + NP: Ensemble of two methods.


      Some interesting observations on model hallucination behavior:


      Error rates are higher for rarer entities in the task of biography
      generation.

      Error rates are higher for facts mentioned later in the generation.

      Using retrieval to ground the model generation significantly helps reduce
      hallucination.
    sentences:
      - >-
        In what ways does the Rethinking with Retrieval (RR) methodology
        leverage Chain-of-Thought (CoT) prompting to enhance the efficacy of
        external knowledge retrieval, and what implications does this have for
        the precision of predictive outcomes generated by models?
      - >-
        In what ways does the retrieval of related passages contribute to
        minimizing hallucinations in large language models, and what specific
        techniques can be employed to evaluate the impact of this approach?
      - >-
        What are the benefits of using retrieval methods in biography generation
        to minimize inaccuracies, especially when compared to traditional
        prompting techniques that lack context?
  - source_sentence: >-
      Yin et al. (2023) studies the concept of self-knowledge, referring to
      whether language models know what they know or don’t know.

      SelfAware, containing 1,032 unanswerable questions across five categories
      and 2,337 answerable questions. Unanswerable questions are sourced from
      online forums with human annotations while answerable questions are
      sourced from SQuAD, HotpotQA and TriviaQA based on text similarity with
      unanswerable questions. A question may be unanswerable due to various
      reasons, such as no scientific consensus, imaginations of the future,
      completely subjective, philosophical reasons that may yield multiple
      responses, etc. Considering separating answerable vs unanswerable
      questions as a binary classification task, we can measure F1-score or
      accuracy and the experiments showed that larger models can do better at
      this task.
    sentences:
      - >-
        What is the relationship between model size and performance metrics,
        such as F1-score and accuracy, in the context of classifying questions
        into answerable and unanswerable categories?
      - >-
        How does the introduction of stochastic perturbations in synthetic
        training data contribute to the enhancement of editor model efficacy
        within LangChain frameworks?
      - >-
        How do the various output values linked to reflection tokens in the
        Self-RAG framework impact the generation process, and why are they
        important?
  - source_sentence: >-
      Fig. 1. Knowledge categorization of close-book QA examples based on how
      likely the model outputs correct answers. (Image source: Gekhman et al.
      2024)

      Some interesting observations of the experiments, where dev set accuracy
      is considered a proxy for hallucinations.


      Unknown examples are fitted substantially slower than Known.

      The best dev performance is obtained when the LLM fits the majority of the
      Known training examples but only a few of the Unknown ones. The model
      starts to hallucinate when it learns most of the Unknown examples.

      Among Known examples, MaybeKnown cases result in better overall
      performance, more essential than HighlyKnown ones.
    sentences:
      - >-
        In what ways does the fitting speed of examples that are not previously
        encountered differ from that of familiar examples, and how does this
        variation influence the overall accuracy of the model on the development
        set?
      - >-
        What role do reflection tokens play in enhancing the efficiency of
        document retrieval and generation within the Self-RAG framework?
      - >-
        How do the results presented by Gekhman et al. in their 2024 study
        inform our understanding of the reliability metrics associated with
        large language models (LLMs) when subjected to fine-tuning with novel
        datasets?
model-index:
  - name: BGE base Financial Matryoshka
    results:
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 768
          type: dim_768
        metrics:
          - type: cosine_accuracy@1
            value: 0.8802083333333334
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.96875
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.984375
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.9947916666666666
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.8802083333333334
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.3229166666666667
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.196875
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.09947916666666667
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.8802083333333334
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.96875
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.984375
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.9947916666666666
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.9433275174124347
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.9261284722222224
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.9264025950292397
            name: Cosine Map@100
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 512
          type: dim_512
        metrics:
          - type: cosine_accuracy@1
            value: 0.8697916666666666
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.9739583333333334
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.9739583333333334
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.9947916666666666
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.8697916666666666
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.3246527777777778
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.1947916666666666
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.09947916666666667
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.8697916666666666
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.9739583333333334
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.9739583333333334
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.9947916666666666
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.939968526552219
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.9216269841269841
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.9220610119047619
            name: Cosine Map@100
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 256
          type: dim_256
        metrics:
          - type: cosine_accuracy@1
            value: 0.8697916666666666
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.9739583333333334
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.984375
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 1
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.8697916666666666
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.3246527777777778
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.196875
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.09999999999999999
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.8697916666666666
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.9739583333333334
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.984375
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 1
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.9419747509776967
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.922676917989418
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.922676917989418
            name: Cosine Map@100
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 128
          type: dim_128
        metrics:
          - type: cosine_accuracy@1
            value: 0.8541666666666666
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.9583333333333334
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.96875
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.9947916666666666
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.8541666666666666
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.3194444444444445
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.19374999999999998
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.09947916666666667
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.8541666666666666
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.9583333333333334
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.96875
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.9947916666666666
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.9306358745697197
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.9094328703703702
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.9098668981481483
            name: Cosine Map@100
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 64
          type: dim_64
        metrics:
          - type: cosine_accuracy@1
            value: 0.7916666666666666
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.953125
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.9739583333333334
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.9895833333333334
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.7916666666666666
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.3177083333333333
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.1947916666666666
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.09895833333333333
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.7916666666666666
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.953125
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.9739583333333334
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.9895833333333334
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.9003914274568845
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.8705935846560847
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.8713150853775854
            name: Cosine Map@100

BGE base Financial Matryoshka

This is a sentence-transformers model finetuned from BAAI/bge-base-en-v1.5. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-base-en-v1.5
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 tokens
• Similarity Function: Cosine Similarity
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("joshuapb/fine-tuned-matryoshka-200")
# Run inference
sentences = [
    'Fig. 1. Knowledge categorization of close-book QA examples based on how likely the model outputs correct answers. (Image source: Gekhman et al. 2024)\nSome interesting observations of the experiments, where dev set accuracy is considered a proxy for hallucinations.\n\nUnknown examples are fitted substantially slower than Known.\nThe best dev performance is obtained when the LLM fits the majority of the Known training examples but only a few of the Unknown ones. The model starts to hallucinate when it learns most of the Unknown examples.\nAmong Known examples, MaybeKnown cases result in better overall performance, more essential than HighlyKnown ones.',
    'In what ways does the fitting speed of examples that are not previously encountered differ from that of familiar examples, and how does this variation influence the overall accuracy of the model on the development set?',
    'How do the results presented by Gekhman et al. in their 2024 study inform our understanding of the reliability metrics associated with large language models (LLMs) when subjected to fine-tuning with novel datasets?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Dataset: dim_768
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.8802
cosine_accuracy@3	0.9688
cosine_accuracy@5	0.9844
cosine_accuracy@10	0.9948
cosine_precision@1	0.8802
cosine_precision@3	0.3229
cosine_precision@5	0.1969
cosine_precision@10	0.0995
cosine_recall@1	0.8802
cosine_recall@3	0.9688
cosine_recall@5	0.9844
cosine_recall@10	0.9948
cosine_ndcg@10	0.9433
cosine_mrr@10	0.9261
cosine_map@100	0.9264

Information Retrieval

• Dataset: dim_512
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.8698
cosine_accuracy@3	0.974
cosine_accuracy@5	0.974
cosine_accuracy@10	0.9948
cosine_precision@1	0.8698
cosine_precision@3	0.3247
cosine_precision@5	0.1948
cosine_precision@10	0.0995
cosine_recall@1	0.8698
cosine_recall@3	0.974
cosine_recall@5	0.974
cosine_recall@10	0.9948
cosine_ndcg@10	0.94
cosine_mrr@10	0.9216
cosine_map@100	0.9221

Information Retrieval

• Dataset: dim_256
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.8698
cosine_accuracy@3	0.974
cosine_accuracy@5	0.9844
cosine_accuracy@10	1.0
cosine_precision@1	0.8698
cosine_precision@3	0.3247
cosine_precision@5	0.1969
cosine_precision@10	0.1
cosine_recall@1	0.8698
cosine_recall@3	0.974
cosine_recall@5	0.9844
cosine_recall@10	1.0
cosine_ndcg@10	0.942
cosine_mrr@10	0.9227
cosine_map@100	0.9227

Information Retrieval

• Dataset: dim_128
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.8542
cosine_accuracy@3	0.9583
cosine_accuracy@5	0.9688
cosine_accuracy@10	0.9948
cosine_precision@1	0.8542
cosine_precision@3	0.3194
cosine_precision@5	0.1937
cosine_precision@10	0.0995
cosine_recall@1	0.8542
cosine_recall@3	0.9583
cosine_recall@5	0.9688
cosine_recall@10	0.9948
cosine_ndcg@10	0.9306
cosine_mrr@10	0.9094
cosine_map@100	0.9099

Information Retrieval

• Dataset: dim_64
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.7917
cosine_accuracy@3	0.9531
cosine_accuracy@5	0.974
cosine_accuracy@10	0.9896
cosine_precision@1	0.7917
cosine_precision@3	0.3177
cosine_precision@5	0.1948
cosine_precision@10	0.099
cosine_recall@1	0.7917
cosine_recall@3	0.9531
cosine_recall@5	0.974
cosine_recall@10	0.9896
cosine_ndcg@10	0.9004
cosine_mrr@10	0.8706
cosine_map@100	0.8713

Training Details

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_eval_batch_size: 16
• learning_rate: 2e-05
• num_train_epochs: 5
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• load_best_model_at_end: True

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 8
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 5
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	dim_128_cosine_map@100	dim_256_cosine_map@100	dim_512_cosine_map@100	dim_64_cosine_map@100	dim_768_cosine_map@100
0.2	5	5.2225	-	-	-	-	-
0.4	10	4.956	-	-	-	-	-
0.6	15	3.6388	-	-	-	-	-
0.8	20	3.1957	-	-	-	-	-
1.0	25	2.6928	0.8661	0.8770	0.8754	0.8312	0.8871
1.2	30	2.5565	-	-	-	-	-
1.4	35	1.5885	-	-	-	-	-
1.6	40	2.1406	-	-	-	-	-
1.8	45	2.193	-	-	-	-	-
2.0	50	1.326	0.8944	0.9110	0.9028	0.8615	0.9037
2.2	55	2.6832	-	-	-	-	-
2.4	60	1.0584	-	-	-	-	-
2.6	65	0.8853	-	-	-	-	-
2.8	70	1.7129	-	-	-	-	-
3.0	75	2.1856	0.9106	0.9293	0.9075	0.8778	0.9266
3.2	80	1.7658	-	-	-	-	-
3.4	85	1.9783	-	-	-	-	-
3.6	90	1.9583	-	-	-	-	-
3.8	95	1.2396	-	-	-	-	-
4.0	100	1.1901	0.9073	0.9253	0.9151	0.8750	0.9312
4.2	105	2.6547	-	-	-	-	-
4.4	110	1.3485	-	-	-	-	-
4.6	115	1.0767	-	-	-	-	-
4.8	120	0.6663	-	-	-	-	-
5.0	125	1.3869	0.9099	0.9227	0.9221	0.8713	0.9264

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.10.12
• Sentence Transformers: 3.0.1
• Transformers: 4.42.4
• PyTorch: 2.3.1+cu121
• Accelerate: 0.32.1
• Datasets: 2.21.0
• Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning}, 
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply}, 
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}