---
base_model: BAAI/bge-base-en-v1.5
datasets: []
language:
- en
library_name: sentence-transformers
license: apache-2.0
metrics:
- cosine_accuracy@1
- cosine_accuracy@3
- cosine_accuracy@5
- cosine_accuracy@10
- cosine_precision@1
- cosine_precision@3
- cosine_precision@5
- cosine_precision@10
- cosine_recall@1
- cosine_recall@3
- cosine_recall@5
- cosine_recall@10
- cosine_ndcg@10
- cosine_mrr@10
- cosine_map@100
pipeline_tag: sentence-similarity
tags:
- sentence-transformers
- sentence-similarity
- feature-extraction
- generated_from_trainer
- dataset_size:100
- loss:MatryoshkaLoss
- loss:MultipleNegativesRankingLoss
widget:
- source_sentence: 'Fig. 8. The accuracy of instruct-GPT series models of different
    sizes (left to right, small to large). Larger model doing better on binary classification
    of answerable and unanswerable questions in SelfAware eval. (Image source: Yin
    et al. 2023)

    Another way to assess the model’s awareness of unknown knowledge is to measure
    the model’s output uncertainty. When a question is in-between known and unknown,
    the model is expected to demonstrate the right level of confidence.

    The experiment by Kadavath et al. (2022) showed that LLMs are shown to be well
    calibrated in their estimation probabilities of answer correctness on diverse
    multiple choice questions in a format with visible lettered answer options (MMLU,
    TruthfulQA, QuALITY, LogiQA), meaning that the predicted probability coincides
    with the frequency of that answer being true. RLHF fine-tuning makes the model
    poorly calibrated, but higher sampling temperature leads to better calibration
    results.'
  sentences:
  - What effect does the slower acquisition of new knowledge compared to established
    knowledge have on the effectiveness of large language models in practical scenarios?
  - How do discrepancies identified during the final output review phase affect the
    overall quality of the generated responses?
  - What effect does reinforcement learning from human feedback (RLHF) fine-tuning
    have on how well large language models assess the accuracy of their answers?
- source_sentence: 'Fig. 1. Knowledge categorization of close-book QA examples based
    on how likely the model outputs correct answers. (Image source: Gekhman et al.
    2024)

    Some interesting observations of the experiments, where dev set accuracy is considered
    a proxy for hallucinations.


    Unknown examples are fitted substantially slower than Known.

    The best dev performance is obtained when the LLM fits the majority of the Known
    training examples but only a few of the Unknown ones. The model starts to hallucinate
    when it learns most of the Unknown examples.

    Among Known examples, MaybeKnown cases result in better overall performance, more
    essential than HighlyKnown ones.'
  sentences:
  - What is the relationship between the structural formatting of inquiries and the
    occurrence of calibration errors in artificial intelligence models, and in what
    ways can this understanding contribute to the optimization of model training processes?
  - What are the benefits of integrating a pretrained Natural Language Inference (NLI)
    model with MPNet when assessing the reliability of reasoning paths in knowledge
    retrieval?
  - In what ways do the classifications of Known versus Unknown examples influence
    the propensity of AI models to generate hallucinations during their training processes?
- source_sentence: 'Fig. 3. The evaluation framework for the FactualityPrompt benchmark.(Image
    source: Lee, et al. 2022)

    Given the model continuation and paired Wikipedia text, two evaluation metrics
    for hallucination are considered:


    Hallucination NE (Named Entity) errors: Using a pretrained entity detection model
    and document-level grounding, this metric measures the fraction of detected named
    entities that do not appear in the ground truth document.

    Entailment ratios: Using a RoBERTa model fine-tuned on MNLI and sentence-level
    knowledge grounding, this metric calculates the fraction of generated sentences
    that are marked as relevant to the paired Wikipedia sentence by the entailment
    model.'
  sentences:
  - What impact does the implementation of a pretrained query-document relevance model
    have on the process of document selection in research methodologies?
  - In what ways does the sequence in which information is delivered in AI-generated
    responses influence the likelihood of generating inaccuracies or hallucinations?
  - In what ways does the FactualityPrompt benchmark assess the performance of named
    entity detection models, particularly in relation to errors arising from hallucinated
    named entities?
- source_sentence: 'Fig. 1. Knowledge categorization of close-book QA examples based
    on how likely the model outputs correct answers. (Image source: Gekhman et al.
    2024)

    Some interesting observations of the experiments, where dev set accuracy is considered
    a proxy for hallucinations.


    Unknown examples are fitted substantially slower than Known.

    The best dev performance is obtained when the LLM fits the majority of the Known
    training examples but only a few of the Unknown ones. The model starts to hallucinate
    when it learns most of the Unknown examples.

    Among Known examples, MaybeKnown cases result in better overall performance, more
    essential than HighlyKnown ones.'
  sentences:
  - In what ways does the inherently adversarial structure of TruthfulQA inquiries
    facilitate the detection of prevalent fallacies in human cognitive processes,
    and what implications does this have for understanding the constraints of expansive
    language models?
  - In what ways do MaybeKnown cases influence the performance of a model when contrasted
    with HighlyKnown examples, particularly in relation to the occurrence of hallucinations?
  - In what ways does the Self-RAG framework leverage reflection tokens to enhance
    the quality of its generated outputs, and what implications does this have for
    the overall generation process?
- source_sentence: 'Fine-tuning New Knowledge#

    Fine-tuning a pre-trained LLM via supervised fine-tuning and RLHF is a common
    technique for improving certain capabilities of the model like instruction following.
    Introducing new knowledge at the fine-tuning stage is hard to avoid.

    Fine-tuning usually consumes much less compute, making it debatable whether the
    model can reliably learn new knowledge via small-scale fine-tuning. Gekhman et
    al. 2024 studied the research question of whether fine-tuning LLMs on new knowledge
    encourages hallucinations. They found that (1) LLMs learn fine-tuning examples
    with new knowledge slower than other examples with knowledge consistent with the
    pre-existing knowledge of the model; (2) Once the examples with new knowledge
    are eventually learned, they increase the model’s tendency to hallucinate.'
  sentences:
  - How does the IsRel token function in the retrieval process, and what impact does
    it have on the relevance of generated content to reduce hallucination?
  - What is the relationship between the calibration of AI models and the effectiveness
    of verbalized probabilities when applied to tasks of varying difficulty levels?
  - How do the results presented by Gekhman et al. in their 2024 study inform our
    understanding of the reliability metrics associated with large language models
    (LLMs) when subjected to fine-tuning with novel datasets?
model-index:
- name: BGE base Financial Matryoshka
  results:
  - task:
      type: information-retrieval
      name: Information Retrieval
    dataset:
      name: dim 768
      type: dim_768
    metrics:
    - type: cosine_accuracy@1
      value: 0.828125
      name: Cosine Accuracy@1
    - type: cosine_accuracy@3
      value: 0.9635416666666666
      name: Cosine Accuracy@3
    - type: cosine_accuracy@5
      value: 0.9739583333333334
      name: Cosine Accuracy@5
    - type: cosine_accuracy@10
      value: 0.9947916666666666
      name: Cosine Accuracy@10
    - type: cosine_precision@1
      value: 0.828125
      name: Cosine Precision@1
    - type: cosine_precision@3
      value: 0.3211805555555556
      name: Cosine Precision@3
    - type: cosine_precision@5
      value: 0.1947916666666666
      name: Cosine Precision@5
    - type: cosine_precision@10
      value: 0.09947916666666667
      name: Cosine Precision@10
    - type: cosine_recall@1
      value: 0.828125
      name: Cosine Recall@1
    - type: cosine_recall@3
      value: 0.9635416666666666
      name: Cosine Recall@3
    - type: cosine_recall@5
      value: 0.9739583333333334
      name: Cosine Recall@5
    - type: cosine_recall@10
      value: 0.9947916666666666
      name: Cosine Recall@10
    - type: cosine_ndcg@10
      value: 0.9220150687007592
      name: Cosine Ndcg@10
    - type: cosine_mrr@10
      value: 0.8976707175925925
      name: Cosine Mrr@10
    - type: cosine_map@100
      value: 0.8981047453703703
      name: Cosine Map@100
  - task:
      type: information-retrieval
      name: Information Retrieval
    dataset:
      name: dim 512
      type: dim_512
    metrics:
    - type: cosine_accuracy@1
      value: 0.8020833333333334
      name: Cosine Accuracy@1
    - type: cosine_accuracy@3
      value: 0.9635416666666666
      name: Cosine Accuracy@3
    - type: cosine_accuracy@5
      value: 0.9739583333333334
      name: Cosine Accuracy@5
    - type: cosine_accuracy@10
      value: 0.9895833333333334
      name: Cosine Accuracy@10
    - type: cosine_precision@1
      value: 0.8020833333333334
      name: Cosine Precision@1
    - type: cosine_precision@3
      value: 0.3211805555555556
      name: Cosine Precision@3
    - type: cosine_precision@5
      value: 0.1947916666666666
      name: Cosine Precision@5
    - type: cosine_precision@10
      value: 0.09895833333333333
      name: Cosine Precision@10
    - type: cosine_recall@1
      value: 0.8020833333333334
      name: Cosine Recall@1
    - type: cosine_recall@3
      value: 0.9635416666666666
      name: Cosine Recall@3
    - type: cosine_recall@5
      value: 0.9739583333333334
      name: Cosine Recall@5
    - type: cosine_recall@10
      value: 0.9895833333333334
      name: Cosine Recall@10
    - type: cosine_ndcg@10
      value: 0.9077325270335209
      name: Cosine Ndcg@10
    - type: cosine_mrr@10
      value: 0.880220734126984
      name: Cosine Mrr@10
    - type: cosine_map@100
      value: 0.8810414411976911
      name: Cosine Map@100
  - task:
      type: information-retrieval
      name: Information Retrieval
    dataset:
      name: dim 256
      type: dim_256
    metrics:
    - type: cosine_accuracy@1
      value: 0.796875
      name: Cosine Accuracy@1
    - type: cosine_accuracy@3
      value: 0.9583333333333334
      name: Cosine Accuracy@3
    - type: cosine_accuracy@5
      value: 0.96875
      name: Cosine Accuracy@5
    - type: cosine_accuracy@10
      value: 0.9791666666666666
      name: Cosine Accuracy@10
    - type: cosine_precision@1
      value: 0.796875
      name: Cosine Precision@1
    - type: cosine_precision@3
      value: 0.3194444444444445
      name: Cosine Precision@3
    - type: cosine_precision@5
      value: 0.19374999999999998
      name: Cosine Precision@5
    - type: cosine_precision@10
      value: 0.09791666666666665
      name: Cosine Precision@10
    - type: cosine_recall@1
      value: 0.796875
      name: Cosine Recall@1
    - type: cosine_recall@3
      value: 0.9583333333333334
      name: Cosine Recall@3
    - type: cosine_recall@5
      value: 0.96875
      name: Cosine Recall@5
    - type: cosine_recall@10
      value: 0.9791666666666666
      name: Cosine Recall@10
    - type: cosine_ndcg@10
      value: 0.9011377823848584
      name: Cosine Ndcg@10
    - type: cosine_mrr@10
      value: 0.8746155753968253
      name: Cosine Mrr@10
    - type: cosine_map@100
      value: 0.8757564484126984
      name: Cosine Map@100
  - task:
      type: information-retrieval
      name: Information Retrieval
    dataset:
      name: dim 128
      type: dim_128
    metrics:
    - type: cosine_accuracy@1
      value: 0.7864583333333334
      name: Cosine Accuracy@1
    - type: cosine_accuracy@3
      value: 0.9322916666666666
      name: Cosine Accuracy@3
    - type: cosine_accuracy@5
      value: 0.9635416666666666
      name: Cosine Accuracy@5
    - type: cosine_accuracy@10
      value: 0.9635416666666666
      name: Cosine Accuracy@10
    - type: cosine_precision@1
      value: 0.7864583333333334
      name: Cosine Precision@1
    - type: cosine_precision@3
      value: 0.3107638888888889
      name: Cosine Precision@3
    - type: cosine_precision@5
      value: 0.19270833333333334
      name: Cosine Precision@5
    - type: cosine_precision@10
      value: 0.09635416666666667
      name: Cosine Precision@10
    - type: cosine_recall@1
      value: 0.7864583333333334
      name: Cosine Recall@1
    - type: cosine_recall@3
      value: 0.9322916666666666
      name: Cosine Recall@3
    - type: cosine_recall@5
      value: 0.9635416666666666
      name: Cosine Recall@5
    - type: cosine_recall@10
      value: 0.9635416666666666
      name: Cosine Recall@10
    - type: cosine_ndcg@10
      value: 0.888061438431803
      name: Cosine Ndcg@10
    - type: cosine_mrr@10
      value: 0.8623263888888889
      name: Cosine Mrr@10
    - type: cosine_map@100
      value: 0.8647421480429293
      name: Cosine Map@100
  - task:
      type: information-retrieval
      name: Information Retrieval
    dataset:
      name: dim 64
      type: dim_64
    metrics:
    - type: cosine_accuracy@1
      value: 0.6875
      name: Cosine Accuracy@1
    - type: cosine_accuracy@3
      value: 0.8645833333333334
      name: Cosine Accuracy@3
    - type: cosine_accuracy@5
      value: 0.9270833333333334
      name: Cosine Accuracy@5
    - type: cosine_accuracy@10
      value: 0.96875
      name: Cosine Accuracy@10
    - type: cosine_precision@1
      value: 0.6875
      name: Cosine Precision@1
    - type: cosine_precision@3
      value: 0.2881944444444445
      name: Cosine Precision@3
    - type: cosine_precision@5
      value: 0.18541666666666665
      name: Cosine Precision@5
    - type: cosine_precision@10
      value: 0.09687499999999999
      name: Cosine Precision@10
    - type: cosine_recall@1
      value: 0.6875
      name: Cosine Recall@1
    - type: cosine_recall@3
      value: 0.8645833333333334
      name: Cosine Recall@3
    - type: cosine_recall@5
      value: 0.9270833333333334
      name: Cosine Recall@5
    - type: cosine_recall@10
      value: 0.96875
      name: Cosine Recall@10
    - type: cosine_ndcg@10
      value: 0.8335872598831777
      name: Cosine Ndcg@10
    - type: cosine_mrr@10
      value: 0.7895895337301586
      name: Cosine Mrr@10
    - type: cosine_map@100
      value: 0.7917890681938919
      name: Cosine Map@100
---

# BGE base Financial Matryoshka

This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5). It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

## Model Details

### Model Description
- **Model Type:** Sentence Transformer
- **Base model:** [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) <!-- at revision a5beb1e3e68b9ab74eb54cfd186867f64f240e1a -->
- **Maximum Sequence Length:** 512 tokens
- **Output Dimensionality:** 768 tokens
- **Similarity Function:** Cosine Similarity
<!-- - **Training Dataset:** Unknown -->
- **Language:** en
- **License:** apache-2.0

### Model Sources

- **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
- **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers)
- **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers)

### Full Model Architecture

```
SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)
```

## Usage

### Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

```bash
pip install -U sentence-transformers
```

Then you can load this model and run inference.
```python
from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("joshuapb/fine-tuned-matryoshka-100")
# Run inference
sentences = [
    'Fine-tuning New Knowledge#\nFine-tuning a pre-trained LLM via supervised fine-tuning and RLHF is a common technique for improving certain capabilities of the model like instruction following. Introducing new knowledge at the fine-tuning stage is hard to avoid.\nFine-tuning usually consumes much less compute, making it debatable whether the model can reliably learn new knowledge via small-scale fine-tuning. Gekhman et al. 2024 studied the research question of whether fine-tuning LLMs on new knowledge encourages hallucinations. They found that (1) LLMs learn fine-tuning examples with new knowledge slower than other examples with knowledge consistent with the pre-existing knowledge of the model; (2) Once the examples with new knowledge are eventually learned, they increase the model’s tendency to hallucinate.',
    'How do the results presented by Gekhman et al. in their 2024 study inform our understanding of the reliability metrics associated with large language models (LLMs) when subjected to fine-tuning with novel datasets?',
    'What is the relationship between the calibration of AI models and the effectiveness of verbalized probabilities when applied to tasks of varying difficulty levels?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]
```

<!--
### Direct Usage (Transformers)

<details><summary>Click to see the direct usage in Transformers</summary>

</details>
-->

<!--
### Downstream Usage (Sentence Transformers)

You can finetune this model on your own dataset.

<details><summary>Click to expand</summary>

</details>
-->

<!--
### Out-of-Scope Use

*List how the model may foreseeably be misused and address what users ought not to do with the model.*
-->

## Evaluation

### Metrics

#### Information Retrieval
* Dataset: `dim_768`
* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)

| Metric              | Value      |
|:--------------------|:-----------|
| cosine_accuracy@1   | 0.8281     |
| cosine_accuracy@3   | 0.9635     |
| cosine_accuracy@5   | 0.974      |
| cosine_accuracy@10  | 0.9948     |
| cosine_precision@1  | 0.8281     |
| cosine_precision@3  | 0.3212     |
| cosine_precision@5  | 0.1948     |
| cosine_precision@10 | 0.0995     |
| cosine_recall@1     | 0.8281     |
| cosine_recall@3     | 0.9635     |
| cosine_recall@5     | 0.974      |
| cosine_recall@10    | 0.9948     |
| cosine_ndcg@10      | 0.922      |
| cosine_mrr@10       | 0.8977     |
| **cosine_map@100**  | **0.8981** |

#### Information Retrieval
* Dataset: `dim_512`
* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)

| Metric              | Value     |
|:--------------------|:----------|
| cosine_accuracy@1   | 0.8021    |
| cosine_accuracy@3   | 0.9635    |
| cosine_accuracy@5   | 0.974     |
| cosine_accuracy@10  | 0.9896    |
| cosine_precision@1  | 0.8021    |
| cosine_precision@3  | 0.3212    |
| cosine_precision@5  | 0.1948    |
| cosine_precision@10 | 0.099     |
| cosine_recall@1     | 0.8021    |
| cosine_recall@3     | 0.9635    |
| cosine_recall@5     | 0.974     |
| cosine_recall@10    | 0.9896    |
| cosine_ndcg@10      | 0.9077    |
| cosine_mrr@10       | 0.8802    |
| **cosine_map@100**  | **0.881** |

#### Information Retrieval
* Dataset: `dim_256`
* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)

| Metric              | Value      |
|:--------------------|:-----------|
| cosine_accuracy@1   | 0.7969     |
| cosine_accuracy@3   | 0.9583     |
| cosine_accuracy@5   | 0.9688     |
| cosine_accuracy@10  | 0.9792     |
| cosine_precision@1  | 0.7969     |
| cosine_precision@3  | 0.3194     |
| cosine_precision@5  | 0.1937     |
| cosine_precision@10 | 0.0979     |
| cosine_recall@1     | 0.7969     |
| cosine_recall@3     | 0.9583     |
| cosine_recall@5     | 0.9688     |
| cosine_recall@10    | 0.9792     |
| cosine_ndcg@10      | 0.9011     |
| cosine_mrr@10       | 0.8746     |
| **cosine_map@100**  | **0.8758** |

#### Information Retrieval
* Dataset: `dim_128`
* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)

| Metric              | Value      |
|:--------------------|:-----------|
| cosine_accuracy@1   | 0.7865     |
| cosine_accuracy@3   | 0.9323     |
| cosine_accuracy@5   | 0.9635     |
| cosine_accuracy@10  | 0.9635     |
| cosine_precision@1  | 0.7865     |
| cosine_precision@3  | 0.3108     |
| cosine_precision@5  | 0.1927     |
| cosine_precision@10 | 0.0964     |
| cosine_recall@1     | 0.7865     |
| cosine_recall@3     | 0.9323     |
| cosine_recall@5     | 0.9635     |
| cosine_recall@10    | 0.9635     |
| cosine_ndcg@10      | 0.8881     |
| cosine_mrr@10       | 0.8623     |
| **cosine_map@100**  | **0.8647** |

#### Information Retrieval
* Dataset: `dim_64`
* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)

| Metric              | Value      |
|:--------------------|:-----------|
| cosine_accuracy@1   | 0.6875     |
| cosine_accuracy@3   | 0.8646     |
| cosine_accuracy@5   | 0.9271     |
| cosine_accuracy@10  | 0.9688     |
| cosine_precision@1  | 0.6875     |
| cosine_precision@3  | 0.2882     |
| cosine_precision@5  | 0.1854     |
| cosine_precision@10 | 0.0969     |
| cosine_recall@1     | 0.6875     |
| cosine_recall@3     | 0.8646     |
| cosine_recall@5     | 0.9271     |
| cosine_recall@10    | 0.9688     |
| cosine_ndcg@10      | 0.8336     |
| cosine_mrr@10       | 0.7896     |
| **cosine_map@100**  | **0.7918** |

<!--
## Bias, Risks and Limitations

*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
-->

<!--
### Recommendations

*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
-->

## Training Details

### Training Hyperparameters
#### Non-Default Hyperparameters

- `eval_strategy`: epoch
- `per_device_eval_batch_size`: 16
- `learning_rate`: 2e-05
- `num_train_epochs`: 5
- `lr_scheduler_type`: cosine
- `warmup_ratio`: 0.1
- `load_best_model_at_end`: True

#### All Hyperparameters
<details><summary>Click to expand</summary>

- `overwrite_output_dir`: False
- `do_predict`: False
- `eval_strategy`: epoch
- `prediction_loss_only`: True
- `per_device_train_batch_size`: 8
- `per_device_eval_batch_size`: 16
- `per_gpu_train_batch_size`: None
- `per_gpu_eval_batch_size`: None
- `gradient_accumulation_steps`: 1
- `eval_accumulation_steps`: None
- `learning_rate`: 2e-05
- `weight_decay`: 0.0
- `adam_beta1`: 0.9
- `adam_beta2`: 0.999
- `adam_epsilon`: 1e-08
- `max_grad_norm`: 1.0
- `num_train_epochs`: 5
- `max_steps`: -1
- `lr_scheduler_type`: cosine
- `lr_scheduler_kwargs`: {}
- `warmup_ratio`: 0.1
- `warmup_steps`: 0
- `log_level`: passive
- `log_level_replica`: warning
- `log_on_each_node`: True
- `logging_nan_inf_filter`: True
- `save_safetensors`: True
- `save_on_each_node`: False
- `save_only_model`: False
- `restore_callback_states_from_checkpoint`: False
- `no_cuda`: False
- `use_cpu`: False
- `use_mps_device`: False
- `seed`: 42
- `data_seed`: None
- `jit_mode_eval`: False
- `use_ipex`: False
- `bf16`: False
- `fp16`: False
- `fp16_opt_level`: O1
- `half_precision_backend`: auto
- `bf16_full_eval`: False
- `fp16_full_eval`: False
- `tf32`: None
- `local_rank`: 0
- `ddp_backend`: None
- `tpu_num_cores`: None
- `tpu_metrics_debug`: False
- `debug`: []
- `dataloader_drop_last`: False
- `dataloader_num_workers`: 0
- `dataloader_prefetch_factor`: None
- `past_index`: -1
- `disable_tqdm`: False
- `remove_unused_columns`: True
- `label_names`: None
- `load_best_model_at_end`: True
- `ignore_data_skip`: False
- `fsdp`: []
- `fsdp_min_num_params`: 0
- `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
- `fsdp_transformer_layer_cls_to_wrap`: None
- `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
- `deepspeed`: None
- `label_smoothing_factor`: 0.0
- `optim`: adamw_torch
- `optim_args`: None
- `adafactor`: False
- `group_by_length`: False
- `length_column_name`: length
- `ddp_find_unused_parameters`: None
- `ddp_bucket_cap_mb`: None
- `ddp_broadcast_buffers`: False
- `dataloader_pin_memory`: True
- `dataloader_persistent_workers`: False
- `skip_memory_metrics`: True
- `use_legacy_prediction_loop`: False
- `push_to_hub`: False
- `resume_from_checkpoint`: None
- `hub_model_id`: None
- `hub_strategy`: every_save
- `hub_private_repo`: False
- `hub_always_push`: False
- `gradient_checkpointing`: False
- `gradient_checkpointing_kwargs`: None
- `include_inputs_for_metrics`: False
- `eval_do_concat_batches`: True
- `fp16_backend`: auto
- `push_to_hub_model_id`: None
- `push_to_hub_organization`: None
- `mp_parameters`: 
- `auto_find_batch_size`: False
- `full_determinism`: False
- `torchdynamo`: None
- `ray_scope`: last
- `ddp_timeout`: 1800
- `torch_compile`: False
- `torch_compile_backend`: None
- `torch_compile_mode`: None
- `dispatch_batches`: None
- `split_batches`: None
- `include_tokens_per_second`: False
- `include_num_input_tokens_seen`: False
- `neftune_noise_alpha`: None
- `optim_target_modules`: None
- `batch_eval_metrics`: False
- `eval_on_start`: False
- `batch_sampler`: batch_sampler
- `multi_dataset_batch_sampler`: proportional

</details>

### Training Logs
| Epoch   | Step   | Training Loss | dim_128_cosine_map@100 | dim_256_cosine_map@100 | dim_512_cosine_map@100 | dim_64_cosine_map@100 | dim_768_cosine_map@100 |
|:-------:|:------:|:-------------:|:----------------------:|:----------------------:|:----------------------:|:---------------------:|:----------------------:|
| 0.3846  | 5      | 5.0472        | -                      | -                      | -                      | -                     | -                      |
| 0.7692  | 10     | 4.0023        | -                      | -                      | -                      | -                     | -                      |
| 1.0     | 13     | -             | 0.7939                 | 0.8135                 | 0.8282                 | 0.7207                | 0.8323                 |
| 1.1538  | 15     | 2.3381        | -                      | -                      | -                      | -                     | -                      |
| 1.5385  | 20     | 3.4302        | -                      | -                      | -                      | -                     | -                      |
| 1.9231  | 25     | 2.08          | -                      | -                      | -                      | -                     | -                      |
| 2.0     | 26     | -             | 0.8494                 | 0.8681                 | 0.8781                 | 0.7959                | 0.8888                 |
| 2.3077  | 30     | 1.4696        | -                      | -                      | -                      | -                     | -                      |
| 2.6923  | 35     | 1.8153        | -                      | -                      | -                      | -                     | -                      |
| **3.0** | **39** | **-**         | **0.8641**             | **0.8844**             | **0.8924**             | **0.7952**            | **0.8997**             |
| 3.0769  | 40     | 1.3498        | -                      | -                      | -                      | -                     | -                      |
| 3.4615  | 45     | 0.9135        | -                      | -                      | -                      | -                     | -                      |
| 3.8462  | 50     | 1.3996        | -                      | -                      | -                      | -                     | -                      |
| 4.0     | 52     | -             | 0.8647                 | 0.8775                 | 0.8819                 | 0.7896                | 0.8990                 |
| 4.2308  | 55     | 1.1582        | -                      | -                      | -                      | -                     | -                      |
| 4.6154  | 60     | 1.2233        | -                      | -                      | -                      | -                     | -                      |
| 5.0     | 65     | 0.9757        | 0.8647                 | 0.8758                 | 0.8810                 | 0.7918                | 0.8981                 |

* The bold row denotes the saved checkpoint.

### Framework Versions
- Python: 3.10.12
- Sentence Transformers: 3.0.1
- Transformers: 4.42.4
- PyTorch: 2.3.1+cu121
- Accelerate: 0.32.1
- Datasets: 2.21.0
- Tokenizers: 0.19.1

## Citation

### BibTeX

#### Sentence Transformers
```bibtex
@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}
```

#### MatryoshkaLoss
```bibtex
@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning}, 
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}
```

#### MultipleNegativesRankingLoss
```bibtex
@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply}, 
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}
```

<!--
## Glossary

*Clearly define terms in order to be accessible across audiences.*
-->

<!--
## Model Card Authors

*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
-->

<!--
## Model Card Contact

*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
-->