README.md

---
base_model: WhereIsAI/UAE-Large-V1
datasets: []
language:
- en
library_name: sentence-transformers
license: apache-2.0
metrics:
- cosine_accuracy@1
- cosine_accuracy@3
- cosine_accuracy@5
- cosine_accuracy@10
- cosine_precision@1
- cosine_precision@3
- cosine_precision@5
- cosine_precision@10
- cosine_recall@1
- cosine_recall@3
- cosine_recall@5
- cosine_recall@10
- cosine_ndcg@10
- cosine_mrr@10
- cosine_map@100
pipeline_tag: sentence-similarity
tags:
- sentence-transformers
- sentence-similarity
- feature-extraction
- generated_from_trainer
- dataset_size:3474
- loss:MatryoshkaLoss
- loss:MultipleNegativesRankingLoss
widget:
- source_sentence: Microsoft Corporation believes that its success is based upon its
    ability to transform to meet the needs of customers. Its growth strategy includes
    innovation across its cloud platforms and services, as well as investing in complementary
    businesses, products, services, and technologies to extend and grow its product
    offerings.
  sentences:
  - What factors caused the surge in Tesla’s stock prices in the first half of 2023?
  - What's Microsoft growth strategy in the cloud computing sector?
  - How has Microsoft Corporation performed in terms of stock prices over the past
    five years?
- source_sentence: Amazon reported the Q3 2023 earnings revealing a 21% year-over-year
    increase in the revenue, which stood at $116.38 billion. Net income increased
    57% to $6.66 billion, or $13.21 per diluted share, compared to $4.23 billion,
    or $8.42 per diluted share, in third quarter 2022. Amazon Web Services (AWS) revenue
    grew 32% in the quarter to $15 billion.
  sentences:
  - Can you tell about Amazon's Q3 2023 earnings?
  - What was the net income of Microsoft in Fiscal Year 2024?
  - What is the significance of EBITDA in financial analysis?
- source_sentence: For the fiscal year 2024, Walmart had an operating profit margin
    of 20%.
  sentences:
  - What is Pfizer's dividend yield for the financial year 2022?
  - What was Exxon Mobil Corporation's net income for the fourth quarter of 2023?
  - What is the operating profit margin for Walmart for the fiscal year 2024?
- source_sentence: The slowdown in construction, particularly in developing markets,
    resulted in a decrease in demand for Caterpillar's machinery and equipment, which
    negatively impacted the revenue for the year 2022.
  sentences:
  - How did the slow down in construction in 2022 affect Caterpillar's revenues?
  - What is JP Morgan's strategy when it comes to sustainability?
  - What was the debt-to-equity ratio for Tesla Inc in Q4 of 2022?
- source_sentence: According to Johnson & Johnson’s 2024 guidance report, their pharmaceutical
    sector was projected to grow by 7% in 2023 after considering crucial factors like
    the overall market demand, introduction of new drugs and potential impact of patent
    expirations.
  sentences:
  - What are Caterpillar's initiatives for enhancing its product sustainability?
  - How is JPMorgan Chase & Co. improving its cybersecurity measures?
  - What was the projected growth of Johnson & Johnson’s pharmaceutical sector in
    2023?
model-index:
- name: UAE-Large-V1-financial-embeddings-matryoshka
  results:
  - task:
      type: information-retrieval
      name: Information Retrieval
    dataset:
      name: dim 1024
      type: dim_1024
    metrics:
    - type: cosine_accuracy@1
      value: 0.8316062176165803
      name: Cosine Accuracy@1
    - type: cosine_accuracy@3
      value: 0.9326424870466321
      name: Cosine Accuracy@3
    - type: cosine_accuracy@5
      value: 0.966321243523316
      name: Cosine Accuracy@5
    - type: cosine_accuracy@10
      value: 0.9896373056994818
      name: Cosine Accuracy@10
    - type: cosine_precision@1
      value: 0.8316062176165803
      name: Cosine Precision@1
    - type: cosine_precision@3
      value: 0.31088082901554404
      name: Cosine Precision@3
    - type: cosine_precision@5
      value: 0.1932642487046632
      name: Cosine Precision@5
    - type: cosine_precision@10
      value: 0.09896373056994817
      name: Cosine Precision@10
    - type: cosine_recall@1
      value: 0.8316062176165803
      name: Cosine Recall@1
    - type: cosine_recall@3
      value: 0.9326424870466321
      name: Cosine Recall@3
    - type: cosine_recall@5
      value: 0.966321243523316
      name: Cosine Recall@5
    - type: cosine_recall@10
      value: 0.9896373056994818
      name: Cosine Recall@10
    - type: cosine_ndcg@10
      value: 0.9113990251008172
      name: Cosine Ndcg@10
    - type: cosine_mrr@10
      value: 0.8860854099843737
      name: Cosine Mrr@10
    - type: cosine_map@100
      value: 0.886565872062324
      name: Cosine Map@100
  - task:
      type: information-retrieval
      name: Information Retrieval
    dataset:
      name: dim 768
      type: dim_768
    metrics:
    - type: cosine_accuracy@1
      value: 0.8290155440414507
      name: Cosine Accuracy@1
    - type: cosine_accuracy@3
      value: 0.9326424870466321
      name: Cosine Accuracy@3
    - type: cosine_accuracy@5
      value: 0.966321243523316
      name: Cosine Accuracy@5
    - type: cosine_accuracy@10
      value: 0.9844559585492227
      name: Cosine Accuracy@10
    - type: cosine_precision@1
      value: 0.8290155440414507
      name: Cosine Precision@1
    - type: cosine_precision@3
      value: 0.31088082901554404
      name: Cosine Precision@3
    - type: cosine_precision@5
      value: 0.1932642487046632
      name: Cosine Precision@5
    - type: cosine_precision@10
      value: 0.09844559585492228
      name: Cosine Precision@10
    - type: cosine_recall@1
      value: 0.8290155440414507
      name: Cosine Recall@1
    - type: cosine_recall@3
      value: 0.9326424870466321
      name: Cosine Recall@3
    - type: cosine_recall@5
      value: 0.966321243523316
      name: Cosine Recall@5
    - type: cosine_recall@10
      value: 0.9844559585492227
      name: Cosine Recall@10
    - type: cosine_ndcg@10
      value: 0.9098442107332023
      name: Cosine Ndcg@10
    - type: cosine_mrr@10
      value: 0.8854439098610082
      name: Cosine Mrr@10
    - type: cosine_map@100
      value: 0.8863342112694444
      name: Cosine Map@100
  - task:
      type: information-retrieval
      name: Information Retrieval
    dataset:
      name: dim 512
      type: dim_512
    metrics:
    - type: cosine_accuracy@1
      value: 0.8238341968911918
      name: Cosine Accuracy@1
    - type: cosine_accuracy@3
      value: 0.9378238341968912
      name: Cosine Accuracy@3
    - type: cosine_accuracy@5
      value: 0.9637305699481865
      name: Cosine Accuracy@5
    - type: cosine_accuracy@10
      value: 0.9844559585492227
      name: Cosine Accuracy@10
    - type: cosine_precision@1
      value: 0.8238341968911918
      name: Cosine Precision@1
    - type: cosine_precision@3
      value: 0.3126079447322971
      name: Cosine Precision@3
    - type: cosine_precision@5
      value: 0.19274611398963729
      name: Cosine Precision@5
    - type: cosine_precision@10
      value: 0.09844559585492228
      name: Cosine Precision@10
    - type: cosine_recall@1
      value: 0.8238341968911918
      name: Cosine Recall@1
    - type: cosine_recall@3
      value: 0.9378238341968912
      name: Cosine Recall@3
    - type: cosine_recall@5
      value: 0.9637305699481865
      name: Cosine Recall@5
    - type: cosine_recall@10
      value: 0.9844559585492227
      name: Cosine Recall@10
    - type: cosine_ndcg@10
      value: 0.9085199240883707
      name: Cosine Ndcg@10
    - type: cosine_mrr@10
      value: 0.8836016530964717
      name: Cosine Mrr@10
    - type: cosine_map@100
      value: 0.8844289493397997
      name: Cosine Map@100
  - task:
      type: information-retrieval
      name: Information Retrieval
    dataset:
      name: dim 256
      type: dim_256
    metrics:
    - type: cosine_accuracy@1
      value: 0.8212435233160622
      name: Cosine Accuracy@1
    - type: cosine_accuracy@3
      value: 0.9326424870466321
      name: Cosine Accuracy@3
    - type: cosine_accuracy@5
      value: 0.961139896373057
      name: Cosine Accuracy@5
    - type: cosine_accuracy@10
      value: 0.9792746113989638
      name: Cosine Accuracy@10
    - type: cosine_precision@1
      value: 0.8212435233160622
      name: Cosine Precision@1
    - type: cosine_precision@3
      value: 0.31088082901554404
      name: Cosine Precision@3
    - type: cosine_precision@5
      value: 0.19222797927461138
      name: Cosine Precision@5
    - type: cosine_precision@10
      value: 0.09792746113989637
      name: Cosine Precision@10
    - type: cosine_recall@1
      value: 0.8212435233160622
      name: Cosine Recall@1
    - type: cosine_recall@3
      value: 0.9326424870466321
      name: Cosine Recall@3
    - type: cosine_recall@5
      value: 0.961139896373057
      name: Cosine Recall@5
    - type: cosine_recall@10
      value: 0.9792746113989638
      name: Cosine Recall@10
    - type: cosine_ndcg@10
      value: 0.9050964679750835
      name: Cosine Ndcg@10
    - type: cosine_mrr@10
      value: 0.8807097623159799
      name: Cosine Mrr@10
    - type: cosine_map@100
      value: 0.8817273654804927
      name: Cosine Map@100
  - task:
      type: information-retrieval
      name: Information Retrieval
    dataset:
      name: dim 128
      type: dim_128
    metrics:
    - type: cosine_accuracy@1
      value: 0.8186528497409327
      name: Cosine Accuracy@1
    - type: cosine_accuracy@3
      value: 0.9352331606217616
      name: Cosine Accuracy@3
    - type: cosine_accuracy@5
      value: 0.961139896373057
      name: Cosine Accuracy@5
    - type: cosine_accuracy@10
      value: 0.9792746113989638
      name: Cosine Accuracy@10
    - type: cosine_precision@1
      value: 0.8186528497409327
      name: Cosine Precision@1
    - type: cosine_precision@3
      value: 0.3117443868739206
      name: Cosine Precision@3
    - type: cosine_precision@5
      value: 0.19222797927461138
      name: Cosine Precision@5
    - type: cosine_precision@10
      value: 0.09792746113989637
      name: Cosine Precision@10
    - type: cosine_recall@1
      value: 0.8186528497409327
      name: Cosine Recall@1
    - type: cosine_recall@3
      value: 0.9352331606217616
      name: Cosine Recall@3
    - type: cosine_recall@5
      value: 0.961139896373057
      name: Cosine Recall@5
    - type: cosine_recall@10
      value: 0.9792746113989638
      name: Cosine Recall@10
    - type: cosine_ndcg@10
      value: 0.9031436826413919
      name: Cosine Ndcg@10
    - type: cosine_mrr@10
      value: 0.8781797433999506
      name: Cosine Mrr@10
    - type: cosine_map@100
      value: 0.8793080516202277
      name: Cosine Map@100
  - task:
      type: information-retrieval
      name: Information Retrieval
    dataset:
      name: dim 64
      type: dim_64
    metrics:
    - type: cosine_accuracy@1
      value: 0.7979274611398963
      name: Cosine Accuracy@1
    - type: cosine_accuracy@3
      value: 0.9222797927461139
      name: Cosine Accuracy@3
    - type: cosine_accuracy@5
      value: 0.9585492227979274
      name: Cosine Accuracy@5
    - type: cosine_accuracy@10
      value: 0.9792746113989638
      name: Cosine Accuracy@10
    - type: cosine_precision@1
      value: 0.7979274611398963
      name: Cosine Precision@1
    - type: cosine_precision@3
      value: 0.307426597582038
      name: Cosine Precision@3
    - type: cosine_precision@5
      value: 0.19170984455958548
      name: Cosine Precision@5
    - type: cosine_precision@10
      value: 0.09792746113989637
      name: Cosine Precision@10
    - type: cosine_recall@1
      value: 0.7979274611398963
      name: Cosine Recall@1
    - type: cosine_recall@3
      value: 0.9222797927461139
      name: Cosine Recall@3
    - type: cosine_recall@5
      value: 0.9585492227979274
      name: Cosine Recall@5
    - type: cosine_recall@10
      value: 0.9792746113989638
      name: Cosine Recall@10
    - type: cosine_ndcg@10
      value: 0.8935743388819871
      name: Cosine Ndcg@10
    - type: cosine_mrr@10
      value: 0.8654926391973025
      name: Cosine Mrr@10
    - type: cosine_map@100
      value: 0.8667278930244052
      name: Cosine Map@100
---

# UAE-Large-V1-financial-embeddings-matryoshka

This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [WhereIsAI/UAE-Large-V1](https://huggingface.co/WhereIsAI/UAE-Large-V1). It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

## Model Details

### Model Description
- **Model Type:** Sentence Transformer
- **Base model:** [WhereIsAI/UAE-Large-V1](https://huggingface.co/WhereIsAI/UAE-Large-V1) <!-- at revision 52d9e291d9fc7fc7f5276ff077b26fd1880c7c4f -->
- **Maximum Sequence Length:** 512 tokens
- **Output Dimensionality:** 1024 tokens
- **Similarity Function:** Cosine Similarity
<!-- - **Training Dataset:** Unknown -->
- **Language:** en
- **License:** apache-2.0

### Model Sources

- **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
- **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers)
- **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers)

### Full Model Architecture

```
SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)
```

## Usage

### Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

```bash
pip install -U sentence-transformers
```

Then you can load this model and run inference.
```python
from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("rbhatia46/UAE-Large-V1-financial-rag-matryoshka")
# Run inference
sentences = [
    'According to Johnson & Johnson’s 2024 guidance report, their pharmaceutical sector was projected to grow by 7% in 2023 after considering crucial factors like the overall market demand, introduction of new drugs and potential impact of patent expirations.',
    'What was the projected growth of Johnson & Johnson’s pharmaceutical sector in 2023?',
    'How is JPMorgan Chase & Co. improving its cybersecurity measures?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]
```

<!--
### Direct Usage (Transformers)

<details><summary>Click to see the direct usage in Transformers</summary>

</details>
-->

<!--
### Downstream Usage (Sentence Transformers)

You can finetune this model on your own dataset.

<details><summary>Click to expand</summary>

</details>
-->

<!--
### Out-of-Scope Use

*List how the model may foreseeably be misused and address what users ought not to do with the model.*
-->

## Evaluation

### Metrics

#### Information Retrieval
* Dataset: `dim_1024`
* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)

| Metric              | Value      |
|:--------------------|:-----------|
| cosine_accuracy@1   | 0.8316     |
| cosine_accuracy@3   | 0.9326     |
| cosine_accuracy@5   | 0.9663     |
| cosine_accuracy@10  | 0.9896     |
| cosine_precision@1  | 0.8316     |
| cosine_precision@3  | 0.3109     |
| cosine_precision@5  | 0.1933     |
| cosine_precision@10 | 0.099      |
| cosine_recall@1     | 0.8316     |
| cosine_recall@3     | 0.9326     |
| cosine_recall@5     | 0.9663     |
| cosine_recall@10    | 0.9896     |
| cosine_ndcg@10      | 0.9114     |
| cosine_mrr@10       | 0.8861     |
| **cosine_map@100**  | **0.8866** |

#### Information Retrieval
* Dataset: `dim_768`
* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)

| Metric              | Value      |
|:--------------------|:-----------|
| cosine_accuracy@1   | 0.829      |
| cosine_accuracy@3   | 0.9326     |
| cosine_accuracy@5   | 0.9663     |
| cosine_accuracy@10  | 0.9845     |
| cosine_precision@1  | 0.829      |
| cosine_precision@3  | 0.3109     |
| cosine_precision@5  | 0.1933     |
| cosine_precision@10 | 0.0984     |
| cosine_recall@1     | 0.829      |
| cosine_recall@3     | 0.9326     |
| cosine_recall@5     | 0.9663     |
| cosine_recall@10    | 0.9845     |
| cosine_ndcg@10      | 0.9098     |
| cosine_mrr@10       | 0.8854     |
| **cosine_map@100**  | **0.8863** |

#### Information Retrieval
* Dataset: `dim_512`
* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)

| Metric              | Value      |
|:--------------------|:-----------|
| cosine_accuracy@1   | 0.8238     |
| cosine_accuracy@3   | 0.9378     |
| cosine_accuracy@5   | 0.9637     |
| cosine_accuracy@10  | 0.9845     |
| cosine_precision@1  | 0.8238     |
| cosine_precision@3  | 0.3126     |
| cosine_precision@5  | 0.1927     |
| cosine_precision@10 | 0.0984     |
| cosine_recall@1     | 0.8238     |
| cosine_recall@3     | 0.9378     |
| cosine_recall@5     | 0.9637     |
| cosine_recall@10    | 0.9845     |
| cosine_ndcg@10      | 0.9085     |
| cosine_mrr@10       | 0.8836     |
| **cosine_map@100**  | **0.8844** |

#### Information Retrieval
* Dataset: `dim_256`
* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)

| Metric              | Value      |
|:--------------------|:-----------|
| cosine_accuracy@1   | 0.8212     |
| cosine_accuracy@3   | 0.9326     |
| cosine_accuracy@5   | 0.9611     |
| cosine_accuracy@10  | 0.9793     |
| cosine_precision@1  | 0.8212     |
| cosine_precision@3  | 0.3109     |
| cosine_precision@5  | 0.1922     |
| cosine_precision@10 | 0.0979     |
| cosine_recall@1     | 0.8212     |
| cosine_recall@3     | 0.9326     |
| cosine_recall@5     | 0.9611     |
| cosine_recall@10    | 0.9793     |
| cosine_ndcg@10      | 0.9051     |
| cosine_mrr@10       | 0.8807     |
| **cosine_map@100**  | **0.8817** |

#### Information Retrieval
* Dataset: `dim_128`
* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)

| Metric              | Value      |
|:--------------------|:-----------|
| cosine_accuracy@1   | 0.8187     |
| cosine_accuracy@3   | 0.9352     |
| cosine_accuracy@5   | 0.9611     |
| cosine_accuracy@10  | 0.9793     |
| cosine_precision@1  | 0.8187     |
| cosine_precision@3  | 0.3117     |
| cosine_precision@5  | 0.1922     |
| cosine_precision@10 | 0.0979     |
| cosine_recall@1     | 0.8187     |
| cosine_recall@3     | 0.9352     |
| cosine_recall@5     | 0.9611     |
| cosine_recall@10    | 0.9793     |
| cosine_ndcg@10      | 0.9031     |
| cosine_mrr@10       | 0.8782     |
| **cosine_map@100**  | **0.8793** |

#### Information Retrieval
* Dataset: `dim_64`
* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)

| Metric              | Value      |
|:--------------------|:-----------|
| cosine_accuracy@1   | 0.7979     |
| cosine_accuracy@3   | 0.9223     |
| cosine_accuracy@5   | 0.9585     |
| cosine_accuracy@10  | 0.9793     |
| cosine_precision@1  | 0.7979     |
| cosine_precision@3  | 0.3074     |
| cosine_precision@5  | 0.1917     |
| cosine_precision@10 | 0.0979     |
| cosine_recall@1     | 0.7979     |
| cosine_recall@3     | 0.9223     |
| cosine_recall@5     | 0.9585     |
| cosine_recall@10    | 0.9793     |
| cosine_ndcg@10      | 0.8936     |
| cosine_mrr@10       | 0.8655     |
| **cosine_map@100**  | **0.8667** |

<!--
## Bias, Risks and Limitations

*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
-->

<!--
### Recommendations

*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
-->

## Training Details

### Training Dataset

#### Unnamed Dataset


* Size: 3,474 training samples
* Columns: <code>positive</code> and <code>anchor</code>
* Approximate statistics based on the first 1000 samples:
  |         | positive                                                                            | anchor                                                                            |
  |:--------|:------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------|
  | type    | string                                                                              | string                                                                            |
  | details | <ul><li>min: 15 tokens</li><li>mean: 44.84 tokens</li><li>max: 112 tokens</li></ul> | <ul><li>min: 8 tokens</li><li>mean: 18.34 tokens</li><li>max: 32 tokens</li></ul> |
* Samples:
  | positive                                                                                                                                                                                                                                                                     | anchor                                                                                 |
  |:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------|
  | <code>Exxon Mobil faces substantial risk factors including fluctuating market prices for oil and gas, regulatory environment changes and the potential for catastrophic accidents such as oil spills.</code>                                                                 | <code>What is the key risk factor faced by Exxon Mobil in the energy sector?</code>    |
  | <code>Tesla’s remarkable revenue growth in 2023 is largely driven by its robust electric vehicle sales in China and the strong demand for its energy storage products.</code>                                                                                                | <code>What is the main reason behind Tesla’s revenue growth in 2023?</code>            |
  | <code>Amazon is expected to see a sales growth of 23% in the next financial year, driven by the increased demand for their ecommerce business and strong growth in AWS. This projection is subject to changes in the market condition and customer spending patterns.</code> | <code>What is the projected sales growth for Amazon in the next financial year?</code> |
* Loss: [<code>MatryoshkaLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#matryoshkaloss) with these parameters:
  ```json
  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          1024,
          768,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  ```

### Training Hyperparameters
#### Non-Default Hyperparameters

- `eval_strategy`: epoch
- `per_device_train_batch_size`: 32
- `per_device_eval_batch_size`: 16
- `gradient_accumulation_steps`: 16
- `learning_rate`: 2e-05
- `num_train_epochs`: 4
- `lr_scheduler_type`: cosine
- `warmup_ratio`: 0.1
- `bf16`: True
- `tf32`: True
- `load_best_model_at_end`: True
- `optim`: adamw_torch_fused
- `batch_sampler`: no_duplicates

#### All Hyperparameters
<details><summary>Click to expand</summary>

- `overwrite_output_dir`: False
- `do_predict`: False
- `eval_strategy`: epoch
- `prediction_loss_only`: True
- `per_device_train_batch_size`: 32
- `per_device_eval_batch_size`: 16
- `per_gpu_train_batch_size`: None
- `per_gpu_eval_batch_size`: None
- `gradient_accumulation_steps`: 16
- `eval_accumulation_steps`: None
- `learning_rate`: 2e-05
- `weight_decay`: 0.0
- `adam_beta1`: 0.9
- `adam_beta2`: 0.999
- `adam_epsilon`: 1e-08
- `max_grad_norm`: 1.0
- `num_train_epochs`: 4
- `max_steps`: -1
- `lr_scheduler_type`: cosine
- `lr_scheduler_kwargs`: {}
- `warmup_ratio`: 0.1
- `warmup_steps`: 0
- `log_level`: passive
- `log_level_replica`: warning
- `log_on_each_node`: True
- `logging_nan_inf_filter`: True
- `save_safetensors`: True
- `save_on_each_node`: False
- `save_only_model`: False
- `restore_callback_states_from_checkpoint`: False
- `no_cuda`: False
- `use_cpu`: False
- `use_mps_device`: False
- `seed`: 42
- `data_seed`: None
- `jit_mode_eval`: False
- `use_ipex`: False
- `bf16`: True
- `fp16`: False
- `fp16_opt_level`: O1
- `half_precision_backend`: auto
- `bf16_full_eval`: False
- `fp16_full_eval`: False
- `tf32`: True
- `local_rank`: 0
- `ddp_backend`: None
- `tpu_num_cores`: None
- `tpu_metrics_debug`: False
- `debug`: []
- `dataloader_drop_last`: False
- `dataloader_num_workers`: 0
- `dataloader_prefetch_factor`: None
- `past_index`: -1
- `disable_tqdm`: False
- `remove_unused_columns`: True
- `label_names`: None
- `load_best_model_at_end`: True
- `ignore_data_skip`: False
- `fsdp`: []
- `fsdp_min_num_params`: 0
- `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
- `fsdp_transformer_layer_cls_to_wrap`: None
- `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
- `deepspeed`: None
- `label_smoothing_factor`: 0.0
- `optim`: adamw_torch_fused
- `optim_args`: None
- `adafactor`: False
- `group_by_length`: False
- `length_column_name`: length
- `ddp_find_unused_parameters`: None
- `ddp_bucket_cap_mb`: None
- `ddp_broadcast_buffers`: False
- `dataloader_pin_memory`: True
- `dataloader_persistent_workers`: False
- `skip_memory_metrics`: True
- `use_legacy_prediction_loop`: False
- `push_to_hub`: False
- `resume_from_checkpoint`: None
- `hub_model_id`: None
- `hub_strategy`: every_save
- `hub_private_repo`: False
- `hub_always_push`: False
- `gradient_checkpointing`: False
- `gradient_checkpointing_kwargs`: None
- `include_inputs_for_metrics`: False
- `eval_do_concat_batches`: True
- `fp16_backend`: auto
- `push_to_hub_model_id`: None
- `push_to_hub_organization`: None
- `mp_parameters`: 
- `auto_find_batch_size`: False
- `full_determinism`: False
- `torchdynamo`: None
- `ray_scope`: last
- `ddp_timeout`: 1800
- `torch_compile`: False
- `torch_compile_backend`: None
- `torch_compile_mode`: None
- `dispatch_batches`: None
- `split_batches`: None
- `include_tokens_per_second`: False
- `include_num_input_tokens_seen`: False
- `neftune_noise_alpha`: None
- `optim_target_modules`: None
- `batch_eval_metrics`: False
- `batch_sampler`: no_duplicates
- `multi_dataset_batch_sampler`: proportional

</details>

### Training Logs
| Epoch      | Step   | Training Loss | dim_1024_cosine_map@100 | dim_128_cosine_map@100 | dim_256_cosine_map@100 | dim_512_cosine_map@100 | dim_64_cosine_map@100 | dim_768_cosine_map@100 |
|:----------:|:------:|:-------------:|:-----------------------:|:----------------------:|:----------------------:|:----------------------:|:---------------------:|:----------------------:|
| 0.8807     | 6      | -             | 0.8708                  | 0.8499                 | 0.8647                 | 0.8705                 | 0.8307                | 0.8700                 |
| 1.4679     | 10     | 0.7358        | -                       | -                      | -                      | -                      | -                     | -                      |
| 1.9083     | 13     | -             | 0.8848                  | 0.8724                 | 0.8782                 | 0.8861                 | 0.8617                | 0.8855                 |
| **2.9358** | **20** | **0.1483**    | **0.8865**              | **0.8793**             | **0.8814**             | **0.8857**             | **0.8667**            | **0.8863**             |
| 3.5229     | 24     | -             | 0.8866                  | 0.8793                 | 0.8817                 | 0.8844                 | 0.8667                | 0.8863                 |

* The bold row denotes the saved checkpoint.

### Framework Versions
- Python: 3.10.6
- Sentence Transformers: 3.0.1
- Transformers: 4.41.2
- PyTorch: 2.1.2+cu121
- Accelerate: 0.32.1
- Datasets: 2.19.1
- Tokenizers: 0.19.1

## Citation

### BibTeX

#### Sentence Transformers
```bibtex
@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}
```

#### MatryoshkaLoss
```bibtex
@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning}, 
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}
```

#### MultipleNegativesRankingLoss
```bibtex
@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply}, 
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}
```

<!--
## Glossary

*Clearly define terms in order to be accessible across audiences.*
-->

<!--
## Model Card Authors

*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
-->

<!--
## Model Card Contact

*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
-->