Add new SentenceTransformer model.

1_Pooling/config.json

@@ -0,0 +1,10 @@
+{
+  "word_embedding_dimension": 768,
+  "pooling_mode_cls_token": true,
+  "pooling_mode_mean_tokens": false,
+  "pooling_mode_max_tokens": false,
+  "pooling_mode_mean_sqrt_len_tokens": false,
+  "pooling_mode_weightedmean_tokens": false,
+  "pooling_mode_lasttoken": false,
+  "include_prompt": true
+}

README.md

@@ -0,0 +1,864 @@
+---
+base_model: BAAI/bge-base-en-v1.5
+datasets: []
+language:
+- en
+library_name: sentence-transformers
+license: apache-2.0
+metrics:
+- cosine_accuracy@1
+- cosine_accuracy@3
+- cosine_accuracy@5
+- cosine_accuracy@10
+- cosine_precision@1
+- cosine_precision@3
+- cosine_precision@5
+- cosine_precision@10
+- cosine_recall@1
+- cosine_recall@3
+- cosine_recall@5
+- cosine_recall@10
+- cosine_ndcg@10
+- cosine_mrr@10
+- cosine_map@100
+pipeline_tag: sentence-similarity
+tags:
+- sentence-transformers
+- sentence-similarity
+- feature-extraction
+- generated_from_trainer
+- dataset_size:200
+- loss:MatryoshkaLoss
+- loss:MultipleNegativesRankingLoss
+widget:
+- source_sentence: 'Verbalized number or word (e.g. “lowest”, “low”, “medium”, “high”,
+    “highest”), such as "Confidence: 60% / Medium".
+
+    Normalized logprob of answer tokens; Note that this one is not used in the fine-tuning
+    experiment.
+
+    Logprob of an indirect "True/False" token after the raw answer.
+
+    Their experiments focused on how well calibration generalizes under distribution
+    shifts in task difficulty or content. Each fine-tuning datapoint is a question,
+    the model’s answer (possibly incorrect), and a calibrated confidence. Verbalized
+    probability generalizes well to both cases, while all setups are doing well on
+    multiply-divide task shift.  Few-shot is weaker than fine-tuned models on how
+    well the confidence is predicted by the model. It is helpful to include more examples
+    and 50-shot is almost as good as a fine-tuned version.'
+  sentences:
+  - What is the relationship between the calibration of AI models and the effectiveness
+    of verbalized probabilities when applied to tasks of varying difficulty levels?
+  - In what ways does the F1 @ K metric contribute to evaluating the factual accuracy
+    and comprehensiveness of outputs generated by long-form language models?
+  - What impact does the implementation of a pretrained query-document relevance model
+    have on the process of document selection in research methodologies?
+- source_sentence: 'Fig. 4. Overview of SAFE for factuality evaluation of long-form
+    LLM generation. (Image source: Wei et al. 2024)
+
+    The SAFE evaluation metric is F1 @ K. The motivation is that model response for
+    long-form factuality should ideally hit both precision and recall, as the response
+    should be both
+
+
+    factual : measured by precision, the percentage of supported facts among all facts
+    in the entire response.
+
+    long : measured by recall, the percentage of provided facts among all relevant
+    facts that should appear in the response. Therefore we want to consider the number
+    of supported facts up to $K$.
+
+
+    Given the model response $y$, the metric F1 @ K is defined as:'
+  sentences:
+  - What methodologies does the agreement model employ to identify discrepancies between
+    the original and revised text, and how do these methodologies impact the overall
+    editing workflow?
+  - In what ways does the SAFE evaluation metric achieve a harmonious equilibrium
+    between precision and recall in the context of evaluating the factual accuracy
+    of long-form outputs generated by large language models?
+  - In what ways does the inherently adversarial structure of TruthfulQA inquiries
+    facilitate the detection of prevalent fallacies in human cognitive processes,
+    and what implications does this have for understanding the constraints of expansive
+    language models?
+- source_sentence: 'Non-context LLM: Prompt LLM directly with <atomic-fact> True or
+    False? without additional context.
+
+    Retrieval→LLM: Prompt with $k$ related passages retrieved from the knowledge source
+    as context.
+
+    Nonparametric probability (NP)): Compute the average likelihood of tokens in the
+    atomic fact by a masked LM and use that to make a prediction.
+
+    Retrieval→LLM + NP: Ensemble of two methods.
+
+
+    Some interesting observations on model hallucination behavior:
+
+
+    Error rates are higher for rarer entities in the task of biography generation.
+
+    Error rates are higher for facts mentioned later in the generation.
+
+    Using retrieval to ground the model generation significantly helps reduce hallucination.'
+  sentences:
+  - In what ways does the Rethinking with Retrieval (RR) methodology leverage Chain-of-Thought
+    (CoT) prompting to enhance the efficacy of external knowledge retrieval, and what
+    implications does this have for the precision of predictive outcomes generated
+    by models?
+  - In what ways does the retrieval of related passages contribute to minimizing hallucinations
+    in large language models, and what specific techniques can be employed to evaluate
+    the impact of this approach?
+  - What are the benefits of using retrieval methods in biography generation to minimize
+    inaccuracies, especially when compared to traditional prompting techniques that
+    lack context?
+- source_sentence: 'Yin et al. (2023) studies the concept of self-knowledge, referring
+    to whether language models know what they know or don’t know.
+
+    SelfAware, containing 1,032 unanswerable questions across five categories and
+    2,337 answerable questions. Unanswerable questions are sourced from online forums
+    with human annotations while answerable questions are sourced from SQuAD, HotpotQA
+    and TriviaQA based on text similarity with unanswerable questions. A question
+    may be unanswerable due to various reasons, such as no scientific consensus, imaginations
+    of the future, completely subjective, philosophical reasons that may yield multiple
+    responses, etc. Considering separating answerable vs unanswerable questions as
+    a binary classification task, we can measure F1-score or accuracy and the experiments
+    showed that larger models can do better at this task.'
+  sentences:
+  - What is the relationship between model size and performance metrics, such as F1-score
+    and accuracy, in the context of classifying questions into answerable and unanswerable
+    categories?
+  - How does the introduction of stochastic perturbations in synthetic training data
+    contribute to the enhancement of editor model efficacy within LangChain frameworks?
+  - How do the various output values linked to reflection tokens in the Self-RAG framework
+    impact the generation process, and why are they important?
+- source_sentence: 'Fig. 1. Knowledge categorization of close-book QA examples based
+    on how likely the model outputs correct answers. (Image source: Gekhman et al.
+    2024)
+
+    Some interesting observations of the experiments, where dev set accuracy is considered
+    a proxy for hallucinations.
+
+
+    Unknown examples are fitted substantially slower than Known.
+
+    The best dev performance is obtained when the LLM fits the majority of the Known
+    training examples but only a few of the Unknown ones. The model starts to hallucinate
+    when it learns most of the Unknown examples.
+
+    Among Known examples, MaybeKnown cases result in better overall performance, more
+    essential than HighlyKnown ones.'
+  sentences:
+  - In what ways does the fitting speed of examples that are not previously encountered
+    differ from that of familiar examples, and how does this variation influence the
+    overall accuracy of the model on the development set?
+  - What role do reflection tokens play in enhancing the efficiency of document retrieval
+    and generation within the Self-RAG framework?
+  - How do the results presented by Gekhman et al. in their 2024 study inform our
+    understanding of the reliability metrics associated with large language models
+    (LLMs) when subjected to fine-tuning with novel datasets?
+model-index:
+- name: BGE base Financial Matryoshka
+  results:
+  - task:
+      type: information-retrieval
+      name: Information Retrieval
+    dataset:
+      name: dim 768
+      type: dim_768
+    metrics:
+    - type: cosine_accuracy@1
+      value: 0.8802083333333334
+      name: Cosine Accuracy@1
+    - type: cosine_accuracy@3
+      value: 0.96875
+      name: Cosine Accuracy@3
+    - type: cosine_accuracy@5
+      value: 0.984375
+      name: Cosine Accuracy@5
+    - type: cosine_accuracy@10
+      value: 0.9947916666666666
+      name: Cosine Accuracy@10
+    - type: cosine_precision@1
+      value: 0.8802083333333334
+      name: Cosine Precision@1
+    - type: cosine_precision@3
+      value: 0.3229166666666667
+      name: Cosine Precision@3
+    - type: cosine_precision@5
+      value: 0.196875
+      name: Cosine Precision@5
+    - type: cosine_precision@10
+      value: 0.09947916666666667
+      name: Cosine Precision@10
+    - type: cosine_recall@1
+      value: 0.8802083333333334
+      name: Cosine Recall@1
+    - type: cosine_recall@3
+      value: 0.96875
+      name: Cosine Recall@3
+    - type: cosine_recall@5
+      value: 0.984375
+      name: Cosine Recall@5
+    - type: cosine_recall@10
+      value: 0.9947916666666666
+      name: Cosine Recall@10
+    - type: cosine_ndcg@10
+      value: 0.9433275174124347
+      name: Cosine Ndcg@10
+    - type: cosine_mrr@10
+      value: 0.9261284722222224
+      name: Cosine Mrr@10
+    - type: cosine_map@100
+      value: 0.9264025950292397
+      name: Cosine Map@100
+  - task:
+      type: information-retrieval
+      name: Information Retrieval
+    dataset:
+      name: dim 512
+      type: dim_512
+    metrics:
+    - type: cosine_accuracy@1
+      value: 0.8697916666666666
+      name: Cosine Accuracy@1
+    - type: cosine_accuracy@3
+      value: 0.9739583333333334
+      name: Cosine Accuracy@3
+    - type: cosine_accuracy@5
+      value: 0.9739583333333334
+      name: Cosine Accuracy@5
+    - type: cosine_accuracy@10
+      value: 0.9947916666666666
+      name: Cosine Accuracy@10
+    - type: cosine_precision@1
+      value: 0.8697916666666666
+      name: Cosine Precision@1
+    - type: cosine_precision@3
+      value: 0.3246527777777778
+      name: Cosine Precision@3
+    - type: cosine_precision@5
+      value: 0.1947916666666666
+      name: Cosine Precision@5
+    - type: cosine_precision@10
+      value: 0.09947916666666667
+      name: Cosine Precision@10
+    - type: cosine_recall@1
+      value: 0.8697916666666666
+      name: Cosine Recall@1
+    - type: cosine_recall@3
+      value: 0.9739583333333334
+      name: Cosine Recall@3
+    - type: cosine_recall@5
+      value: 0.9739583333333334
+      name: Cosine Recall@5
+    - type: cosine_recall@10
+      value: 0.9947916666666666
+      name: Cosine Recall@10
+    - type: cosine_ndcg@10
+      value: 0.939968526552219
+      name: Cosine Ndcg@10
+    - type: cosine_mrr@10
+      value: 0.9216269841269841
+      name: Cosine Mrr@10
+    - type: cosine_map@100
+      value: 0.9220610119047619
+      name: Cosine Map@100
+  - task:
+      type: information-retrieval
+      name: Information Retrieval
+    dataset:
+      name: dim 256
+      type: dim_256
+    metrics:
+    - type: cosine_accuracy@1
+      value: 0.8697916666666666
+      name: Cosine Accuracy@1
+    - type: cosine_accuracy@3
+      value: 0.9739583333333334
+      name: Cosine Accuracy@3
+    - type: cosine_accuracy@5
+      value: 0.984375
+      name: Cosine Accuracy@5
+    - type: cosine_accuracy@10
+      value: 1.0
+      name: Cosine Accuracy@10
+    - type: cosine_precision@1
+      value: 0.8697916666666666
+      name: Cosine Precision@1
+    - type: cosine_precision@3
+      value: 0.3246527777777778
+      name: Cosine Precision@3
+    - type: cosine_precision@5
+      value: 0.196875
+      name: Cosine Precision@5
+    - type: cosine_precision@10
+      value: 0.09999999999999999
+      name: Cosine Precision@10
+    - type: cosine_recall@1
+      value: 0.8697916666666666
+      name: Cosine Recall@1
+    - type: cosine_recall@3
+      value: 0.9739583333333334
+      name: Cosine Recall@3
+    - type: cosine_recall@5
+      value: 0.984375
+      name: Cosine Recall@5
+    - type: cosine_recall@10
+      value: 1.0
+      name: Cosine Recall@10
+    - type: cosine_ndcg@10
+      value: 0.9419747509776967
+      name: Cosine Ndcg@10
+    - type: cosine_mrr@10
+      value: 0.922676917989418
+      name: Cosine Mrr@10
+    - type: cosine_map@100
+      value: 0.922676917989418
+      name: Cosine Map@100
+  - task:
+      type: information-retrieval
+      name: Information Retrieval
+    dataset:
+      name: dim 128
+      type: dim_128
+    metrics:
+    - type: cosine_accuracy@1
+      value: 0.8541666666666666
+      name: Cosine Accuracy@1
+    - type: cosine_accuracy@3
+      value: 0.9583333333333334
+      name: Cosine Accuracy@3
+    - type: cosine_accuracy@5
+      value: 0.96875
+      name: Cosine Accuracy@5
+    - type: cosine_accuracy@10
+      value: 0.9947916666666666
+      name: Cosine Accuracy@10
+    - type: cosine_precision@1
+      value: 0.8541666666666666
+      name: Cosine Precision@1
+    - type: cosine_precision@3
+      value: 0.3194444444444445
+      name: Cosine Precision@3
+    - type: cosine_precision@5
+      value: 0.19374999999999998
+      name: Cosine Precision@5
+    - type: cosine_precision@10
+      value: 0.09947916666666667
+      name: Cosine Precision@10
+    - type: cosine_recall@1
+      value: 0.8541666666666666
+      name: Cosine Recall@1
+    - type: cosine_recall@3
+      value: 0.9583333333333334
+      name: Cosine Recall@3
+    - type: cosine_recall@5
+      value: 0.96875
+      name: Cosine Recall@5
+    - type: cosine_recall@10
+      value: 0.9947916666666666
+      name: Cosine Recall@10
+    - type: cosine_ndcg@10
+      value: 0.9306358745697197
+      name: Cosine Ndcg@10
+    - type: cosine_mrr@10
+      value: 0.9094328703703702
+      name: Cosine Mrr@10
+    - type: cosine_map@100
+      value: 0.9098668981481483
+      name: Cosine Map@100
+  - task:
+      type: information-retrieval
+      name: Information Retrieval
+    dataset:
+      name: dim 64
+      type: dim_64
+    metrics:
+    - type: cosine_accuracy@1
+      value: 0.7916666666666666
+      name: Cosine Accuracy@1
+    - type: cosine_accuracy@3
+      value: 0.953125
+      name: Cosine Accuracy@3
+    - type: cosine_accuracy@5
+      value: 0.9739583333333334
+      name: Cosine Accuracy@5
+    - type: cosine_accuracy@10
+      value: 0.9895833333333334
+      name: Cosine Accuracy@10
+    - type: cosine_precision@1
+      value: 0.7916666666666666
+      name: Cosine Precision@1
+    - type: cosine_precision@3
+      value: 0.3177083333333333
+      name: Cosine Precision@3
+    - type: cosine_precision@5
+      value: 0.1947916666666666
+      name: Cosine Precision@5
+    - type: cosine_precision@10
+      value: 0.09895833333333333
+      name: Cosine Precision@10
+    - type: cosine_recall@1
+      value: 0.7916666666666666
+      name: Cosine Recall@1
+    - type: cosine_recall@3
+      value: 0.953125
+      name: Cosine Recall@3
+    - type: cosine_recall@5
+      value: 0.9739583333333334
+      name: Cosine Recall@5
+    - type: cosine_recall@10
+      value: 0.9895833333333334
+      name: Cosine Recall@10
+    - type: cosine_ndcg@10
+      value: 0.9003914274568845
+      name: Cosine Ndcg@10
+    - type: cosine_mrr@10
+      value: 0.8705935846560847
+      name: Cosine Mrr@10
+    - type: cosine_map@100
+      value: 0.8713150853775854
+      name: Cosine Map@100
+---
+
+# BGE base Financial Matryoshka
+
+This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5). It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
+
+## Model Details
+
+### Model Description
+- **Model Type:** Sentence Transformer
+- **Base model:** [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) <!-- at revision a5beb1e3e68b9ab74eb54cfd186867f64f240e1a -->
+- **Maximum Sequence Length:** 512 tokens
+- **Output Dimensionality:** 768 tokens
+- **Similarity Function:** Cosine Similarity
+<!-- - **Training Dataset:** Unknown -->
+- **Language:** en
+- **License:** apache-2.0
+
+### Model Sources
+
+- **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
+- **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers)
+- **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers)
+
+### Full Model Architecture
+
+```
+SentenceTransformer(
+  (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel 
+  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
+  (2): Normalize()
+)
+```
+
+## Usage
+
+### Direct Usage (Sentence Transformers)
+
+First install the Sentence Transformers library:
+
+```bash
+pip install -U sentence-transformers
+```
+
+Then you can load this model and run inference.
+```python
+from sentence_transformers import SentenceTransformer
+
+# Download from the 🤗 Hub
+model = SentenceTransformer("joshuapb/fine-tuned-matryoshka-200")
+# Run inference
+sentences = [
+    'Fig. 1. Knowledge categorization of close-book QA examples based on how likely the model outputs correct answers. (Image source: Gekhman et al. 2024)\nSome interesting observations of the experiments, where dev set accuracy is considered a proxy for hallucinations.\n\nUnknown examples are fitted substantially slower than Known.\nThe best dev performance is obtained when the LLM fits the majority of the Known training examples but only a few of the Unknown ones. The model starts to hallucinate when it learns most of the Unknown examples.\nAmong Known examples, MaybeKnown cases result in better overall performance, more essential than HighlyKnown ones.',
+    'In what ways does the fitting speed of examples that are not previously encountered differ from that of familiar examples, and how does this variation influence the overall accuracy of the model on the development set?',
+    'How do the results presented by Gekhman et al. in their 2024 study inform our understanding of the reliability metrics associated with large language models (LLMs) when subjected to fine-tuning with novel datasets?',
+]
+embeddings = model.encode(sentences)
+print(embeddings.shape)
+# [3, 768]
+
+# Get the similarity scores for the embeddings
+similarities = model.similarity(embeddings, embeddings)
+print(similarities.shape)
+# [3, 3]
+```
+
+<!--
+### Direct Usage (Transformers)
+
+<details><summary>Click to see the direct usage in Transformers</summary>
+
+</details>
+-->
+
+<!--
+### Downstream Usage (Sentence Transformers)
+
+You can finetune this model on your own dataset.
+
+<details><summary>Click to expand</summary>
+
+</details>
+-->
+
+<!--
+### Out-of-Scope Use
+
+*List how the model may foreseeably be misused and address what users ought not to do with the model.*
+-->
+
+## Evaluation
+
+### Metrics
+
+#### Information Retrieval
+* Dataset: `dim_768`
+* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)
+
+| Metric              | Value      |
+|:--------------------|:-----------|
+| cosine_accuracy@1   | 0.8802     |
+| cosine_accuracy@3   | 0.9688     |
+| cosine_accuracy@5   | 0.9844     |
+| cosine_accuracy@10  | 0.9948     |
+| cosine_precision@1  | 0.8802     |
+| cosine_precision@3  | 0.3229     |
+| cosine_precision@5  | 0.1969     |
+| cosine_precision@10 | 0.0995     |
+| cosine_recall@1     | 0.8802     |
+| cosine_recall@3     | 0.9688     |
+| cosine_recall@5     | 0.9844     |
+| cosine_recall@10    | 0.9948     |
+| cosine_ndcg@10      | 0.9433     |
+| cosine_mrr@10       | 0.9261     |
+| **cosine_map@100**  | **0.9264** |
+
+#### Information Retrieval
+* Dataset: `dim_512`
+* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)
+
+| Metric              | Value      |
+|:--------------------|:-----------|
+| cosine_accuracy@1   | 0.8698     |
+| cosine_accuracy@3   | 0.974      |
+| cosine_accuracy@5   | 0.974      |
+| cosine_accuracy@10  | 0.9948     |
+| cosine_precision@1  | 0.8698     |
+| cosine_precision@3  | 0.3247     |
+| cosine_precision@5  | 0.1948     |
+| cosine_precision@10 | 0.0995     |
+| cosine_recall@1     | 0.8698     |
+| cosine_recall@3     | 0.974      |
+| cosine_recall@5     | 0.974      |
+| cosine_recall@10    | 0.9948     |
+| cosine_ndcg@10      | 0.94       |
+| cosine_mrr@10       | 0.9216     |
+| **cosine_map@100**  | **0.9221** |
+
+#### Information Retrieval
+* Dataset: `dim_256`
+* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)
+
+| Metric              | Value      |
+|:--------------------|:-----------|
+| cosine_accuracy@1   | 0.8698     |
+| cosine_accuracy@3   | 0.974      |
+| cosine_accuracy@5   | 0.9844     |
+| cosine_accuracy@10  | 1.0        |
+| cosine_precision@1  | 0.8698     |
+| cosine_precision@3  | 0.3247     |
+| cosine_precision@5  | 0.1969     |
+| cosine_precision@10 | 0.1        |
+| cosine_recall@1     | 0.8698     |
+| cosine_recall@3     | 0.974      |
+| cosine_recall@5     | 0.9844     |
+| cosine_recall@10    | 1.0        |
+| cosine_ndcg@10      | 0.942      |
+| cosine_mrr@10       | 0.9227     |
+| **cosine_map@100**  | **0.9227** |
+
+#### Information Retrieval
+* Dataset: `dim_128`
+* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)
+
+| Metric              | Value      |
+|:--------------------|:-----------|
+| cosine_accuracy@1   | 0.8542     |
+| cosine_accuracy@3   | 0.9583     |
+| cosine_accuracy@5   | 0.9688     |
+| cosine_accuracy@10  | 0.9948     |
+| cosine_precision@1  | 0.8542     |
+| cosine_precision@3  | 0.3194     |
+| cosine_precision@5  | 0.1937     |
+| cosine_precision@10 | 0.0995     |
+| cosine_recall@1     | 0.8542     |
+| cosine_recall@3     | 0.9583     |
+| cosine_recall@5     | 0.9688     |
+| cosine_recall@10    | 0.9948     |
+| cosine_ndcg@10      | 0.9306     |
+| cosine_mrr@10       | 0.9094     |
+| **cosine_map@100**  | **0.9099** |
+
+#### Information Retrieval
+* Dataset: `dim_64`
+* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)
+
+| Metric              | Value      |
+|:--------------------|:-----------|
+| cosine_accuracy@1   | 0.7917     |
+| cosine_accuracy@3   | 0.9531     |
+| cosine_accuracy@5   | 0.974      |
+| cosine_accuracy@10  | 0.9896     |
+| cosine_precision@1  | 0.7917     |
+| cosine_precision@3  | 0.3177     |
+| cosine_precision@5  | 0.1948     |
+| cosine_precision@10 | 0.099      |
+| cosine_recall@1     | 0.7917     |
+| cosine_recall@3     | 0.9531     |
+| cosine_recall@5     | 0.974      |
+| cosine_recall@10    | 0.9896     |
+| cosine_ndcg@10      | 0.9004     |
+| cosine_mrr@10       | 0.8706     |
+| **cosine_map@100**  | **0.8713** |
+
+<!--
+## Bias, Risks and Limitations
+
+*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
+-->
+
+<!--
+### Recommendations
+
+*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
+-->
+
+## Training Details
+
+### Training Hyperparameters
+#### Non-Default Hyperparameters
+
+- `eval_strategy`: epoch
+- `per_device_eval_batch_size`: 16
+- `learning_rate`: 2e-05
+- `num_train_epochs`: 5
+- `lr_scheduler_type`: cosine
+- `warmup_ratio`: 0.1
+- `load_best_model_at_end`: True
+
+#### All Hyperparameters
+<details><summary>Click to expand</summary>
+
+- `overwrite_output_dir`: False
+- `do_predict`: False
+- `eval_strategy`: epoch
+- `prediction_loss_only`: True
+- `per_device_train_batch_size`: 8
+- `per_device_eval_batch_size`: 16
+- `per_gpu_train_batch_size`: None
+- `per_gpu_eval_batch_size`: None
+- `gradient_accumulation_steps`: 1
+- `eval_accumulation_steps`: None
+- `learning_rate`: 2e-05
+- `weight_decay`: 0.0
+- `adam_beta1`: 0.9
+- `adam_beta2`: 0.999
+- `adam_epsilon`: 1e-08
+- `max_grad_norm`: 1.0
+- `num_train_epochs`: 5
+- `max_steps`: -1
+- `lr_scheduler_type`: cosine
+- `lr_scheduler_kwargs`: {}
+- `warmup_ratio`: 0.1
+- `warmup_steps`: 0
+- `log_level`: passive
+- `log_level_replica`: warning
+- `log_on_each_node`: True
+- `logging_nan_inf_filter`: True
+- `save_safetensors`: True
+- `save_on_each_node`: False
+- `save_only_model`: False
+- `restore_callback_states_from_checkpoint`: False
+- `no_cuda`: False
+- `use_cpu`: False
+- `use_mps_device`: False
+- `seed`: 42
+- `data_seed`: None
+- `jit_mode_eval`: False
+- `use_ipex`: False
+- `bf16`: False
+- `fp16`: False
+- `fp16_opt_level`: O1
+- `half_precision_backend`: auto
+- `bf16_full_eval`: False
+- `fp16_full_eval`: False
+- `tf32`: None
+- `local_rank`: 0
+- `ddp_backend`: None
+- `tpu_num_cores`: None
+- `tpu_metrics_debug`: False
+- `debug`: []
+- `dataloader_drop_last`: False
+- `dataloader_num_workers`: 0
+- `dataloader_prefetch_factor`: None
+- `past_index`: -1
+- `disable_tqdm`: False
+- `remove_unused_columns`: True
+- `label_names`: None
+- `load_best_model_at_end`: True
+- `ignore_data_skip`: False
+- `fsdp`: []
+- `fsdp_min_num_params`: 0
+- `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
+- `fsdp_transformer_layer_cls_to_wrap`: None
+- `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
+- `deepspeed`: None
+- `label_smoothing_factor`: 0.0
+- `optim`: adamw_torch
+- `optim_args`: None
+- `adafactor`: False
+- `group_by_length`: False
+- `length_column_name`: length
+- `ddp_find_unused_parameters`: None
+- `ddp_bucket_cap_mb`: None
+- `ddp_broadcast_buffers`: False
+- `dataloader_pin_memory`: True
+- `dataloader_persistent_workers`: False
+- `skip_memory_metrics`: True
+- `use_legacy_prediction_loop`: False
+- `push_to_hub`: False
+- `resume_from_checkpoint`: None
+- `hub_model_id`: None
+- `hub_strategy`: every_save
+- `hub_private_repo`: False
+- `hub_always_push`: False
+- `gradient_checkpointing`: False
+- `gradient_checkpointing_kwargs`: None
+- `include_inputs_for_metrics`: False
+- `eval_do_concat_batches`: True
+- `fp16_backend`: auto
+- `push_to_hub_model_id`: None
+- `push_to_hub_organization`: None
+- `mp_parameters`: 
+- `auto_find_batch_size`: False
+- `full_determinism`: False
+- `torchdynamo`: None
+- `ray_scope`: last
+- `ddp_timeout`: 1800
+- `torch_compile`: False
+- `torch_compile_backend`: None
+- `torch_compile_mode`: None
+- `dispatch_batches`: None
+- `split_batches`: None
+- `include_tokens_per_second`: False
+- `include_num_input_tokens_seen`: False
+- `neftune_noise_alpha`: None
+- `optim_target_modules`: None
+- `batch_eval_metrics`: False
+- `eval_on_start`: False
+- `batch_sampler`: batch_sampler
+- `multi_dataset_batch_sampler`: proportional
+
+</details>
+
+### Training Logs
+| Epoch   | Step    | Training Loss | dim_128_cosine_map@100 | dim_256_cosine_map@100 | dim_512_cosine_map@100 | dim_64_cosine_map@100 | dim_768_cosine_map@100 |
+|:-------:|:-------:|:-------------:|:----------------------:|:----------------------:|:----------------------:|:---------------------:|:----------------------:|
+| 0.2     | 5       | 5.2225        | -                      | -                      | -                      | -                     | -                      |
+| 0.4     | 10      | 4.956         | -                      | -                      | -                      | -                     | -                      |
+| 0.6     | 15      | 3.6388        | -                      | -                      | -                      | -                     | -                      |
+| 0.8     | 20      | 3.1957        | -                      | -                      | -                      | -                     | -                      |
+| 1.0     | 25      | 2.6928        | 0.8661                 | 0.8770                 | 0.8754                 | 0.8312                | 0.8871                 |
+| 1.2     | 30      | 2.5565        | -                      | -                      | -                      | -                     | -                      |
+| 1.4     | 35      | 1.5885        | -                      | -                      | -                      | -                     | -                      |
+| 1.6     | 40      | 2.1406        | -                      | -                      | -                      | -                     | -                      |
+| 1.8     | 45      | 2.193         | -                      | -                      | -                      | -                     | -                      |
+| 2.0     | 50      | 1.326         | 0.8944                 | 0.9110                 | 0.9028                 | 0.8615                | 0.9037                 |
+| 2.2     | 55      | 2.6832        | -                      | -                      | -                      | -                     | -                      |
+| 2.4     | 60      | 1.0584        | -                      | -                      | -                      | -                     | -                      |
+| 2.6     | 65      | 0.8853        | -                      | -                      | -                      | -                     | -                      |
+| 2.8     | 70      | 1.7129        | -                      | -                      | -                      | -                     | -                      |
+| 3.0     | 75      | 2.1856        | 0.9106                 | 0.9293                 | 0.9075                 | 0.8778                | 0.9266                 |
+| 3.2     | 80      | 1.7658        | -                      | -                      | -                      | -                     | -                      |
+| 3.4     | 85      | 1.9783        | -                      | -                      | -                      | -                     | -                      |
+| 3.6     | 90      | 1.9583        | -                      | -                      | -                      | -                     | -                      |
+| 3.8     | 95      | 1.2396        | -                      | -                      | -                      | -                     | -                      |
+| 4.0     | 100     | 1.1901        | 0.9073                 | 0.9253                 | 0.9151                 | 0.8750                | 0.9312                 |
+| 4.2     | 105     | 2.6547        | -                      | -                      | -                      | -                     | -                      |
+| 4.4     | 110     | 1.3485        | -                      | -                      | -                      | -                     | -                      |
+| 4.6     | 115     | 1.0767        | -                      | -                      | -                      | -                     | -                      |
+| 4.8     | 120     | 0.6663        | -                      | -                      | -                      | -                     | -                      |
+| **5.0** | **125** | **1.3869**    | **0.9099**             | **0.9227**             | **0.9221**             | **0.8713**            | **0.9264**             |
+
+* The bold row denotes the saved checkpoint.
+
+### Framework Versions
+- Python: 3.10.12
+- Sentence Transformers: 3.0.1
+- Transformers: 4.42.4
+- PyTorch: 2.3.1+cu121
+- Accelerate: 0.32.1
+- Datasets: 2.21.0
+- Tokenizers: 0.19.1
+
+## Citation
+
+### BibTeX
+
+#### Sentence Transformers
+```bibtex
+@inproceedings{reimers-2019-sentence-bert,
+    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
+    author = "Reimers, Nils and Gurevych, Iryna",
+    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
+    month = "11",
+    year = "2019",
+    publisher = "Association for Computational Linguistics",
+    url = "https://arxiv.org/abs/1908.10084",
+}
+```
+
+#### MatryoshkaLoss
+```bibtex
+@misc{kusupati2024matryoshka,
+    title={Matryoshka Representation Learning}, 
+    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
+    year={2024},
+    eprint={2205.13147},
+    archivePrefix={arXiv},
+    primaryClass={cs.LG}
+}
+```
+
+#### MultipleNegativesRankingLoss
+```bibtex
+@misc{henderson2017efficient,
+    title={Efficient Natural Language Response Suggestion for Smart Reply}, 
+    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
+    year={2017},
+    eprint={1705.00652},
+    archivePrefix={arXiv},
+    primaryClass={cs.CL}
+}
+```
+
+<!--
+## Glossary
+
+*Clearly define terms in order to be accessible across audiences.*
+-->
+
+<!--
+## Model Card Authors
+
+*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
+-->
+
+<!--
+## Model Card Contact
+
+*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
+-->

config.json

@@ -0,0 +1,32 @@
+{
+  "_name_or_path": "fine-tuned-models/fine-tuned-matryoshka-200",
+  "architectures": [
+    "BertModel"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "classifier_dropout": null,
+  "gradient_checkpointing": false,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "id2label": {
+    "0": "LABEL_0"
+  },
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "label2id": {
+    "LABEL_0": 0
+  },
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "torch_dtype": "float32",
+  "transformers_version": "4.42.4",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 30522
+}

config_sentence_transformers.json

@@ -0,0 +1,10 @@
+{
+  "__version__": {
+    "sentence_transformers": "3.0.1",
+    "transformers": "4.42.4",
+    "pytorch": "2.3.1+cu121"
+  },
+  "prompts": {},
+  "default_prompt_name": null,
+  "similarity_fn_name": null
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f8f444921efaebd00d38ea538f488f678edc38328e7382766665e882e6f00156
+size 437951328

modules.json

@@ -0,0 +1,20 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.models.Transformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_Pooling",
+    "type": "sentence_transformers.models.Pooling"
+  },
+  {
+    "idx": 2,
+    "name": "2",
+    "path": "2_Normalize",
+    "type": "sentence_transformers.models.Normalize"
+  }
+]

sentence_bert_config.json

@@ -0,0 +1,4 @@
+{
+  "max_seq_length": 512,
+  "do_lower_case": true
+}

special_tokens_map.json

@@ -0,0 +1,37 @@
+{
+  "cls_token": {
+    "content": "[CLS]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "[MASK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "[PAD]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "[SEP]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "[UNK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,64 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "101": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "102": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "103": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "do_basic_tokenize": true,
+  "do_lower_case": true,
+  "mask_token": "[MASK]",
+  "max_length": 512,
+  "model_max_length": 512,
+  "never_split": null,
+  "pad_to_multiple_of": null,
+  "pad_token": "[PAD]",
+  "pad_token_type_id": 0,
+  "padding_side": "right",
+  "sep_token": "[SEP]",
+  "stride": 0,
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "truncation_side": "right",
+  "truncation_strategy": "longest_first",
+  "unk_token": "[UNK]"
+}