--- tags: - sentence-transformers - sentence-similarity - feature-extraction - generated_from_trainer - dataset_size:100006 - loss:CachedMultipleNegativesRankingLoss base_model: answerdotai/ModernBERT-base widget: - source_sentence: how much weight can you lose in a week healthy? sentences: - Biology - 'Summary: According to experts, losing 1–2 pounds (0.45–0.9 kg) per week is a healthy and safe rate, while losing more than this is considered too fast. However, you may lose more than that during your first week of an exercise or diet plan.' - The number of valence electrons is the number of electrons in the outer shell, that the atom uses for bonding. Nitrogen has 5 electrons in its n=2 (outer) shell. - source_sentence: how long after having a baby can i get a tattoo? sentences: - It is suggested that mothers wait at least until 9-12 months after birth, when the child is no longer dependent solely on breastmilk before getting a tattoo. Reputable tattoo artists will have a waiver for the client to sign that asks about pregnancy and breastfeeding. - Medicine - Americans on average are down to 44 gallons of soda per year, and up to about 58 gallons of water. That's 7,242 ounces of water annually -- 20 ounces daily, which is 2.5 cups. - source_sentence: is all uhmw anti static? sentences: - The bacteria Streptococcus pyogenes causes it. It's most common in infants and children, but it frequently occurs in teenagers and adults as well. It causes white streaks or spots in the throat. - Chemistry - UHMW is available in a special anti-static grade that helps protect against EsD (static discharge) or to help keep dust and particles from building up on the product surface. The anti-static additives are built-in so the anti-static properties will last throughout the life of the material. - source_sentence: is closing cost tax deductible? sentences: - Medicine - 1 tablespoon (tbsp) of granulated sugar equals to 12.5998 grams (g) in granulated sugar mass. - In general, the only settlement or closing costs you can deduct are home mortgage interest and certain real estate taxes. You deduct them in the year you buy your home if you itemize your deductions. ... See IRS Publication 530, "Tax Information for Homeowners" and look for "Settlement or closing costs" for more details. - source_sentence: what is the connection between cancer and the cell cycle? sentences: - Biology - Conclusion. Cancer is unchecked cell growth. Mutations in genes can cause cancer by accelerating cell division rates or inhibiting normal controls on the system, such as cell cycle arrest or programmed cell death. As a mass of cancerous cells grows, it can develop into a tumor. - Your vomit may appear black if the blood has been oxidized by the acids in your stomach. The iron in your blood turns brown to black with time. Since the blood is no longer bright red, it means that the bleeding has either stopped or is only happening in a small amount. pipeline_tag: sentence-similarity library_name: sentence-transformers metrics: - cosine_accuracy@1 - cosine_accuracy@3 - cosine_accuracy@5 - cosine_accuracy@10 - cosine_precision@1 - cosine_precision@3 - cosine_precision@5 - cosine_precision@10 - cosine_recall@1 - cosine_recall@3 - cosine_recall@5 - cosine_recall@10 - cosine_ndcg@10 - cosine_mrr@10 - cosine_map@100 model-index: - name: SentenceTransformer based on answerdotai/ModernBERT-base results: - task: type: information-retrieval name: Information Retrieval dataset: name: NanoNQ type: NanoNQ metrics: - type: cosine_accuracy@1 value: 0.1 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.18 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.24 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.34 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.1 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.06 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.04800000000000001 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.034 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.1 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.15 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.21 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.31 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.19343658524041285 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.16590476190476192 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.17642959153410534 name: Cosine Map@100 - task: type: information-retrieval name: Information Retrieval dataset: name: NanoMSMARCO type: NanoMSMARCO metrics: - type: cosine_accuracy@1 value: 0.12 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.28 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.4 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.52 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.12 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.09333333333333332 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.08 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.052000000000000005 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.12 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.28 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.4 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.52 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.2984940860938879 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.2304365079365079 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.24691442502099614 name: Cosine Map@100 - task: type: nano-beir name: Nano BEIR dataset: name: NanoBEIR mean type: NanoBEIR_mean metrics: - type: cosine_accuracy@1 value: 0.11 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.23 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.32 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.43000000000000005 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.11 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.07666666666666666 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.064 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.043000000000000003 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.11 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.21500000000000002 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.305 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.41500000000000004 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.24596533566715037 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.1981706349206349 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.21167200827755073 name: Cosine Map@100 --- # SentenceTransformer based on answerdotai/ModernBERT-base This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) on the csv dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more. ## Model Details ### Model Description - **Model Type:** Sentence Transformer - **Base model:** [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) - **Maximum Sequence Length:** 8192 tokens - **Output Dimensionality:** 768 dimensions - **Similarity Function:** Cosine Similarity - **Training Dataset:** - csv ### Model Sources - **Documentation:** [Sentence Transformers Documentation](https://sbert.net) - **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers) - **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers) ### Full Model Architecture ``` SentenceTransformer( (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: ModernBertModel (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True}) ) ``` ## Usage ### Direct Usage (Sentence Transformers) First install the Sentence Transformers library: ```bash pip install -U sentence-transformers ``` Then you can load this model and run inference. ```python from sentence_transformers import SentenceTransformer # Download from the 🤗 Hub model = SentenceTransformer("sentence_transformers_model_id") # Run inference sentences = [ 'what is the connection between cancer and the cell cycle?', 'Conclusion. Cancer is unchecked cell growth. Mutations in genes can cause cancer by accelerating cell division rates or inhibiting normal controls on the system, such as cell cycle arrest or programmed cell death. As a mass of cancerous cells grows, it can develop into a tumor.', 'Biology', ] embeddings = model.encode(sentences) print(embeddings.shape) # [3, 768] # Get the similarity scores for the embeddings similarities = model.similarity(embeddings, embeddings) print(similarities.shape) # [3, 3] ``` ## Evaluation ### Metrics #### Information Retrieval * Datasets: `NanoNQ` and `NanoMSMARCO` * Evaluated with [InformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) | Metric | NanoNQ | NanoMSMARCO | |:--------------------|:-----------|:------------| | cosine_accuracy@1 | 0.1 | 0.12 | | cosine_accuracy@3 | 0.18 | 0.28 | | cosine_accuracy@5 | 0.24 | 0.4 | | cosine_accuracy@10 | 0.34 | 0.52 | | cosine_precision@1 | 0.1 | 0.12 | | cosine_precision@3 | 0.06 | 0.0933 | | cosine_precision@5 | 0.048 | 0.08 | | cosine_precision@10 | 0.034 | 0.052 | | cosine_recall@1 | 0.1 | 0.12 | | cosine_recall@3 | 0.15 | 0.28 | | cosine_recall@5 | 0.21 | 0.4 | | cosine_recall@10 | 0.31 | 0.52 | | **cosine_ndcg@10** | **0.1934** | **0.2985** | | cosine_mrr@10 | 0.1659 | 0.2304 | | cosine_map@100 | 0.1764 | 0.2469 | #### Nano BEIR * Dataset: `NanoBEIR_mean` * Evaluated with [NanoBEIREvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.NanoBEIREvaluator) | Metric | Value | |:--------------------|:----------| | cosine_accuracy@1 | 0.11 | | cosine_accuracy@3 | 0.23 | | cosine_accuracy@5 | 0.32 | | cosine_accuracy@10 | 0.43 | | cosine_precision@1 | 0.11 | | cosine_precision@3 | 0.0767 | | cosine_precision@5 | 0.064 | | cosine_precision@10 | 0.043 | | cosine_recall@1 | 0.11 | | cosine_recall@3 | 0.215 | | cosine_recall@5 | 0.305 | | cosine_recall@10 | 0.415 | | **cosine_ndcg@10** | **0.246** | | cosine_mrr@10 | 0.1982 | | cosine_map@100 | 0.2117 | ## Training Details ### Training Dataset #### csv * Dataset: csv * Size: 100,006 training samples * Columns: question, answer, and category * Approximate statistics based on the first 1000 samples: | | question | answer | category | |:--------|:----------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------| | type | string | string | string | | details | | | | * Samples: | question | answer | category | |:---------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:----------------------| | how many times a week should you use heat on your hair? | Don't style hair with heat every day. Hot tools can also make hair look crispy and create split ends if overused. Blow out hair 3-5 times a week and try to limit your flat iron/curling iron usage to 1-2 times a week.” | Medicine | | do african violets like to be root bound? | African violets only bloom when they're root bound. When it is time to repot, be sure to use an organic potting soil made specifically for African violets, such as Espoma's African Violet Mix. They flower best in small pots — choose one that's about a third of the diameter of their leaf spread. | Biology | | is pgwp exempt from lmia? | The PGWP is exempt from Labour Market Impact Assessment (LMIA) requirements. The candidate must have attended a recognized post-secondary school, or a secondary school that offers qualifying programs, for at least eight months. | Medicine | * Loss: [CachedMultipleNegativesRankingLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cachedmultiplenegativesrankingloss) with these parameters: ```json { "scale": 20.0, "similarity_fct": "cos_sim" } ``` ### Training Hyperparameters #### Non-Default Hyperparameters - `eval_strategy`: steps - `per_device_train_batch_size`: 256 - `per_device_eval_batch_size`: 256 - `learning_rate`: 0.0001 - `num_train_epochs`: 1 - `warmup_ratio`: 0.05 - `bf16`: True - `batch_sampler`: no_duplicates #### All Hyperparameters
Click to expand - `overwrite_output_dir`: False - `do_predict`: False - `eval_strategy`: steps - `prediction_loss_only`: True - `per_device_train_batch_size`: 256 - `per_device_eval_batch_size`: 256 - `per_gpu_train_batch_size`: None - `per_gpu_eval_batch_size`: None - `gradient_accumulation_steps`: 1 - `eval_accumulation_steps`: None - `torch_empty_cache_steps`: None - `learning_rate`: 0.0001 - `weight_decay`: 0.0 - `adam_beta1`: 0.9 - `adam_beta2`: 0.999 - `adam_epsilon`: 1e-08 - `max_grad_norm`: 1.0 - `num_train_epochs`: 1 - `max_steps`: -1 - `lr_scheduler_type`: linear - `lr_scheduler_kwargs`: {} - `warmup_ratio`: 0.05 - `warmup_steps`: 0 - `log_level`: passive - `log_level_replica`: warning - `log_on_each_node`: True - `logging_nan_inf_filter`: True - `save_safetensors`: True - `save_on_each_node`: False - `save_only_model`: False - `restore_callback_states_from_checkpoint`: False - `no_cuda`: False - `use_cpu`: False - `use_mps_device`: False - `seed`: 42 - `data_seed`: None - `jit_mode_eval`: False - `use_ipex`: False - `bf16`: True - `fp16`: False - `fp16_opt_level`: O1 - `half_precision_backend`: auto - `bf16_full_eval`: False - `fp16_full_eval`: False - `tf32`: None - `local_rank`: 0 - `ddp_backend`: None - `tpu_num_cores`: None - `tpu_metrics_debug`: False - `debug`: [] - `dataloader_drop_last`: False - `dataloader_num_workers`: 0 - `dataloader_prefetch_factor`: None - `past_index`: -1 - `disable_tqdm`: False - `remove_unused_columns`: True - `label_names`: None - `load_best_model_at_end`: False - `ignore_data_skip`: False - `fsdp`: [] - `fsdp_min_num_params`: 0 - `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False} - `fsdp_transformer_layer_cls_to_wrap`: None - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None} - `deepspeed`: None - `label_smoothing_factor`: 0.0 - `optim`: adamw_torch - `optim_args`: None - `adafactor`: False - `group_by_length`: False - `length_column_name`: length - `ddp_find_unused_parameters`: None - `ddp_bucket_cap_mb`: None - `ddp_broadcast_buffers`: False - `dataloader_pin_memory`: True - `dataloader_persistent_workers`: False - `skip_memory_metrics`: True - `use_legacy_prediction_loop`: False - `push_to_hub`: False - `resume_from_checkpoint`: None - `hub_model_id`: None - `hub_strategy`: every_save - `hub_private_repo`: None - `hub_always_push`: False - `gradient_checkpointing`: False - `gradient_checkpointing_kwargs`: None - `include_inputs_for_metrics`: False - `include_for_metrics`: [] - `eval_do_concat_batches`: True - `fp16_backend`: auto - `push_to_hub_model_id`: None - `push_to_hub_organization`: None - `mp_parameters`: - `auto_find_batch_size`: False - `full_determinism`: False - `torchdynamo`: None - `ray_scope`: last - `ddp_timeout`: 1800 - `torch_compile`: False - `torch_compile_backend`: None - `torch_compile_mode`: None - `dispatch_batches`: None - `split_batches`: None - `include_tokens_per_second`: False - `include_num_input_tokens_seen`: False - `neftune_noise_alpha`: None - `optim_target_modules`: None - `batch_eval_metrics`: False - `eval_on_start`: False - `use_liger_kernel`: False - `eval_use_gather_object`: False - `average_tokens_across_devices`: False - `prompts`: None - `batch_sampler`: no_duplicates - `multi_dataset_batch_sampler`: proportional
### Training Logs | Epoch | Step | Training Loss | NanoNQ_cosine_ndcg@10 | NanoMSMARCO_cosine_ndcg@10 | NanoBEIR_mean_cosine_ndcg@10 | |:------:|:----:|:-------------:|:---------------------:|:--------------------------:|:----------------------------:| | 0 | 0 | - | 0.0388 | 0.0863 | 0.0626 | | 0.0763 | 10 | 0.5482 | - | - | - | | 0.1527 | 20 | 0.1079 | - | - | - | | 0.2290 | 30 | 0.1491 | - | - | - | | 0.3053 | 40 | 0.1381 | - | - | - | | 0.3817 | 50 | 0.0873 | 0.0909 | 0.2197 | 0.1553 | | 0.4580 | 60 | 0.133 | - | - | - | | 0.5344 | 70 | 0.0539 | - | - | - | | 0.6107 | 80 | 0.029 | - | - | - | | 0.6870 | 90 | 0.0008 | - | - | - | | 0.7634 | 100 | 0.0997 | 0.1982 | 0.2657 | 0.2320 | | 0.8397 | 110 | 0.04 | - | - | - | | 0.9160 | 120 | 0.0053 | - | - | - | | 0.9924 | 130 | 0.0095 | - | - | - | | 1.0 | 131 | - | 0.1934 | 0.2985 | 0.2460 | ### Framework Versions - Python: 3.12.3 - Sentence Transformers: 3.3.1 - Transformers: 4.48.0.dev0 - PyTorch: 2.5.1 - Accelerate: 1.2.1 - Datasets: 3.2.0 - Tokenizers: 0.21.0 ## Citation ### BibTeX #### Sentence Transformers ```bibtex @inproceedings{reimers-2019-sentence-bert, title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", author = "Reimers, Nils and Gurevych, Iryna", booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", month = "11", year = "2019", publisher = "Association for Computational Linguistics", url = "https://arxiv.org/abs/1908.10084", } ``` #### CachedMultipleNegativesRankingLoss ```bibtex @misc{gao2021scaling, title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup}, author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan}, year={2021}, eprint={2101.06983}, archivePrefix={arXiv}, primaryClass={cs.LG} } ```