SentenceTransformer

This is a sentence-transformers model trained on the bge-full-data dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Maximum Sequence Length: 512 tokens
Output Dimensionality: 768 dimensions
Similarity Function: Cosine Similarity
Training Dataset:
- bge-full-data

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: ModernBertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("NohTow/ModernBERT-base-DPR-fullneg-gte-0.0002")
# Run inference
sentences = [
    'Who won 23 World Rally Championships, two in particular with the Lancia Delta Group A rally car?',
    "Lancia Delta Group A The Lancia Delta Group A is a Group A rally car built for the Martini Lancia by Lancia to compete in the World Rally Championship. It is based upon the Lancia Delta road car and replaced the Lancia Delta S4. The car was introduced for the 1987 World Rally Championship season and dominated the World Rally Championship, scoring 46 WRC victories overall and winning the constructors' championship a record six times in a row from 1987 to 1992, in addition to drivers' championship titles for Juha Kankkunen (1987 and 1991) and Miki Biasion (1988 and 1989), making Lancia the most successful marque in the history of the WRC and the Delta the most successful car.",
    'Luis Moya Luis Rodríguez Moya, better known as Luis Moya (born 23 September 1960 in La Coruña, Spain) is a now-retired rally co-driver, synonymous with driver Carlos Sainz. He is the third most successful co-driver in the history of the World Rally Championship (WRC), after Daniel Elena and Timo Rautiainen',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

Datasets: NanoClimateFEVER, NanoDBPedia, NanoFEVER, NanoFiQA2018, NanoHotpotQA, NanoMSMARCO, NanoNFCorpus, NanoNQ, NanoQuoraRetrieval, NanoSCIDOCS, NanoArguAna, NanoSciFact and NanoTouche2020
Evaluated with InformationRetrievalEvaluator

Metric	NanoClimateFEVER	NanoDBPedia	NanoFEVER	NanoFiQA2018	NanoHotpotQA	NanoMSMARCO	NanoNFCorpus	NanoNQ	NanoQuoraRetrieval	NanoSCIDOCS	NanoArguAna	NanoSciFact	NanoTouche2020
cosine_accuracy@1	0.22	0.7	0.88	0.46	0.82	0.34	0.36	0.46	0.94	0.48	0.26	0.34	0.4898
cosine_accuracy@3	0.52	0.74	0.96	0.62	0.94	0.6	0.5	0.66	0.98	0.66	0.64	0.48	0.8367
cosine_accuracy@5	0.6	0.8	1.0	0.68	0.94	0.72	0.56	0.7	0.98	0.74	0.8	0.54	0.898
cosine_accuracy@10	0.64	0.86	1.0	0.74	0.96	0.82	0.62	0.78	1.0	0.86	0.9	0.6	0.9796
cosine_precision@1	0.22	0.7	0.88	0.46	0.82	0.34	0.36	0.46	0.94	0.48	0.26	0.34	0.4898
cosine_precision@3	0.2067	0.48	0.3333	0.2867	0.3867	0.2	0.2933	0.22	0.4067	0.3333	0.2133	0.18	0.5034
cosine_precision@5	0.144	0.432	0.208	0.224	0.248	0.144	0.296	0.144	0.252	0.276	0.16	0.128	0.4653
cosine_precision@10	0.084	0.376	0.108	0.134	0.132	0.082	0.22	0.084	0.136	0.202	0.09	0.07	0.3612
cosine_recall@1	0.0883	0.0726	0.8267	0.2445	0.41	0.34	0.0156	0.45	0.8173	0.1017	0.26	0.305	0.0355
cosine_recall@3	0.2667	0.1134	0.9233	0.4038	0.58	0.6	0.0349	0.61	0.9453	0.2067	0.64	0.47	0.1075
cosine_recall@5	0.3083	0.1586	0.9533	0.489	0.62	0.72	0.0641	0.66	0.956	0.2847	0.8	0.54	0.1652
cosine_recall@10	0.3567	0.2345	0.9733	0.5964	0.66	0.82	0.0797	0.75	0.9933	0.4157	0.9	0.6	0.243
cosine_ndcg@10	0.284	0.4733	0.9203	0.4901	0.67	0.5747	0.2547	0.6061	0.9594	0.3972	0.5856	0.4572	0.418
cosine_mrr@10	0.3747	0.7389	0.9267	0.5513	0.8795	0.4967	0.4444	0.5691	0.9625	0.5928	0.4839	0.4177	0.6742
cosine_map@100	0.2232	0.3348	0.8908	0.4201	0.5984	0.505	0.0923	0.5645	0.9423	0.3043	0.4893	0.4156	0.308

Nano BEIR

Dataset: NanoBEIR_mean
Evaluated with NanoBEIREvaluator

Metric	Value
cosine_accuracy@1	0.5192
cosine_accuracy@3	0.7028
cosine_accuracy@5	0.766
cosine_accuracy@10	0.8277
cosine_precision@1	0.5192
cosine_precision@3	0.311
cosine_precision@5	0.2401
cosine_precision@10	0.1599
cosine_recall@1	0.3052
cosine_recall@3	0.454
cosine_recall@5	0.5169
cosine_recall@10	0.5864
cosine_ndcg@10	0.5454
cosine_mrr@10	0.624
cosine_map@100	0.4684

Training Details

Training Dataset

bge-full-data

Dataset: bge-full-data at 78f5c99
Size: 1,770,649 training samples
Columns: anchor, positive, negative_0, negative_1, negative_2, negative_3, and negative_4

Approximate statistics based on the first 1000 samples:

	anchor	positive	negative_0	negative_1	negative_2	negative_3	negative_4
type	string	string	string	string	string	string	string
details	min: 4 tokens mean: 20.15 tokens max: 512 tokens	min: 3 tokens mean: 173.18 tokens max: 512 tokens	min: 5 tokens mean: 170.06 tokens max: 512 tokens	min: 4 tokens mean: 167.88 tokens max: 512 tokens	min: 6 tokens mean: 167.95 tokens max: 512 tokens	min: 6 tokens mean: 166.32 tokens max: 512 tokens	min: 5 tokens mean: 167.63 tokens max: 512 tokens

Samples:

anchor	positive	negative_0	negative_1	negative_2	negative_3	negative_4
`What happens if you eat raw chicken?`	`What are the dangers of eating raw chicken?`	`Does all raw chicken have salmonella?`	`How safe is to eat chicken during pregnancy?`	`What meats are safe to eat raw?`	`What are some natural obligations of chicken?`	`Is it safe to eat raw egg?`
`how long does it take for a wren egg to hatch`	`How often does a mother Wren sit on her nest? I don't know for sure about how long Wrens usually spend on the nest at one sitting.. (Sorry couldn't resist the joke) However, the eggs usually hatch in 13-18 days, so if there were no hatchlings when that time elapsed, then you'd know for sure that she hadn't been behaving normally.`	`- When you are trying to hatch Tennessee red quail eggs, it will take approximately 23 days. You should perform lock down on the egg at 20 days. This is a period of time whe … n there should be no disturbances because hatching is likely to begin.urkey eggs usually take 21 to 28 days to hatch depending on what they are incubated in like an incubator or by a hen.`	`How long does it take an egg to hatch? For an average Eagle it would have a time for about 32-36 days, but the average time for an Eagle egg to hatch is about 35 days. 28 people found this useful.`	`- When you are trying to hatch Tennessee red quail eggs, it will take approximately 23 days. You should perform lock down on the egg at 20 days. This is a period of time whe … n there should be no disturbances because hatching is likely to begin.urkey eggs usually take 21 to 28 days to hatch depending on what they are incubated in like an incubator or by a hen. It also depends on how fertile it is and how it is cared … for.`	`- Actually this may vary depending on the kind of bird finch, the eggs hatch in between 12 - 16 days or 3 weeks.The nestlings fledge in 18 - 19 days.ctually this may vary depending on the kind of bird finch, the eggs hatch in between 12 - 16 days or 3 weeks.`	`- Welcome, and thanks for visiting the virtual home of the Whitestown Fire Department. Whether you’re stopping by to obtain information on our department, place a comment, track our progress and events, or just looking at the great pictures of our top notch personnel in action, we hope that you find what you’re after. Please feel free to provide feedback or contact us for any questions you may have.`
`can you have schizophrenia and bipolar`	Can you have both bipolar disorder and schizophrenia? Health Mental Health Can you have both bipolar disorder and schizophrenia? I'm 19 and was diagnosed with Bipolar Disorder almost 2 years ago. I also have some symptoms of schizophrenia such as auditory hallucinations and occasional visual ones as well and occasional paranoia. Ok the paranoia is pretty frequent. So yea, Can you have both of them? I know some of the symptoms can be... show more Follow 6 answers Answers Relevance Rating Newest Oldest Best Answer: yes you can, but some people with bipolar disorder have hallucinations and delusions from the bipolar disorder. only a psychiatrist could diagnose you i guess. Source (s):er nurse Zach · 9 years ago0 0 Comment Asker's rating Yes, one can have both bipolar disorder and schizophrenia, as the cause is one and the same - a spirit (ghost). Not only are the mood swings imparted by the associated spirit, but the alleged hallucinations are as well. The voices that those diagnosed as h...	Dual Diagnosis: Understanding Sex Addiction With Bipolar Disorder Dual Diagnosis: Understanding Sex Addiction With Bipolar Disorder February 5, 2015 Dual Diagnosis Bipolar disorder manifests itself in one college student’s “need” to sexually expose himself on campus. Marty was diagnosed with bipolar 1 disorder in the spring of his junior year in college. The symptoms had emerged during adolescence, but it wasn’t until a particularly startling manic episode that Marty’s doctor knew his depression was more than unipolar (i.e., clinical depression by itself). The gifted art student had painted his naked body in elaborate geometric patterns and shown up at the fountain in front of his university’s grand administrative building during the middle of a sunny afternoon. He proceeded to dramatically quote Michel Foucault’s Madness and Civilization, even as he was carried away by campus security. The combination of SSRIs and mood stabilizers prescribed to Marty for the treatment of bipolar disor...	Understanding Schizoaffective Disorder Medication Understanding Schizoaffective Disorder Medication Because schizoaffective disorder has symptoms of both psychosis and a mood disorder, ✱ doctors often prescribe different medicines to treat different symptoms of the condition. For example, they may prescribe: An antipsychotic, which helps symptoms like delusions and hallucinations A mood-stabilizing medicine, which can help level out “highs” and “lows”An antidepressant, which can help feelings of sadness, hopelessness, and difficulty with sleep and concentration One medicine for schizoaffective disorder's symptoms INVEGA SUSTENNA ® treats the symptoms of schizoaffective disorder (psychosis and mood), so it may be possible for you to manage symptoms with one medicine if your doctor feels it’s right for you. And that means one less pill to think about every day. Approved for the treatment of schizophrenia and schizoaffective disorder.✱ Please discuss your symptoms with your healthcare pro...	Paranoia and schizophrenia: What you need to know Newsletter MNT - Hourly Medical News Since 2003Search Log in Newsletter MNT - Hourly Medical News Since 2003Search Login Paranoia and schizophrenia: What you need to know Last updated Thu 25 May 2017By Yvette Brazier Reviewed by Timothy J. Legg, Ph D, CRNPOverview Symptoms Causes Diagnosis Treatment Complications A person who has a condition on the schizophrenia spectrum may experience delusions and what is commonly known as paranoia. These delusions may give rise to fears that others are plotting against the individual. Everyone can have a paranoid thought from time to time. On a rough day, we may find ourselves saying "Oh boy, the whole world is out to get me!" But we recognize that this is not the case. People with paranoia often have an extensive network of paranoid thoughts and ideas. This can result in a disproportionate amount of time spent thinking up ways for the individual to protect themselves from their perceived persecutors...	Same Genes Suspected in Both Depression and Bipolar Illness Same Genes Suspected in Both Depression and Bipolar Illness Increased Risk May Stem From Variation in Gene On/Off Switch January 28, 2010 • Science Update Protein produced by PBRM1 gene Researchers, for the first time, have pinpointed a genetic hotspot that confers risk for both bipolar disorder and depression. People with either of these mood disorders were significantly more likely to have risk versions of genes at this site than healthy controls. One of the genes, which codes for part of a cell's machinery that tells genes when to turn on and off, was also found to be over-expressed in the executive hub of bipolar patients' brains, making it a prime suspect. The results add to mounting evidence that major mental disorders overlap at the molecular level. "People who carry the risk versions may differ in some dimension of brain development that may increase risk for mood disorders later in life," explained Francis Mc Mahon, M...	`Schizophrenia Definition and Characteristics Schizophrenia Schizophrenia Definition and Characteristics Symptoms, Treatments and Risk Factors By Marcia Purse`

Loss: CachedMultipleNegativesRankingLoss with these parameters:

{
    "scale": 20.0,
    "similarity_fct": "cos_sim"
}

Training Hyperparameters

Non-Default Hyperparameters

eval_strategy: steps
per_device_train_batch_size: 2048
per_device_eval_batch_size: 2048
learning_rate: 0.0002
num_train_epochs: 2
warmup_ratio: 0.05
bf16: True
batch_sampler: no_duplicates

All Hyperparameters

Click to expand

overwrite_output_dir: False
do_predict: False
eval_strategy: steps
prediction_loss_only: True
per_device_train_batch_size: 2048
per_device_eval_batch_size: 2048
per_gpu_train_batch_size: None
per_gpu_eval_batch_size: None
gradient_accumulation_steps: 1
eval_accumulation_steps: None
torch_empty_cache_steps: None
learning_rate: 0.0002
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1.0
num_train_epochs: 2
max_steps: -1
lr_scheduler_type: linear
lr_scheduler_kwargs: {}
warmup_ratio: 0.05
warmup_steps: 0
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
save_safetensors: True
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
no_cuda: False
use_cpu: False
use_mps_device: False
seed: 42
data_seed: None
jit_mode_eval: False
use_ipex: False
bf16: True
fp16: False
fp16_opt_level: O1
half_precision_backend: auto
bf16_full_eval: False
fp16_full_eval: False
tf32: None
local_rank: 5
ddp_backend: None
tpu_num_cores: None
tpu_metrics_debug: False
debug: []
dataloader_drop_last: True
dataloader_num_workers: 0
dataloader_prefetch_factor: None
past_index: -1
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: False
ignore_data_skip: False
fsdp: []
fsdp_min_num_params: 0
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
fsdp_transformer_layer_cls_to_wrap: None
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch
optim_args: None
adafactor: False
group_by_length: False
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
use_legacy_prediction_loop: False
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: None
hub_always_push: False
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_inputs_for_metrics: False
include_for_metrics: []
eval_do_concat_batches: True
fp16_backend: auto
push_to_hub_model_id: None
push_to_hub_organization: None
mp_parameters:
auto_find_batch_size: False
full_determinism: False
torchdynamo: None
ray_scope: last
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
dispatch_batches: None
split_batches: None
include_tokens_per_second: False
include_num_input_tokens_seen: False
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
eval_on_start: False
use_liger_kernel: False
eval_use_gather_object: False
average_tokens_across_devices: False
prompts: None
batch_sampler: no_duplicates
multi_dataset_batch_sampler: proportional

Training Logs