pankajrajdeo/Bioformer-16L-UMLS-Pubmed-TCE-Epoch-1

SentenceTransformer

This is a sentence-transformers model trained on the parquet dataset. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Maximum Sequence Length: 1024 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • parquet

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 1024, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("pankajrajdeo/Bioformer-16L-UMLS-Pubmed-TCE-Epoch-1")
# Run inference
sentences = [
    'Evaluation of human antibodies from vaccinated volunteers for protection against Yersinia pestis infection.',
    'Yersinia pestis has a broad host range and has caused lethal bubonic and pneumonic plague in humans. With the emergence of multiple resistant strains and the potential for biothreat use, there is an urgent need for new therapeutic strategies that can protect populations from natural or deliberate infection. Targeting F1 has been proven to be the main strategy for developing vaccines and therapeutic antibodies, but data on anti-F1 antibodies, especially in humans, are scarce. To date, three human anti-F1 monoclonal antibodies (m252, αF1Ig2, and αF1Ig8) from naive populations have been reported. Here, we constructed an antibody library from vaccinees immunized with the plague subunit vaccine IIa by phage display. The genetic basis, epitopes, and biological functions of the obtained mAbs were assessed and evaluated in plague-challenged mice. Three human mAbs, namely, F3, F19, and F23, were identified. Their biolayer responses were 0.4, 0.6, and 0.6 nm, respectively. The dissociation constants (KD) of the F1 antigen were 1 pM, 0.165 nM, and 1 pM, respectively. Although derived from distinct Ab lineages, that is, VH3-30-D3-10-JH4 (F3&F23) and VH3-43-D6-19-JH4 (F19), these mAbs share similar binding sites in F1 with some overlap with αF1Ig8 but are distinct from αF1Ig2. Each of them provided a significant protective effect for Balb/c mice against a 100 median lethal dose (MLD) challenge of a virulent Y. pestis strain when administered at a dose of 100 μg. No synergistic or antagonistic effects were observed among them. These mAbs are novel and excellent candidates for further drug development and use in clinical practice.IMPORTANCEIn this study, we identified three human monoclonal antibodies with a high affinity to F1 protein of Yersinia pestis. We discovered that they have relatively lower somatic hypermutations compared with antibodies, m252, αF1Ig2, and αF1Ig8, derived from the naive library reported previously. We also observed that these mAbs share similar binding sites in F1 with some overlapping with αF1Ig8 but distinct from that of αF1Ig2. Furthermore, each of them could provide complete protection for mice against a lethal dose of Yersinia pestis challenge. Our data provided new insights into the anti-F1 Ab repertories and their associated epitopes during vaccination in humans. The findings support the additional novel protective human anti-F1Abs for potential therapeutics against plaque.',
    'BACKGROUND: Interventional therapies for severe pulmonary arterial hypertension (PAH) can provide right ventricular (RV) decompression and preserve cardiac output. Transcatheter stent placement in a residual ductus arteriosus (PDA) is one potentially effective option in critically ill infants and young children with PAH. We sought to assess recovery of RV function by echocardiographic strain in infants and young children following PDA stenting for acute PAH. METHODS: Retrospective review of patients < 2 years old who underwent PDA stenting for acute PAH. Clinical data were abstracted from the electronic medical record. RV strain (both total and free wall components) was assessed from echocardiographic images at baseline and 3, 6, and 12 months post-intervention, as well as at last echocardiogram. RESULTS: Nine patients underwent attempted ductal stenting for PAH. The median age at intervention was 38 days and median weight 3.7 kg. One-third (3of 9) of patients had PAH associated with a congenital diaphragmatic hernia. PDA stents were successfully deployed in eight patients. Mean RV total strain was - 14.9 ± 5.6% at baseline and improved to - 23.8 ± 2.2% at 6 months post-procedure (p < 0.001). Mean free wall RV strain was - 19.5 ± 5.4% at baseline and improved to - 27.7 ± 4.1% at 6 months (p = 0.002). Five patients survived to discharge, and four patients survived 1 year post-discharge. CONCLUSION: PDA stenting for severe, acute PAH can improve RV function as assessed by strain echocardiography. The quantitative improvement is more prominent in the first 6 months post-procedure and stabilizes thereafter.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Training Details

Training Dataset

parquet

• Dataset: parquet
• Size: 1,571,676 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 5 tokens mean: 18.78 tokens max: 58 tokens	min: 20 tokens mean: 184.85 tokens max: 1008 tokens

• Samples:

  anchor	positive
  Endotracheal Anæsthesia.	With certain exceptions, endotracheal anaesthesia is the best method for operations on the head and neck and for any other operation in which there may be a difficulty in controlling the patient's air-way. (2) Expiration should be provided for, in endotracheal anaesthesia, either by means of a second tube or by a tube of calibre sufficient to permit to-and-fro respiration. (3) Cocainization of the upper air-passages has decided advantages in endotracheal anaesthesia. (4) "Blind" intubation through the nose renders the method possible in cases where it is impossible to use a speculum. (5) The insufflation method is not specially indicated in abdominal surgery. (6) The routine use of endotracheal anaesthesia in teaching-hospitals for every class of case is detrimental to the production of sound anaesthetic knowledge in students who are likely to become general practitioners.
  A Further Contribution to the Subject of Aplastic Anæmia.	In the description of the condition of benign aplastic anaemia in pigs, attention is directed, among other things, to changes in the bone-marrow which seem of fundamental importance in understanding normal erythrogenesis.It is possible that an indirect van den Bergh reaction can be converted into an immediate direct by controlling the H-ion concentration suitably, either with buffer solutions or with less dissociable acids, such as acetic, used in preparing the van den Bergh reagent. The question of the interpretation of direct and indirect reactions, therefore, would seem to be reopened.Following Minot and Murphy's work, a pig with aplastic anaemia was fed with liver. The lesions in the liver, considered with the marked improvement in the blood which followed on liver feeding in this case, lead one to regard the condition as one of blood and marrow inefficiency due primarily to hepatic insufficiency.A comparison of pernicious anaemia with the benign aplastic anaemia of pigs seems to i...
  DISCUSSION ON THE TREATMENT OF URETHRAL STRICTURE AND FISTULÆ BY EXCISION.	For hard tunnel strictures and in cases of perineal fistula we should be bolder to adopt the principle of excision rather than that of mere external urethrotomy.A preliminary suprapubic cystotomy is advised, and also the avoidance, as far as possible, of the indwelling catheter. All fistulae should be excised completely, not merely opened, scraped and drained.The operator should not be in too great a hurry to pass sounds of too large a calibre after the operation, as in many cases there is little tendency to re-formation of the stricture.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Evaluation Dataset

parquet

• Dataset: parquet
• Size: 1,571,676 evaluation samples
• Columns: anchor and positive
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 5 tokens mean: 25.45 tokens max: 59 tokens	min: 4 tokens mean: 276.42 tokens max: 1022 tokens

• Samples:

  anchor	positive
  Impact of Forced Swimming Stress on Serum Adiponectin and Endothelin-1 Levels in Wistar Rats: Comparative Analysis of Dietary Effects.	Aim This study aimed to assess the impact of forced repeated swimming stress on serum adiponectin and endothelin-1 levels in Wistar rats, comparing the effects between those fed a standard diet and those on a high-fat diet. Methods Twenty adult male Wistar rats were divided into two dietary groups: a standard food diet group (NFD, n=10) and a high-fat diet group (HFD, n=10). Both groups underwent daily forced swimming stress for six days, with durations increasing from 5 to 30 minutes. The protocol finished in an acute bout of swimming exercise on the seventh day with a duration of 40 minutes. Adherence to ethical guidelines was strictly maintained, and serum adiponectin and endothelin-1 levels were measured pre- and post-exercise using the ELISA method. Results Before swimming, the mean adiponectin levels were 4.30±1.50 ng/mL in the NFD group and 3.53±0.70 ng/mL in the HFD group. Post-exercise, these levels significantly decreased to 2.4±0.84 ng/mL (p=0.003) and 1.58±0.23 ng/mL (p=0.0...
  A Case of Anti-Leucine-Rich Glioma-Inactivated Protein 1 (Anti-LGI1) Limbic Encephalitis With New-Onset Panic Attacks.	Anti-leucine-rich glioma-inactivated protein 1. This report discusses a unique onset of anti-LGI1 limbic encephalitis where an elderly female presented with symptoms of new-onset panic attacks and rhythmic facial movements for one week. She was then admitted to neurology for further serum, cerebrospinal fluid(CSF), and lab testing. She was eventually found to be positive for antibodies against LGI1 voltage-gated potassium channels, which confirmed the diagnosis of limbic encephalitis. The quick recognition of symptoms and escalation of management allowed the patient to experience drastic improvements after the initiation of steroids, immunotherapy, and lacosamide. Since anti-LGI1 limbic encephalitis is underdiagnosed, it can lead to irreversible long-term cognitive sequelae (that is, memory deficits). Thus, awareness of the typically associated findings of FBDS, cognitive disturbances, psychiatric changes, and hyponatremia can aid in early diagnosis and prompt treatment with immunother...
  A Case of Horner's Syndrome Aiding the Diagnosis of Internal Carotid Artery Dissection (ICAD): A Life-Saving Twist of Fate.	A 59-year-old male patient came to the outpatient department with complaints of left-sided hemicranial headache with drooping of the left upper eyelid (UL) for three days associated with difficulty in swallowing and deviation of the tongue. The patient had a history of vigorous coughing for the past 15 days for which he did not take any medications. He was thoroughly evaluated in the outpatient department and diagnosed with Horner's syndrome. Acute Horner's syndrome with pain is nearly a hallmark of carotid dissection, and MRI of the brain and orbit was thus advised. On MRI, a hyperdense area was noted around the left internal carotid artery for which he was advised magnetic resonance angiography, which revealed internal carotid artery dissection (ICAD) of the left side. The patient was diagnosed with left-sided Horner's syndrome following left ICAD with involvement of the left hypoglossal nerve. He was started on antiplatelets and anticoagulants and closely followed up. Early diagnosi...

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 128
• learning_rate: 2e-05
• num_train_epochs: 1
• max_steps: 11664
• log_level: info
• fp16: True
• dataloader_num_workers: 16
• load_best_model_at_end: True
• resume_from_checkpoint: True

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 128
• per_device_eval_batch_size: 8
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: 11664
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: info
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 16
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: True
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	Validation Loss
0.0001	1	0.2281	-
0.0857	1000	0.0192	-
0.1715	2000	0.0221	-
0.2572	3000	0.0133	-
0.3429	4000	0.0141	-
0.4286	5000	0.0145	-
0.5144	6000	0.0161	-
0.6001	7000	0.0133	-
0.6858	8000	0.0146	-
0.7715	9000	0.0137	-
0.8573	10000	0.0155	-
0.9430	11000	0.0138	-
0.9999	11664	-	0.0003

Framework Versions

• Python: 3.11.11
• Sentence Transformers: 3.4.1
• Transformers: 4.48.2
• PyTorch: 2.6.0+cu124
• Accelerate: 1.3.0
• Datasets: 3.2.0
• Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}