ModernBioBERT
A modern variant of BioBERT based on ModernBERT. We continued the masked language modeling pre-training task for 1.000.000 steps on PubMed abstracts.
Pre-Training Details
Batchsize: 512
Learningrate: 1e-4
Warmupsteps: 500
Learning Rate Scheduler: Cosine Schedule
Max. Sequence Length: 512
Precision: bfloat16
datasets: - ncbi/pubmed language: - en base_model: - answerdotai/ModernBERT-base
- Downloads last month
- 10
Inference Providers
NEW
This model is not currently available via any of the supported third-party Inference Providers, and
HF Inference API was unable to determine this model's library.