--- license: apache-2.0 language: - en pipeline_tag: fill-mask widget: - text: "The Standard Model (SM) of [MASK] physics has been tested by many experiments over the last four decades and has been shown to successfully describe high energy particle interactions." example_title: "particle physics" - text: "Clear evidence for the production of a neutral boson with a measured mass of [MASK].0 ± 0.4 (stat) ± 0.4 (sys) GeV is presented." example_title: "126.0 ± 0.4 (stat) ± 0.4 (sys) GeV" - text: "An excess of [MASK] is observed above the expected background, with a local significance of 5.0 standard deviations, at a mass near 125 GeV, signalling the production of a new particle." example_title: "excess of events" - text: "On September 14, 2015 at 09:50:45 UTC the two [MASK] of the Laser Interferometer Gravitational-Wave Observatory simultaneously observed a transient gravitational-wave signal." example_title: "two detectors" - text: "These first images from the EHT achieve the highest [MASK] resolution in the history of ground-based VLBI." example_title: "angular resolution" - text: "We propose a comprehensive theory of [MASK] matter that explains the recent proliferation of unexpected observations in high-energy astrophysics." example_title: "dark matter" - text: "Formation of galaxy clusters corresponds to the collapse of the largest gravitationally bound overdensities in the initial [MASK] field and is accompanied by the most energetic phenomena since the Big Bang and by the complex interplay between gravity-induced dynamics of collapse and baryonic processes associated with galaxy formation." example_title: "initial density field" - text: "The Event [MASK] Telescope (EHT) has led to the first images of a supermassive black hole, revealing the central compact objects in the elliptical galaxy M87 and the Milky Way." example_title: "Event Horizon Telescope" datasets: - wikipedia - bookcorpus tags: - physics - astrophysics - high-energy physics (HEP) - history of science - philosophy of science - sociology of science - word embeddings - semantic shift detection - conceptual change - epistemic change - arXiv --- # Model Card for Astro-HEP-BERT **Astro-HEP-BERT** is a bidirectional transformer designed primarily to generate contextualized word embeddings for analyzing conceptual change in astrophysics and high-energy physics (HEP). Built upon Google's `bert-base-uncased`, the model underwent additional training for three epochs using approximately 21.5 million paragraphs extracted from around 600,000 scholarly articles sourced from arXiv, all pertaining to astrophysics and/or high-energy physics (HEP). The sole training objective was masked language modeling. The Astro-HEP-BERT project embodies the spirit of a tabletop experiment or grassroots scientific effort. It exclusively utilized open-source inputs during training, and the entire training process was completed on a single MacBook Pro M2/96GB in 48 days for 3 epochs. This project stands as a proof of concept, showcasing the viability of employing a bidirectional transformer for research ventures in the history, philosophy, and sociology of science (HPSS) even with limited financial resources. For further insights into the model, the corpus, and the underlying research project (Network Epistemology in Practice) please refer to the Astro-HEP-BERT paper [link coming soon]. ## Model Details - **Developer:** Arno Simons - **Funded by:** European Research Council (ERC) under Grant agreement ID: 101044932 - **Language (NLP):** English - **License:** apache-2.0 - **Parent model:** Google's `bert-base-uncased`