arnosimons
commited on
Commit
•
579aaa7
1
Parent(s):
d0100a5
Update README.md
Browse files
README.md
CHANGED
@@ -5,11 +5,13 @@ language:
|
|
5 |
pipeline_tag: fill-mask
|
6 |
---
|
7 |
|
8 |
-
# Model Card for
|
9 |
|
10 |
**Astro-HEP-BERT** is a bidirectional transformer designed primarily to generate contextualized word embeddings for analyzing epistemic change in astrophysics and high-energy physics (<a target="_blank" rel="noopener noreferrer" href="https://doi.org/10.3030/101044932" >NEPI project</a> at TU Berlin). Built upon Google's "bert-base-uncased," the model underwent additional training for three epochs using approximately 21.5 million paragraphs extracted from around 600,000 scholarly articles sourced from arXiv, all pertaining to astrophysics and/or high-energy physics (HEP). The sole training objective was masked language modeling.
|
11 |
|
12 |
-
|
|
|
|
|
13 |
|
14 |
<!-- <a target="_blank" rel="noopener noreferrer" href="">Astro-HEP-BERT paper</a>. -->
|
15 |
|
|
|
5 |
pipeline_tag: fill-mask
|
6 |
---
|
7 |
|
8 |
+
# Model Card for Astro-HEP-BERT
|
9 |
|
10 |
**Astro-HEP-BERT** is a bidirectional transformer designed primarily to generate contextualized word embeddings for analyzing epistemic change in astrophysics and high-energy physics (<a target="_blank" rel="noopener noreferrer" href="https://doi.org/10.3030/101044932" >NEPI project</a> at TU Berlin). Built upon Google's "bert-base-uncased," the model underwent additional training for three epochs using approximately 21.5 million paragraphs extracted from around 600,000 scholarly articles sourced from arXiv, all pertaining to astrophysics and/or high-energy physics (HEP). The sole training objective was masked language modeling.
|
11 |
|
12 |
+
The Astro-HEP-BERT project embodies the spirit of a tabletop experiment or grassroots scientific effort. It exclusively utilized open-source inputs during training, and the entire training process was completed on a single MacBook Pro M2/96GB over a span of 6 weeks for 3 epochs. This project stands as a proof of concept, showcasing the viability of employing a bidirectional transformer for research ventures in the history, philosophy, and sociology of science (HPSS) even with limited financial resources.
|
13 |
+
|
14 |
+
For further insights into the model, the corpus, and the underlying research project please refer to the Astro-HEP-BERT paper [link coming soon].
|
15 |
|
16 |
<!-- <a target="_blank" rel="noopener noreferrer" href="">Astro-HEP-BERT paper</a>. -->
|
17 |
|