--- license: mit language: - en pipeline_tag: text-generation tags: - llama-2 - astronomy - astrophysics - arxiv inference: false base_model: - meta-llama/Llama-2-70b-hf --- # AstroLLaMA-2-70B-Base_AIC AstroLLaMA-2-70B-Base_AIC is a specialized base language model for astronomy, developed by fine-tuning Meta's LLaMA-2-70b architecture on astronomical literature. This model was developed by the AstroMLab team and is, to our best knowledge, the first specialized 70B parameter-level LLM in astronomy. It is designed for next token prediction tasks and is not an instruct/chat model. ## Model Details - **Base Architecture**: LLaMA-2-70b - **Training Data**: Abstract, Introduction, and Conclusion (AIC) sections from arXiv's astro-ph category papers (from arXiv's inception up to July 2023) - **Data Processing**: The training data was derived from LaTeX source files using regex-based extraction methods to identify and extract the relevant sections (Abstract, Introduction, and Conclusion). - **Fine-tuning Method**: Continual Pre-Training (CPT) using the LMFlow framework - **Training Details**: - Learning rate: 2 × 10⁻⁵ - Total batch size: 160 - Maximum token length: 2048 - Warmup ratio: 0.03 - Cosine decay schedule for learning rate reduction - Training duration: 1 epoch (approximately 2,000 A100 GPU hours) - **Primary Use**: Next token prediction for astronomy-related text generation and analysis - **Reference**: Pan et al. 2024 [Link to be added] ## Generating text from a prompt ```python from transformers import AutoModelForCausalLM, AutoTokenizer import torch # Load the model and tokenizer tokenizer = AutoTokenizer.from_pretrained("AstroMLab/astrollama-2-70b-base_aic") model = AutoModelForCausalLM.from_pretrained("AstroMLab/astrollama-2-70b-base_aic", device_map="auto") # Create the pipeline with explicit truncation from transformers import pipeline generator = pipeline( "text-generation", model=model, tokenizer=tokenizer, device_map="auto", truncation=True, max_length=512 ) # Example prompt from an astronomy paper prompt = "In this letter, we report the discovery of the highest redshift, " \ "heavily obscured, radio-loud QSO candidate selected using JWST NIRCam/MIRI, " \ "mid-IR, sub-mm, and radio imaging in the COSMOS-Web field. " # Set seed for reproducibility torch.manual_seed(42) # Generate text generated_text = generator(prompt, do_sample=True) print(generated_text[0]['generated_text']) ``` ## Model Performance and Significance AstroLLaMA-2-70B-Base_AIC demonstrates notable improvements over its baseline LLaMA-2-70B model, marking a crucial step in specialized astronomical LLMs. Here's a performance comparison chart based upon the astronomical benchmarking Q&A as described in [Ting et al. 2024](https://arxiv.org/abs/2407.11194), and Pan et al. 2024: | Model | Score (%) | |-------|-----------| | **AstroLLaMA-2-70B-Base_AIC (AstroMLab)** | **76.0** | | LLaMA-3.1-8B | 73.7 | | LLaMA-2-70B | 73.9 | | Gemma-2-9B | 71.5 | | Qwen-2.5-7B | 70.4 | | Yi-1.5-9B | 68.4 | | InternLM-2.5-7B | 64.0 | | Mistral-7B-v0.3 | 63.9 | | ChatGLM3-6B | 50.4 | It demonstrates that training specialized LLMs can be effective, especially at larger model scales. ## Ethical Considerations While this model is designed for scientific use, users should be mindful of potential misuse, such as generating misleading scientific content. Always verify model outputs against peer-reviewed sources for critical applications. ## Citation If you use this model in your research, please cite: ``` [Citation for Pan et al. 2024 to be added] ```