Update README.md

README.md

@@ -12,93 +12,110 @@ metrics:
 - accuracy
 ---
 
-Model Card for GPT-NEO-1.3B-wiki
-Model Details
-Model Description
-This model is a fine-tuned version of EleutherAI/gpt-neo-1.3B. It has been fine-tuned on the Wikipedia dataset for tasks such as text generation, summarization, and question-answering in the English language. The model uses a causal language modeling objective and is capable of generating contextually coherent text.
-
-Developed by: Kimargin
-Model type: Fine-tuned model
-Language(s): English
-License: Apache 2.0
-Finetuned from model: EleutherAI/gpt-neo-1.3B
-Model Sources
-Repository: Kimargin/GPT-NEO-1.3B-wiki
-Uses
-Direct Use
-This model can be used directly for generating text, summarizing documents, and answering factual questions based on context. It is suitable for general-purpose NLP tasks where coherent and fluent text generation is needed.
-
-Downstream Use
-Users can fine-tune this model further for specialized tasks such as summarization of domain-specific texts (e.g., legal or medical texts), generating code, or answering specific types of questions.
-
-Out-of-Scope Use
-The model is not suitable for real-time decision-making in critical applications, such as medical or legal advice. It may produce biased or inaccurate text if given ambiguous or politically sensitive input.
-
-Bias, Risks, and Limitations
-This model was trained on Wikipedia data, which could contain biases inherent in the dataset. The model may reflect those biases in its output. Additionally, the model may not handle very specialized knowledge domains accurately.
-
-Recommendations
-Users should carefully review and verify the text generated by the model before using it in any critical applications. The model should be used in scenarios where generated outputs can be reviewed by a human to mitigate any potential biases or inaccuracies.
-
-How to Get Started with the Model
-To use this model, you can load it with the following code:
-
-python
-Copy code
+# Model Card for GPT-NEO-1.3B-wiki
+
+## Model Details
+
+### Model Description
+
+This model is based on [EleutherAI/gpt-neo-1.3B](https://huggingface.co/EleutherAI/gpt-neo-1.3B) and has been fine-tuned on the Wikipedia dataset. It is designed for text generation tasks such as summarization, question answering, and text completion in English. The model is fine-tuned to improve the fluency and factual accuracy of the generated content.
+
+- **Developed by:** Kimargin
+- **Model type:** Fine-tuned model
+- **Language(s):** English
+- **License:** Apache 2.0
+- **Finetuned from model:** EleutherAI/gpt-neo-1.3B
+
+### Model Sources
+
+- **Repository:** [Kimargin/GPT-NEO-1.3B-wiki](https://huggingface.co/Kimargin/GPT-NEO-1.3B-wiki)
+
+## Uses
+
+### Direct Use
+
+This model can be used for tasks like text generation, summarization, and question-answering. It is useful for generating coherent and factual text based on English-language prompts.
+
+### Downstream Use
+
+The model can be fine-tuned further for domain-specific applications such as legal or medical text generation, creating specialized question-answering systems, or generating structured content from prompts.
+
+### Out-of-Scope Use
+
+The model should not be used in critical applications (e.g., legal, medical, or financial advice) as it may generate biased, inaccurate, or misleading information. It is also not suited for real-time decision-making.
+
+## Bias, Risks, and Limitations
+
+Since the model was trained on Wikipedia data, it may inherit biases present in the dataset. Users should be cautious when using the model for generating sensitive or potentially biased content. The model may produce inaccurate or misleading text if given ambiguous or misleading prompts.
+
+### Recommendations
+
+Users should verify the outputs of the model, especially in critical use cases, and should not rely solely on the model for factual accuracy without human verification.
+
+## How to Get Started with the Model
+
+To use the model, you can load it as follows:
+
+```python
 from transformers import AutoModelForCausalLM, AutoTokenizer
 
 tokenizer = AutoTokenizer.from_pretrained("Kimargin/GPT-NEO-1.3B-wiki")
 model = AutoModelForCausalLM.from_pretrained("Kimargin/GPT-NEO-1.3B-wiki")
 
-input_text = "Explain the history of the internet."
+input_text = "What happened during World War II?"
 inputs = tokenizer(input_text, return_tensors="pt")
 outputs = model.generate(inputs["input_ids"], max_length=100)
 print(tokenizer.decode(outputs[0], skip_special_tokens=True))
-Training Details
-Training Data
-The model was fine-tuned on a subset of the English Wikipedia dataset, which includes a broad range of topics and domains. The dataset is generally factual but may still contain biases.
-
-Training Procedure
-The model was trained using mixed precision (float16) on GPU hardware.
-
-Training Hyperparameters
-Learning rate: 5e-5
-Batch size: 16
-Epochs: 3
-Precision: float16 (mixed precision)
-Evaluation
-Testing Data
-The model was evaluated on a held-out validation subset of the Wikipedia dataset.
-
-Factors
-General domain knowledge: The model performs well on generating factual and coherent text on common knowledge topics covered in Wikipedia.
-Contextual understanding: The model can maintain coherence over relatively long text sequences but may struggle with very specialized or niche topics.
-Metrics
-Perplexity: The model achieved a perplexity of 25.3 on the validation set.
-Accuracy: Measured by manual evaluation of text generation for accuracy in answering factual questions.
-Results
-The model demonstrates strong capabilities in general-purpose text generation and answering factual questions. However, it can generate irrelevant or biased responses in edge cases, especially with ambiguous input.
-
-Environmental Impact
-Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).
-
-Hardware Type: NVIDIA A100 GPUs
-Hours used: 20 hours
-Cloud Provider: Google Cloud
-Compute Region: US-Central
-Carbon Emitted: ~50 kg CO2
-Technical Specifications
-Model Architecture and Objective
-The model is a causal language model with 1.3 billion parameters based on the GPT-Neo architecture.
-
-Compute Infrastructure
+
+
+## Training Details
+
+### Training Data
+The model was fine-tuned on a subset of the Wikipedia dataset, which contains a broad range of general knowledge topics. This dataset was chosen to improve the model's capability to generate accurate, general-domain knowledge.
+
+### Training Procedure
+The model was fine-tuned using mixed precision (float16) on multiple GPUs for three epochs. The training was done to minimize perplexity and improve fluency in generated text.
+
+### Training Hyperparameters
+- **Learning rate:** 5e-5
+- **Batch size:** 16
+- **Epochs:** 3
+- **Precision:** float16 (mixed precision)
+
+## Evaluation
+
+### Testing Data
+The model was evaluated using a validation subset of the Wikipedia dataset to measure its performance on general text generation tasks.
+
+### Metrics
+- **Perplexity:** The model achieved a perplexity of 25.3 on the validation set.
+- **Accuracy:** The accuracy of the model in generating factual answers was evaluated qualitatively.
+
+### Results
+The model demonstrates good performance in generating coherent and contextually relevant text, but it may still struggle with niche or specialized topics that are underrepresented in its training data.
+
+## Environmental Impact
+Training large models like GPT-Neo has a significant carbon footprint due to the computational resources required. The estimated environmental impact of fine-tuning this model is as follows:
+
+- **Hardware Type:** NVIDIA A100 GPUs
+- **Hours used:** 20 hours
+- **Cloud Provider:** Google Cloud
+- **Compute Region:** US-Central
+- **Carbon Emitted:** ~50 kg CO2 (estimated using the [ML Impact calculator](https://mlco2.github.io/impact#compute))
+
+## Technical Specifications
+
+### Model Architecture and Objective
+The model is a causal language model with 1.3 billion parameters, based on the GPT-Neo architecture. It generates text by predicting the next word in a sequence, making it suitable for text completion and generation tasks.
+
+### Compute Infrastructure
 The model was trained on NVIDIA A100 GPUs using Google Cloud infrastructure.
 
-Citation
-If you use this model, please cite it as follows:
+## Citation
+
+If you use this model, please cite the original GPT-Neo model as follows:
 
-bibtex
-Copy code
+```bibtex
 @article{gpt-neo,
   author = {EleutherAI},
   title = {GPT-Neo: Large Scale Autoregressive Language Model},