---
base_model: roneneldan/TinyStories-33M
tags:
- generated_from_trainer
metrics:
- accuracy
inference:
  parameters:
    max_new_tokens: 64
    do_sample: true
    repetition_penalty: 1.1
    no_repeat_ngram_size: 5
    guidance_scale: 1.01
    eta_cutoff: 0.001
widget:
- text: My name is El Microondas the Wise and
  example_title: El Microondas
- text: A meme is
  example_title: meme
- text: >-
    Barack Obama nominated Hilary Clinton as his secretary of state on Monday.
    He chose her because she had
  example_title: Coreference resolution
- text: >-
    On a shelf, there are five books: a gray book, a red book, a purple book, a
    blue book, and a black book
  example_title: Logic puzzles
- text: >-
    The two men running to become New York City's next mayor will face off in
    their first debate Wednesday night
  example_title: Reading comprehension
pipeline_tag: text-generation
datasets:
- pszemraj/simple_wikipedia_LM
license: apache-2.0
language:
- en
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# GPT-Neo-33M-simplewiki-2048-scratch

Initialized from random weights based on config from [roneneldan/TinyStories-33M](https://huggingface.co/roneneldan/TinyStories-33M), 3 epochs bf16.

It achieves the following results on the evaluation set:
- Loss: 3.9511
- Accuracy: 0.3843

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0005
- train_batch_size: 2
- eval_batch_size: 2
- seed: 80085
- gradient_accumulation_steps: 64
- total_train_batch_size: 128
- optimizer: Adam with betas=(0.9,0.99) and epsilon=1e-07
- lr_scheduler_type: inverse_sqrt
- lr_scheduler_warmup_ratio: 0.05
- num_epochs: 3.0

### Training results

| Training Loss | Epoch | Step | Validation Loss | Accuracy |
|:-------------:|:-----:|:----:|:---------------:|:--------:|
| 5.4676        | 0.45  | 100  | 5.0139          | 0.2811   |
| 5.1729        | 0.89  | 200  | 4.6737          | 0.3050   |
| 4.8702        | 1.34  | 300  | 4.4922          | 0.3170   |
| 4.5538        | 1.79  | 400  | 4.3026          | 0.3348   |
| 4.4818        | 2.23  | 500  | 4.0908          | 0.3649   |
| 4.4583        | 2.68  | 600  | 3.9511          | 0.3843   |


### Framework versions

- Transformers 4.33.1
- Pytorch 2.2.0.dev20230907+cu118
- Datasets 2.14.5
- Tokenizers 0.13.3