---
license: apache-2.0
base_model: EleutherAI/pythia-70m-deduped
tags:
- generated_from_trainer
model-index:
- name: grandpythia-200k-70m
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# grandpythia-200k-70m

This model is a fine-tuned version of [EleutherAI/pythia-70m-deduped](https://huggingface.co/EleutherAI/pythia-70m-deduped) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 0.8419

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 64
- eval_batch_size: 64
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- num_epochs: 1

### Training results

| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 1.1766        | 0.01  | 68   | 1.2007          |
| 1.0903        | 0.02  | 136  | 1.1284          |
| 1.0809        | 0.03  | 204  | 1.0993          |
| 1.0928        | 0.04  | 272  | 1.0712          |
| 0.989         | 0.05  | 340  | 1.0473          |
| 1.0044        | 0.06  | 408  | 1.0373          |
| 0.985         | 0.07  | 476  | 1.0241          |
| 1.0272        | 0.08  | 544  | 1.0130          |
| 1.0295        | 0.09  | 612  | 1.0036          |
| 1.0172        | 0.1   | 680  | 0.9985          |
| 0.9582        | 0.11  | 748  | 0.9924          |
| 1.0342        | 0.12  | 816  | 0.9916          |
| 1.0053        | 0.13  | 884  | 0.9844          |
| 0.9321        | 0.14  | 952  | 0.9798          |
| 0.9473        | 0.15  | 1020 | 0.9727          |
| 0.9197        | 0.16  | 1088 | 0.9688          |
| 0.9827        | 0.17  | 1156 | 0.9632          |
| 0.9423        | 0.18  | 1224 | 0.9613          |
| 0.9662        | 0.19  | 1292 | 0.9578          |
| 0.9417        | 0.2   | 1360 | 0.9549          |
| 0.9501        | 0.21  | 1428 | 0.9461          |
| 0.9744        | 0.22  | 1496 | 0.9466          |
| 0.8693        | 0.23  | 1564 | 0.9394          |
| 0.9467        | 0.24  | 1632 | 0.9393          |
| 0.9274        | 0.25  | 1700 | 0.9362          |
| 0.8793        | 0.26  | 1768 | 0.9338          |
| 0.99          | 0.27  | 1836 | 0.9276          |
| 0.8983        | 0.28  | 1904 | 0.9291          |
| 0.9177        | 0.29  | 1972 | 0.9246          |
| 0.9586        | 0.3   | 2040 | 0.9224          |
| 0.9364        | 0.31  | 2108 | 0.9178          |
| 0.9248        | 0.32  | 2176 | 0.9175          |
| 0.9294        | 0.33  | 2244 | 0.9171          |
| 0.9142        | 0.34  | 2312 | 0.9136          |
| 0.9533        | 0.35  | 2380 | 0.9102          |
| 0.9193        | 0.36  | 2448 | 0.9094          |
| 0.9072        | 0.37  | 2516 | 0.9075          |
| 0.8927        | 0.38  | 2584 | 0.9043          |
| 0.9055        | 0.39  | 2652 | 0.9032          |
| 0.9276        | 0.4   | 2720 | 0.9030          |
| 0.8847        | 0.41  | 2788 | 0.8966          |
| 0.9449        | 0.42  | 2856 | 0.8963          |
| 0.8754        | 0.43  | 2924 | 0.8971          |
| 0.8612        | 0.44  | 2992 | 0.8935          |
| 0.9028        | 0.45  | 3060 | 0.8895          |
| 0.8641        | 0.46  | 3128 | 0.8925          |
| 0.8668        | 0.47  | 3196 | 0.8887          |
| 0.8935        | 0.48  | 3264 | 0.8863          |
| 0.8889        | 0.49  | 3332 | 0.8837          |
| 0.8854        | 0.5   | 3400 | 0.8849          |
| 0.8725        | 0.51  | 3468 | 0.8831          |
| 0.9425        | 0.52  | 3536 | 0.8796          |
| 0.8577        | 0.53  | 3604 | 0.8780          |
| 0.8281        | 0.54  | 3672 | 0.8747          |
| 0.9141        | 0.55  | 3740 | 0.8736          |
| 0.8684        | 0.56  | 3808 | 0.8738          |
| 0.8476        | 0.57  | 3876 | 0.8718          |
| 0.8761        | 0.58  | 3944 | 0.8735          |
| 0.8464        | 0.59  | 4012 | 0.8708          |
| 0.8732        | 0.6   | 4080 | 0.8681          |
| 0.9441        | 0.61  | 4148 | 0.8669          |
| 0.881         | 0.62  | 4216 | 0.8657          |
| 0.8635        | 0.63  | 4284 | 0.8640          |
| 0.827         | 0.64  | 4352 | 0.8625          |
| 0.9123        | 0.65  | 4420 | 0.8628          |
| 0.8557        | 0.66  | 4488 | 0.8605          |
| 0.8157        | 0.67  | 4556 | 0.8591          |
| 0.9008        | 0.68  | 4624 | 0.8580          |
| 0.8574        | 0.69  | 4692 | 0.8580          |
| 0.8374        | 0.7   | 4760 | 0.8563          |
| 0.8698        | 0.71  | 4828 | 0.8554          |
| 0.8817        | 0.72  | 4896 | 0.8545          |
| 0.8375        | 0.73  | 4964 | 0.8532          |
| 0.8504        | 0.74  | 5032 | 0.8524          |
| 0.8526        | 0.75  | 5100 | 0.8516          |
| 0.9306        | 0.76  | 5168 | 0.8511          |
| 0.7999        | 0.77  | 5236 | 0.8502          |
| 0.8337        | 0.78  | 5304 | 0.8495          |
| 0.7934        | 0.79  | 5372 | 0.8488          |
| 0.8159        | 0.8   | 5440 | 0.8480          |
| 0.7997        | 0.81  | 5508 | 0.8473          |
| 0.8909        | 0.82  | 5576 | 0.8470          |
| 0.852         | 0.83  | 5644 | 0.8461          |
| 0.8285        | 0.84  | 5712 | 0.8455          |
| 0.8437        | 0.85  | 5780 | 0.8448          |
| 0.8784        | 0.86  | 5848 | 0.8444          |
| 0.8123        | 0.87  | 5916 | 0.8440          |
| 0.8439        | 0.88  | 5984 | 0.8436          |
| 0.8847        | 0.89  | 6052 | 0.8433          |
| 0.8165        | 0.9   | 6120 | 0.8429          |
| 0.8405        | 0.91  | 6188 | 0.8427          |
| 0.8641        | 0.92  | 6256 | 0.8425          |
| 0.8536        | 0.93  | 6324 | 0.8424          |
| 0.8426        | 0.94  | 6392 | 0.8421          |
| 0.8547        | 0.95  | 6460 | 0.8421          |
| 0.8144        | 0.96  | 6528 | 0.8419          |
| 0.8475        | 0.97  | 6596 | 0.8419          |
| 0.8063        | 0.98  | 6664 | 0.8419          |
| 0.7943        | 0.99  | 6732 | 0.8419          |


### Framework versions

- Transformers 4.38.2
- Pytorch 2.1.0+cu121
- Datasets 2.18.0
- Tokenizers 0.15.2