File size: 1,566 Bytes
957343c 027812b 957343c 7bbad8c 027812b a893f57 7bbad8c a893f57 7bbad8c e1dbc62 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 |
---
license: bsd-3-clause
datasets:
- EleutherAI/pile
language:
- en
library_name: transformers
---
# Lovelace Medium Alpha1
551M parameter Transformer-XL style model trained on 100B tokens of The Pile!
This model was originally trained for the "Direct Prefrence Heads" paper, but will also be used as the basis for much of my future research.
All code used to train and run these models is available here: https://github.com/Avelina9X/direct-preference-heads and our paper is available here: https://arxiv.org/abs/2405.20053
## Model Architecture
| Name | Value |
| --- | --- |
| Total Parameters | 551M |
| Non-Embedding Parameters | 512M |
| Vocab Size | 50272 |
| \\(d_\text{vocab}\\) | 768 |
| \\(d_\text{model}\\) | 1536 |
| \\(n_\text{layers}\\) | 18 |
| FFN Activation | SwiGLU |
| \\(d_\text{ffn}\\) | 4096 |
| Attention Type | Full |
| Positon Embedding | Reversed RoPE with ABF |
| \\(n_\text{heads}\\) | 24 |
| \\(d_\text{key}\\) | 64 |
| Trained Context | 2048 |
| Trained Memory | 2048 |
| Max Inference Context | 4096 |
## Model Collection
| Model | Link |
| --- | --- |
| Pre-Trained Model | [lovelace-medium-alpha1](https://huggingface.co/Avelina/lovelace-medium-alpha1) |
| Fine-Tuned Model | [lovelace-medium-alpha1-sft](https://huggingface.co/Avelina/lovelace-medium-alpha1-sft) |
| DPH Aligned Model | [lovelace-medium-alpha1-dph](https://huggingface.co/Avelina/lovelace-medium-alpha1-dph) | |