|
--- |
|
license: cc-by-nc-4.0 |
|
language: |
|
- en |
|
widget: |
|
- text: | |
|
Who are you? |
|
example_title: "Introduction" |
|
--- |
|
 |
|
# TinyViking-1.1B-v0.1 |
|
|
|
TinyViking-1.1B-v0.1 is a specialized language model designed for generating Viking-themed content. Developed by phanerozoic, this model is fine-tuned from TinyLlamaTinyLlama-1.1B-Chat-v1.0, optimized for environments with limited computing resources. |
|
|
|
### Performance |
|
TinyViking is capable of generating engaging Viking narratives, reflecting an understanding of Viking culture. However, it is not designed for general language tasks and may struggle with complex scientific or technical queries. |
|
|
|
### Direct Use |
|
Ideal for thematic language generation, particularly in settings like NPCs in games, where fun and thematic engagement are prioritized over detailed factual accuracy. |
|
|
|
### Training Data |
|
Trained on "The Saga of Grettir the Strong: Grettir's Saga" to ensure authentic thematic content. |
|
|
|
### Custom Stopping Strings |
|
Custom stopping strings are employed to enhance output quality: |
|
- "}," |
|
- "User:" |
|
- "You:" |
|
- "\nUser" |
|
- "\nUser:" |
|
- "me:" |
|
- "user" |
|
- "\n" |
|
|
|
### Training Hyperparameters and Fine-Tuning Details |
|
- **Learning Rate**: 2e-5 |
|
- **Epochs**: 1 |
|
- **Training Duration**: Approximately 5.6 minutes on an RTX 6000 Ada GPU |
|
- **LoRA Rank**: 2048 |
|
- **LoRA Alpha**: 4096 |
|
- **LoRA Dropout**: 0.05 |
|
- **Cutoff Length**: 256 |
|
- **Batch Size**: 4 (micro batch size) |
|
- **Warmup Steps**: 8 |
|
- **Optimizer**: adamw_torch |
|
- **Gradient Accumulation Steps**: 1 |
|
|
|
### Limitations |
|
Specialized in Viking dialect and narratives, TinyViking is less effective outside its thematic focus. |
|
|
|
### Compute Infrastructure |
|
Trained on an RTX 6000 Ada Lovelace GPU |
|
|
|
### Results |
|
Successfully generates Viking-themed responses, maintaining thematic consistency while displaying improved coherence and depth over previous models due to advancements in dataset generation and parsing. |
|
|
|
### Summary |
|
TinyViking-1.1B-v0.1 shows an improvement in quality compared to earlier thematic models, thanks to a new dataset generation method that helps to conserve the base model's already tenuous ability to hold a conversation. While it excels in Viking-themed interactions, its specialized focus limits broader application. |
|
|
|
### Acknowledgments |
|
Gratitude to the TinyLlama team, whose foundational work was, as always, essential for developing TinyViking. |
|
|
