phanerozoic/Tiny-Knight-1.1b-v0.1

Tiny Knight-1.1b-v0.1

Tiny Knight-1.1b-v0.1 is a specialized language model crafted for generating knight and medieval-themed content. This iteration is built upon the foundations of TinyLlama-1.1B-Chat-v1.0, tailored to operate within environments constrained by computing resources.

Performance

While this model excels in creating knight-themed narratives, its specialization, however, limits its effectiveness in broader language tasks, especially those requiring detailed knowledge outside the medieval theme.

Direct Use

Tiny Knight-1.1b-v0.1 is particularly suited for generating content within medieval, knightly, or fantasy settings, ideal for storytelling, educational content, and thematic exploration. It is not recommended for general-purpose tasks or technical domains.

Context Setting and Interaction Guidelines

Given its specialized nature, Tiny Knight-1.1b-v0.1 benefits significantly from detailed context-setting. Providing a rich thematic backdrop in prompts enhances the model's performance, guiding it to generate more accurate and immersive content.

Training Data

Incorporates a dataset focused on knightly tales, medieval history, and literature, derived from the foundational TinyLlama-1.1B model.

Custom Stopping Strings

Custom stopping strings were used to refine output quality:

• "},"
• "User:"
• "You:"
• "\nUser"
• "\nUser:"
• "me:"
• "user"
• "\n"

Training Hyperparameters and Fine-Tuning Details

• Base Model Name: TinyLlama-1.1B-Chat-v1.0
• Base Model Class: LlamaForCausalLM
• Projections: gate, down, up, q, k, v, o
• LoRA Rank: 16
• LoRA Alpha: 32
• True Batch Size: 32
• Gradient Accumulation Steps: 1
• Epochs: 0.18
• Learning Rate: 3e-4
• LR Scheduler: Linear
• Step: 75
• Loss: 1.87

Limitations

While adept at producing themed content, Tiny Knight-1.1b-v0.1's applicability is limited outside its specialized domain of knights and medieval themes.

Summary

Tiny Knight-1.1b-v0.1 represents a significant advancement in thematic language models, offering a specialized tool for exploring the medieval era. Its emphasis on context for optimal performance and the use of custom stopping strings make it a sophisticated asset for generating historically rich content.

Acknowledgments

Special thanks to the TinyLlama-1.1B team, whose pioneering work laid the groundwork for the creation of Tiny Knight-1.1b-v0.1.