|
--- |
|
license: mit |
|
language: |
|
- en |
|
base_model: |
|
- meta-llama/Llama-3.1-70B-Instruct |
|
tags: |
|
- axolotl |
|
--- |
|
# 🎭 Cakrawala-Llama-3.1-70B |
|
|
|
> *Where Worlds Converge and Adventures Begin!* |
|
|
|
## 🌟 What's Special About This Model? |
|
|
|
Cakrawala-Llama-3.1-70B is a fine-tuned variant of the Llama-3.1-70B-Instruct model, specifically optimised for generating rich roleplaying conversations and character interactions. The model has been trained to excel at producing detailed, contextually appropriate character dialogues with rich descriptions of physical actions, expressions, and emotional states while maintaining consistent character voices and perspectives throughout extended interactions. |
|
|
|
## 🧪 The Secret Sauce |
|
|
|
### Training Diet: |
|
- Fed with 13,000 conversation pairs |
|
- Each conversation is a minimum 12-13 turns long |
|
- Focused heavily details like facial expressions, environmental descriptions, and character reactions that are focused a lot on **keeping the model in character.** |
|
|
|
### Tech Wizardry: |
|
- Trained on Llama-3.1-70B-Instruct |
|
- Fine-tuned using QLoRA |
|
- Trained over 2 epochs |
|
|
|
## Training Parameters |
|
- Gradient Accumulation Steps: 1 |
|
- Micro Batch Size: 4 |
|
- Learning Rate: 0.0002 |
|
- Optimizer: AdamW |
|
- Scheduler: Cosine |
|
- Mixed Precision: BF16 & FP16 with TF32 support |
|
|
|
## 🔧 Under the Hood |
|
- Trained on 8 x H100 SXM GPUs |
|
|
|
## 🎬 License & Credits |
|
|
|
- Licensed under MIT |
|
- Based on meta-llama/Llama-3.1-70B-Instruct |
|
|
|
--- |
|
|
|
*Built with ❤️ for roleplayers, by roleplayers* |
