File size: 1,634 Bytes
915f6e4 5ab2e7b 915f6e4 a7c4c00 ffcb927 a7c4c00 175dc21 a7c4c00 175dc21 a7c4c00 915f6e4 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 |
---
language:
- en
base_model: mistralai/Mistral-7B-v0.1
---
# Summary
The name is self-explanatory. This LoRA was trained on 50MB of text taken from Archive Of Our Own (AO3). In total, 1441 stories were selected from the Furry fandom category. I don't remember what filters I used. This LoRA is meant to improve a model's roleplaying capabilities, but I'll let you be the judge of that. Feel free to leave feedback, I'd like to hear your opinions on this LoRA.
# Dataset Settings
- Context length: 4096
- Epochs: 3
# LoRA Settings
- Rank: 128
- Alpha: 256
- Targeted modules: Q, K, V, O, Gate, Up, Down
- NEFTune alpha: 10 (to try to reduce overfitting)
- Learning rate: 1e-4
- Dropout: 0 (unsloth doesn't support LoRA dropout)
# Model Settings
- Base model: Mistral 7B
- Data Type: BF16, 4 bit quantization (thanks BitsandBytes)
# Misc Settings
- Batch size: 2
- Gradient Accumulation steps: 16
- LR Scheduler: Linear
# Software and Hardware
- Unsloth was used to speed up training.
- Training was done on 1x RTX 3090 (with 24 GB of VRAM) and took 11 hours.
# Warnings
- Obviously, having been trained on AO3 fanfics, this LoRA will probably increase the chances of a model generating 18+ content. Furthermore, it is possible that, if prompted to do so, the LoRA may help generate illegal content. So yknow, don't ask it to do that.
- Additionally, there is a chance this LoRA will output training data. The training graph seems to suggest that the LoRA was overfitting.
# Training Graph
 |