File size: 2,515 Bytes
fa9e476 a05dcda fa9e476 a05dcda fa9e476 5a8082d 7b1dbe9 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 |
---
license: cc-by-nc-4.0
---
# This model is experimental and thus results cannot be gauranteed.
# Dendrite-L3-10B
In a similar vein to [Libra-19B](https://huggingface.co/Envoid/Libra-19B) this model was created by taking all of the layers of one model and stacking along with them the first number of layers (8 in this case) from a donor model but in the reverse order.
In this case the base model used was [Poppy_Porpoise-DADA-8B](https://huggingface.co/Envoid/Poppy_Porpoise-DADA-8B) and the donor model used was [Llama-3-8B-Instruct-DADA](https://huggingface.co/Envoid/Llama-3-8B-Instruct-DADA)
It was then finetuned for 10 epochs on the Dendrite dataset at a low learning rate to repair the disorder and integrate the donor layers.
The following mergekit config was used:
```
slices:
- sources:
- model: ./Poppy_Porpoise-DADA-8B
layer_range: [0, 32]
- sources:
- model: ./Llama-3-8B-Instruct-DADA
layer_range: [7, 8]
- sources:
- model: ./Llama-3-8B-Instruct-DADA
layer_range: [6, 7]
- sources:
- model: ./Llama-3-8B-Instruct-DADA
layer_range: [5, 6]
- sources:
- model: ./Llama-3-8B-Instruct-DADA
layer_range: [4, 5]
- sources:
- model: ./Llama-3-8B-Instruct-DADA
layer_range: [3, 4]
- sources:
- model: ./Llama-3-8B-Instruct-DADA
layer_range: [2, 3]
- sources:
- model: ./Llama-3-8B-Instruct-DADA
layer_range: [1, 2]
- sources:
- model: ./Llama-3-8B-Instruct-DADA
layer_range: [0, 1]
merge_method: passthrough
dtype: float16
```
Unlike in the case of Libra-19B this models moral alignment seems very much intact.
In order to get the best results from this model you should uncheck "skip special tokens" on your front-end and add "<|eot_id|>" to your custom stopping strings.
It has been tested with a number of different Llama-3 prompt templates and seems to work well.
It regained its base assistant personality during the retraining process, however, using assistant style prompt templates and assistant cards in SillyTavern gives it fairly interesting replies.
It has been tested in RP, assistant and creative writing use cases and at a quick glance seems to work well.
Training was done using [qlora-pipe](https://github.com/tdrussell/qlora-pipe)
exl2 RPCAL care of [Qaunt Cartel](https://huggingface.co/Quant-Cartel/Dendrite-L3-10B-exl2-rpcal)
GGUFs care of [Quant Cartel](https://huggingface.co/Quant-Cartel/Dendrite-L3-10B-iMat-GGUF) |