jeiku commited on
Commit
b37554c
·
verified ·
1 Parent(s): 4944c6b

End of training

Browse files
README.md ADDED
@@ -0,0 +1,152 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: transformers
3
+ license: llama3.1
4
+ base_model: huihui-ai/Llama-3.1-Tulu-3-8B-abliterated
5
+ tags:
6
+ - axolotl
7
+ - generated_from_trainer
8
+ datasets:
9
+ - FourOhFour/RP_Phase
10
+ model-index:
11
+ - name: evil8b
12
+ results: []
13
+ ---
14
+
15
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
16
+ should probably proofread and complete it, then remove this comment. -->
17
+
18
+ [<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/axolotl-ai-cloud/axolotl)
19
+ <details><summary>See axolotl config</summary>
20
+
21
+ axolotl version: `0.5.2`
22
+ ```yaml
23
+ base_model: huihui-ai/Llama-3.1-Tulu-3-8B-abliterated
24
+ model_type: AutoModelForCausalLM
25
+ tokenizer_type: AutoTokenizer
26
+
27
+ load_in_8bit: false
28
+ load_in_4bit: false
29
+ strict: false
30
+
31
+ datasets:
32
+ - path: FourOhFour/RP_Phase
33
+ type: chat_template
34
+ chat_template: llama3
35
+ roles_to_train: ["gpt"]
36
+ field_messages: conversations
37
+ message_field_role: from
38
+ message_field_content: value
39
+ train_on_eos: turn
40
+
41
+ shuffle_merged_datasets: true
42
+ default_system_message:
43
+ dataset_prepared_path:
44
+ val_set_size: 0.0125
45
+ output_dir: ./output/out
46
+
47
+ hub_model_id: jeiku/evil8b
48
+ hub_strategy: "all_checkpoints"
49
+ push_dataset_to_hub:
50
+ hf_use_auth_token: true
51
+
52
+ sequence_len: 8192
53
+ sample_packing: true
54
+ eval_sample_packing: false
55
+ pad_to_sequence_len:
56
+
57
+ wandb_project: evil
58
+ wandb_entity:
59
+ wandb_watch:
60
+ wandb_name: evil
61
+ wandb_log_model:
62
+
63
+ gradient_accumulation_steps: 16
64
+ micro_batch_size: 2
65
+ num_epochs: 2
66
+ optimizer: paged_adamw_8bit
67
+ lr_scheduler: cosine
68
+
69
+ learning_rate: 1e-5
70
+
71
+ train_on_inputs: false
72
+ group_by_length: false
73
+ bf16: auto
74
+ fp16:
75
+ tf32: false
76
+
77
+ gradient_checkpointing: true
78
+ gradient_checkpointing_kwargs:
79
+ use_reentrant: false
80
+ early_stopping_patience:
81
+ resume_from_checkpoint:
82
+ local_rank:
83
+ logging_steps: 1
84
+ xformers_attention:
85
+ flash_attention: true
86
+
87
+ warmup_steps: 10
88
+ evals_per_epoch: 2
89
+ eval_table_size:
90
+ eval_max_new_tokens:
91
+ saves_per_epoch: 1
92
+ debug:
93
+ deepspeed:
94
+ weight_decay: 0.05
95
+ fsdp:
96
+ fsdp_config:
97
+ special_tokens:
98
+ pad_token: <|finetune_right_pad_id|>
99
+ eos_token: <|eot_id|>
100
+ ```
101
+
102
+ </details><br>
103
+
104
+ # evil8b
105
+
106
+ This model is a fine-tuned version of [huihui-ai/Llama-3.1-Tulu-3-8B-abliterated](https://huggingface.co/huihui-ai/Llama-3.1-Tulu-3-8B-abliterated) on the FourOhFour/RP_Phase dataset.
107
+ It achieves the following results on the evaluation set:
108
+ - Loss: 1.0089
109
+
110
+ ## Model description
111
+
112
+ More information needed
113
+
114
+ ## Intended uses & limitations
115
+
116
+ More information needed
117
+
118
+ ## Training and evaluation data
119
+
120
+ More information needed
121
+
122
+ ## Training procedure
123
+
124
+ ### Training hyperparameters
125
+
126
+ The following hyperparameters were used during training:
127
+ - learning_rate: 1e-05
128
+ - train_batch_size: 2
129
+ - eval_batch_size: 2
130
+ - seed: 42
131
+ - gradient_accumulation_steps: 16
132
+ - total_train_batch_size: 32
133
+ - optimizer: Use OptimizerNames.PAGED_ADAMW_8BIT with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
134
+ - lr_scheduler_type: cosine
135
+ - lr_scheduler_warmup_steps: 10
136
+ - num_epochs: 2
137
+
138
+ ### Training results
139
+
140
+ | Training Loss | Epoch | Step | Validation Loss |
141
+ |:-------------:|:------:|:----:|:---------------:|
142
+ | 1.5229 | 0.5004 | 131 | 1.0768 |
143
+ | 2.103 | 1.0012 | 262 | 1.0223 |
144
+ | 1.3982 | 1.5016 | 393 | 1.0089 |
145
+
146
+
147
+ ### Framework versions
148
+
149
+ - Transformers 4.46.3
150
+ - Pytorch 2.3.1+cu121
151
+ - Datasets 3.1.0
152
+ - Tokenizers 0.20.3
pytorch_model-00001-of-00004.bin CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:644ea489ca6163d8448a661b33459ee6f0563a94ba15ebbeb4a77b96e39198ab
3
  size 4976784002
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:64f59697a895e355c0e9ff6692132f787a943f28704b8b86e592a97195b75d69
3
  size 4976784002
pytorch_model-00002-of-00004.bin CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:b0061a8c8d15923f281fd977bbf3a639848a29a3e598f2d7c5c19d8469f1478f
3
  size 4999827718
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d17d6c8303c34fef4cb00c5d6a9291edf6ef861550ac7d5e8373855634edb94e
3
  size 4999827718
pytorch_model-00003-of-00004.bin CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:3e6404d4a04ecf1337b70a4eab63851a7a441c01c6a84caca2bca2ebc20178cc
3
  size 4915940170
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8920bba085f75249a859dd11986dc4cc0f018eebad4f9fe547acf5c34ccb3ad0
3
  size 4915940170
pytorch_model-00004-of-00004.bin CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:b452396b3d23a4f0aa733bbbb6bf864e149a280850b6cff87c9b7b134e98353a
3
  size 1168206409
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:59aca6309be7a2c2e2890513b4c396d7a8a11b17a8f885a31ede5ef5e5d78179
3
  size 1168206409