Hugging Face's logo

Krisbiantoro
/
mixtral_full

PEFT
TensorBoard
Safetensors
trl
sft
Generated from Trainer
Model card Files Metrics Community
Krisbiantoro commited on
Commit
4b48457
•
1 Parent(s): 56442c6

Model save

Browse files
Files changed (3) hide show
  1. README.md +86 -0
  2. adapter_model.safetensors +1 -1
  3. runs/Apr04_03-39-24_llm-a100-40/events.out.tfevents.1712201995.llm-a100-40.16389.0 +2 -2
README.md ADDED
@@ -0,0 +1,86 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ library_name: peft
4
+ tags:
5
+ - trl
6
+ - sft
7
+ - generated_from_trainer
8
+ datasets:
9
+ - generator
10
+ base_model: mistralai/Mixtral-8x7B-v0.1
11
+ model-index:
12
+ - name: mixtral_full
13
+ results: []
14
+ ---
15
+
16
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
17
+ should probably proofread and complete it, then remove this comment. -->
18
+
19
+ # mixtral_full
20
+
21
+ This model is a fine-tuned version of [mistralai/Mixtral-8x7B-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-v0.1) on the generator dataset.
22
+ It achieves the following results on the evaluation set:
23
+ - Loss: 0.9766
24
+
25
+ ## Model description
26
+
27
+ More information needed
28
+
29
+ ## Intended uses & limitations
30
+
31
+ More information needed
32
+
33
+ ## Training and evaluation data
34
+
35
+ More information needed
36
+
37
+ ## Training procedure
38
+
39
+ ### Training hyperparameters
40
+
41
+ The following hyperparameters were used during training:
42
+ - learning_rate: 3e-05
43
+ - train_batch_size: 2
44
+ - eval_batch_size: 1
45
+ - seed: 42
46
+ - gradient_accumulation_steps: 32
47
+ - total_train_batch_size: 64
48
+ - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
49
+ - lr_scheduler_type: cosine
50
+ - lr_scheduler_warmup_ratio: 0.03
51
+ - num_epochs: 1.5
52
+
53
+ ### Training results
54
+
55
+ | Training Loss | Epoch | Step | Validation Loss |
56
+ |:-------------:|:-----:|:----:|:---------------:|
57
+ | 1.3844 | 0.07 | 20 | 1.2695 |
58
+ | 1.1368 | 0.14 | 40 | 1.0996 |
59
+ | 1.0758 | 0.21 | 60 | 1.0459 |
60
+ | 1.0537 | 0.28 | 80 | 1.0269 |
61
+ | 1.0397 | 0.35 | 100 | 1.0147 |
62
+ | 1.0075 | 0.43 | 120 | 1.0059 |
63
+ | 1.0145 | 0.5 | 140 | 0.9990 |
64
+ | 0.9939 | 0.57 | 160 | 0.9937 |
65
+ | 1.0228 | 0.64 | 180 | 0.9895 |
66
+ | 1.0056 | 0.71 | 200 | 0.9858 |
67
+ | 0.999 | 0.78 | 220 | 0.9831 |
68
+ | 1.0084 | 0.85 | 240 | 0.9809 |
69
+ | 0.9957 | 0.92 | 260 | 0.9792 |
70
+ | 1.0033 | 0.99 | 280 | 0.9781 |
71
+ | 0.9884 | 1.06 | 300 | 0.9774 |
72
+ | 0.9906 | 1.13 | 320 | 0.9770 |
73
+ | 0.9893 | 1.2 | 340 | 0.9768 |
74
+ | 1.0005 | 1.28 | 360 | 0.9766 |
75
+ | 0.9824 | 1.35 | 380 | 0.9767 |
76
+ | 0.9886 | 1.42 | 400 | 0.9766 |
77
+ | 0.9828 | 1.49 | 420 | 0.9766 |
78
+
79
+
80
+ ### Framework versions
81
+
82
+ - PEFT 0.7.2.dev0
83
+ - Transformers 4.38.1
84
+ - Pytorch 2.1.2+cu121
85
+ - Datasets 2.16.1
86
+ - Tokenizers 0.15.0
adapter_model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:6877c9ecdbc1a1e1e541f5badccd9828a725476307cfe928bb4b9253559f8c09
3
  size 969176736
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c487c591d0c935613751e84d41bb012ed1562c476c54958bf2de82d0b510a1fc
3
  size 969176736
runs/Apr04_03-39-24_llm-a100-40/events.out.tfevents.1712201995.llm-a100-40.16389.0 CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:a51fe55414ad7fdfe70cb8828fef3f3a8e85f4a2b423359bf09dcc196e5021ef
3
- size 18898
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:245e443ab6b245247b5d8069c5531227bb3b1c313ecb14f4356cb0f446a6553d
3
+ size 19945