Hugging Face's logo

distily
/
distily_smollm_dataset_sweep

TensorBoard
Safetensors
Distily
llama
Generated from Trainer
Model card Files Metrics Community
lapp0 commited on
Commit
60891c9
•
1 Parent(s): 29dd07b

Training in progress, step 5000

Browse files
Files changed (11) hide show
  1. README.md +26 -26
  2. benchmarks.shelve.bak +1 -0
  3. benchmarks.shelve.dat +0 -0
  4. benchmarks.shelve.dir +1 -0
  5. logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=20231101.en, dataset_uri=wikimedia_wikipedia, per_device_train_batch_size=8/events.out.tfevents.1727190862.1c1a426a2fee +3 -0
  6. logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=None, dataset_uri=distily_filtered_redpajama_en, per_device_train_batch_size=8/events.out.tfevents.1727190862.1c1a426a2fee +3 -0
  7. logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=sample-10BT, dataset_uri=HuggingFaceFW_fineweb, per_device_train_batch_size=8/events.out.tfevents.1727218715.1c1a426a2fee +3 -0
  8. logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=sample-10BT, dataset_uri=HuggingFaceFW_fineweb-edu, per_device_train_batch_size=8/events.out.tfevents.1727190569.1c1a426a2fee +3 -0
  9. logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=sample-10BT, dataset_uri=HuggingFaceFW_fineweb-edu, per_device_train_batch_size=8/events.out.tfevents.1727190862.1c1a426a2fee +3 -0
  10. model.safetensors +1 -1
  11. training_args.bin +1 -1
README.md CHANGED
@@ -16,7 +16,7 @@ should probably proofread and complete it, then remove this comment. -->
16
 
17
  This model is a fine-tuned version of [HuggingFaceTB/SmolLM-135M](https://huggingface.co/HuggingFaceTB/SmolLM-135M) on an unknown dataset.
18
  It achieves the following results on the evaluation set:
19
- - Loss: 0.2261
20
 
21
  ## Model description
22
 
@@ -48,31 +48,31 @@ The following hyperparameters were used during training:
48
 
49
  | Training Loss | Epoch | Step | Validation Loss |
50
  |:-------------:|:------:|:------:|:---------------:|
51
- | No log | 0 | 0 | 19.2449 |
52
- | 1.1157 | 0.0401 | 5000 | 1.0851 |
53
- | 0.6609 | 0.0802 | 10000 | 0.6537 |
54
- | 0.5549 | 0.1202 | 15000 | 0.5350 |
55
- | 0.4606 | 0.1603 | 20000 | 0.4633 |
56
- | 0.465 | 0.2004 | 25000 | 0.4193 |
57
- | 0.3826 | 0.2405 | 30000 | 0.3880 |
58
- | 0.3807 | 0.2806 | 35000 | 0.3668 |
59
- | 0.3315 | 0.3206 | 40000 | 0.3475 |
60
- | 0.3352 | 0.3607 | 45000 | 0.3333 |
61
- | 0.3222 | 0.4008 | 50000 | 0.3209 |
62
- | 0.2994 | 0.4409 | 55000 | 0.3103 |
63
- | 0.3059 | 0.4810 | 60000 | 0.3016 |
64
- | 0.3154 | 0.5210 | 65000 | 0.2924 |
65
- | 0.2986 | 0.5611 | 70000 | 0.2855 |
66
- | 0.2696 | 0.6012 | 75000 | 0.2785 |
67
- | 0.254 | 0.6413 | 80000 | 0.2726 |
68
- | 0.2561 | 0.6814 | 85000 | 0.2662 |
69
- | 0.2602 | 0.7214 | 90000 | 0.2605 |
70
- | 0.2592 | 0.7615 | 95000 | 0.2558 |
71
- | 0.2417 | 0.8016 | 100000 | 0.2506 |
72
- | 0.23 | 0.8417 | 105000 | 0.2455 |
73
- | 0.2329 | 0.8818 | 110000 | 0.2406 |
74
- | 0.2302 | 0.9218 | 115000 | 0.2360 |
75
- | 0.2722 | 0.9619 | 120000 | 0.2312 |
76
 
77
 
78
  ### Framework versions
 
16
 
17
  This model is a fine-tuned version of [HuggingFaceTB/SmolLM-135M](https://huggingface.co/HuggingFaceTB/SmolLM-135M) on an unknown dataset.
18
  It achieves the following results on the evaluation set:
19
+ - Loss: 0.2647
20
 
21
  ## Model description
22
 
 
48
 
49
  | Training Loss | Epoch | Step | Validation Loss |
50
  |:-------------:|:------:|:------:|:---------------:|
51
+ | No log | 0 | 0 | 18.8388 |
52
+ | 1.2041 | 0.0401 | 5000 | 1.1584 |
53
+ | 0.7528 | 0.0802 | 10000 | 0.7396 |
54
+ | 0.5961 | 0.1202 | 15000 | 0.6070 |
55
+ | 0.5023 | 0.1603 | 20000 | 0.5307 |
56
+ | 0.4706 | 0.2004 | 25000 | 0.4836 |
57
+ | 0.4605 | 0.2405 | 30000 | 0.4512 |
58
+ | 0.417 | 0.2806 | 35000 | 0.4251 |
59
+ | 0.4027 | 0.3206 | 40000 | 0.4071 |
60
+ | 0.3693 | 0.3607 | 45000 | 0.3898 |
61
+ | 0.3745 | 0.4008 | 50000 | 0.3759 |
62
+ | 0.3652 | 0.4409 | 55000 | 0.3632 |
63
+ | 0.3537 | 0.4810 | 60000 | 0.3529 |
64
+ | 0.3665 | 0.5210 | 65000 | 0.3440 |
65
+ | 0.3177 | 0.5611 | 70000 | 0.3346 |
66
+ | 0.3102 | 0.6012 | 75000 | 0.3269 |
67
+ | 0.3023 | 0.6413 | 80000 | 0.3198 |
68
+ | 0.3076 | 0.6814 | 85000 | 0.3125 |
69
+ | 0.3388 | 0.7214 | 90000 | 0.3062 |
70
+ | 0.298 | 0.7615 | 95000 | 0.3003 |
71
+ | 0.3052 | 0.8016 | 100000 | 0.2941 |
72
+ | 0.2678 | 0.8417 | 105000 | 0.2880 |
73
+ | 0.2684 | 0.8818 | 110000 | 0.2824 |
74
+ | 0.274 | 0.9218 | 115000 | 0.2764 |
75
+ | 0.2647 | 0.9619 | 120000 | 0.2706 |
76
 
77
 
78
  ### Framework versions
benchmarks.shelve.bak CHANGED
@@ -1,3 +1,4 @@
1
  'logs/teacher', (0, 448)
2
  'distily_smollm_dataset_sweep/logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=20231101.en, dataset_uri=wikimedia_wikipedia, per_device_train_batch_size=8', (512, 448)
3
  'distily_smollm_dataset_sweep/logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=None, dataset_uri=distily_filtered_redpajama_en, per_device_train_batch_size=8', (1024, 448)
 
 
1
  'logs/teacher', (0, 448)
2
  'distily_smollm_dataset_sweep/logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=20231101.en, dataset_uri=wikimedia_wikipedia, per_device_train_batch_size=8', (512, 448)
3
  'distily_smollm_dataset_sweep/logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=None, dataset_uri=distily_filtered_redpajama_en, per_device_train_batch_size=8', (1024, 448)
4
+ 'distily_smollm_dataset_sweep/logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=sample-10BT, dataset_uri=HuggingFaceFW_fineweb-edu, per_device_train_batch_size=8', (1536, 448)
benchmarks.shelve.dat CHANGED
Binary files a/benchmarks.shelve.dat and b/benchmarks.shelve.dat differ
 
benchmarks.shelve.dir CHANGED
@@ -1,3 +1,4 @@
1
  'logs/teacher', (0, 448)
2
  'distily_smollm_dataset_sweep/logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=20231101.en, dataset_uri=wikimedia_wikipedia, per_device_train_batch_size=8', (512, 448)
3
  'distily_smollm_dataset_sweep/logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=None, dataset_uri=distily_filtered_redpajama_en, per_device_train_batch_size=8', (1024, 448)
 
 
1
  'logs/teacher', (0, 448)
2
  'distily_smollm_dataset_sweep/logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=20231101.en, dataset_uri=wikimedia_wikipedia, per_device_train_batch_size=8', (512, 448)
3
  'distily_smollm_dataset_sweep/logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=None, dataset_uri=distily_filtered_redpajama_en, per_device_train_batch_size=8', (1024, 448)
4
+ 'distily_smollm_dataset_sweep/logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=sample-10BT, dataset_uri=HuggingFaceFW_fineweb-edu, per_device_train_batch_size=8', (1536, 448)
logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=20231101.en, dataset_uri=wikimedia_wikipedia, per_device_train_batch_size=8/events.out.tfevents.1727190862.1c1a426a2fee ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:55af61ebf005c883bc35a1adfcf0b5420d28ec78dcbd64ed1b3f84f82d776b3a
3
+ size 562
logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=None, dataset_uri=distily_filtered_redpajama_en, per_device_train_batch_size=8/events.out.tfevents.1727190862.1c1a426a2fee ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b88152c1b05826bb414c6cba875f7cd5ce87cbbf58fb31bb85602e4b443efb64
3
+ size 562
logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=sample-10BT, dataset_uri=HuggingFaceFW_fineweb, per_device_train_batch_size=8/events.out.tfevents.1727218715.1c1a426a2fee ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fa405b8cfcc0702318bdc996a4b52939b23010f2346366b3b2e8c15455836d74
3
+ size 140459
logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=sample-10BT, dataset_uri=HuggingFaceFW_fineweb-edu, per_device_train_batch_size=8/events.out.tfevents.1727190569.1c1a426a2fee ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7cf0b774d4cafe0099c76452479df826174a6e069c25f28c6b8313cf77841e34
3
+ size 529
logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=sample-10BT, dataset_uri=HuggingFaceFW_fineweb-edu, per_device_train_batch_size=8/events.out.tfevents.1727190862.1c1a426a2fee ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7291319a20c87f7017bc74c0b6246e8212ad716d566034ba267c4324ff638cb6
3
+ size 562
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:bcb053b5939e5bd40cda4cb66fd2cb0c166e653392f58b828f3d5b7f0a8c6ed6
3
  size 325669528
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1328e15742ffa35f799789136277adab0d50e3c0925879c4cda26d354fdb233e
3
  size 325669528
training_args.bin CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:1c79528f67cf39ac6bcfd0016bcfceaf1909dad5dbbcdd1ff89d1efedcdb61a7
3
  size 5624
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:66494727ec03541c5c991424dd80273f96b4325bc02006dd5123394dcf73c030
3
  size 5624