End of training

README.md

@@ -1,7 +1,7 @@
 ---
 base_model: HuggingFaceTB/SmolLM-135M
 datasets:
-- HuggingFaceFW/fineweb-edu
+- wikimedia/wikipedia
 library_name: Distily
 license: creativeml-openrail-m
 tags:
@@ -18,7 +18,7 @@ model-index:
 
 Distilled with [Distily](https://github.com/lapp0/distily) library
 using teacher model [HuggingFaceTB/SmolLM-135M](https://huggingface.co/HuggingFaceTB/SmolLM-135M)
-on dataset [HuggingFaceFW/fineweb-edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu).
+on dataset [wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia).
 
 <!-- This model card has been generated automatically according to the information the Trainer had access to. You
 should probably proofread and complete it, then remove this comment.
@@ -81,20 +81,21 @@ LlamaForCausalLM(
 - student 3: `dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=sample-10BT, dataset_uri=HuggingFaceFW_fineweb, per_device_train_batch_size=8`
 - student 4: `dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=sample-10BT, dataset_uri=HuggingFaceFW_fineweb, learning_rate=6e-05, per_device_train_batch_size=8`
 - student 5: `dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=sample-10BT, dataset_uri=HuggingFaceFW_fineweb-edu, learning_rate=6e-05, per_device_train_batch_size=8`
-
-| Metric | teacher | student 0 | student 1 | student 2 | student 3 | student 4 | student 5 |
-| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |
-| tinyArc.acc_norm,none | 0.37 | 0.303 | 0.295 | 0.302 | 0.26 | 0.269 | **0.319** |
-| tinyGSM8k.exact_match,flexible-extract | 0.006 | 0.029 | **0.03** | 0.025 | 0.006 | 0.006 | 0.012 |
-| tinyGSM8k.exact_match,strict-match | 0.006 | **0.006** | **0.006** | **0.006** | **0.006** | **0.006** | **0.006** |
-| tinyHellaswag.acc_norm,none | 0.452 | **0.341** | 0.281 | 0.327 | 0.3 | 0.303 | 0.301 |
-| tinyMMLU.acc_norm,none | 0.341 | 0.276 | 0.281 | **0.31** | 0.286 | 0.279 | 0.292 |
-| tinyTruthfulQA.acc,none | 0.38 | **0.463** | 0.447 | 0.423 | 0.419 | 0.421 | 0.427 |
-| tinyWinogrande.acc_norm,none | 0.509 | 0.466 | 0.436 | 0.46 | **0.492** | 0.473 | 0.417 |
+- student 6: `dataset_max_seq_length=1024, dataset_sample_size=4000000, dataset_subset=20231101.en, dataset_uri=wikimedia_wikipedia, per_device_train_batch_size=8`
+
+| Metric | teacher | student 0 | student 1 | student 2 | student 3 | student 4 | student 5 | student 6 |
+| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |
+| tinyArc.acc_norm,none | 0.37 | 0.303 | 0.295 | 0.302 | 0.26 | 0.269 | **0.319** | 0.286 |
+| tinyGSM8k.exact_match,flexible-extract | 0.006 | 0.029 | **0.03** | 0.025 | 0.006 | 0.006 | 0.012 | 0.012 |
+| tinyGSM8k.exact_match,strict-match | 0.006 | **0.006** | **0.006** | **0.006** | **0.006** | **0.006** | **0.006** | **0.006** |
+| tinyHellaswag.acc_norm,none | 0.452 | 0.341 | 0.281 | 0.327 | 0.3 | 0.303 | 0.301 | **0.364** |
+| tinyMMLU.acc_norm,none | 0.341 | 0.276 | 0.281 | **0.31** | 0.286 | 0.279 | 0.292 | 0.295 |
+| tinyTruthfulQA.acc,none | 0.38 | **0.463** | 0.447 | 0.423 | 0.419 | 0.421 | 0.427 | 0.44 |
+| tinyWinogrande.acc_norm,none | 0.509 | 0.466 | 0.436 | 0.46 | **0.492** | 0.473 | 0.417 | 0.439 |
 
 # Resource Usage
 
-- Max Train VRAM Use: 13.1273 GB
+- Max Train VRAM Use: 13.1269 GB
 - Available VRAM: 23.4329 GB
 - GPUs: 
   - 1x NVIDIA GeForce RTX 4090
@@ -124,6 +125,28 @@ LlamaForCausalLM(
          (self_attn): LlamaSdpaAttention(
            (q_proj): Linear(in_features=576, out_features=576, bias=False)
            (k_proj): Linear(in_features=576, out_features=192, bias=False)
+@@ -10,17 +10,16 @@
+           (o_proj): Linear(in_features=576, out_features=576, bias=False)
+           (rotary_emb): LlamaRotaryEmbedding()
+         )
+-        (mlp): LlamaMLP(
++        (mlp): LigerSwiGLUMLP(
+           (gate_proj): Linear(in_features=576, out_features=1536, bias=False)
+           (up_proj): Linear(in_features=576, out_features=1536, bias=False)
+           (down_proj): Linear(in_features=1536, out_features=576, bias=False)
+-          (act_fn): SiLU()
+         )
+-        (input_layernorm): LlamaRMSNorm((576,), eps=1e-05)
+-        (post_attention_layernorm): LlamaRMSNorm((576,), eps=1e-05)
++        (input_layernorm): LigerRMSNorm((576,), eps=1e-05, offset=0.0)
++        (post_attention_layernorm): LigerRMSNorm((576,), eps=1e-05, offset=0.0)
+       )
+     )
+-    (norm): LlamaRMSNorm((576,), eps=1e-05)
++    (norm): LigerRMSNorm((576,), eps=1e-05, offset=0.0)
+     (rotary_emb): LlamaRotaryEmbedding()
+   )
+   (lm_head): Linear(in_features=576, out_features=49152, bias=False)
 
 ```
 
@@ -131,10 +154,10 @@ LlamaForCausalLM(
 <br/>
 
 # Train Dataset
-Trained on 640,425,804 tokens from the [HuggingFaceFW/fineweb-edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) dataset.
+Trained on 1,857,293,914 tokens from the [wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) dataset.
 
-- Num Samples: `998,000`
-- Subset: `sample-10BT`
+- Num Samples: `3,992,000`
+- Subset: `20231101.en`
 - Split: `train`
 
 
@@ -161,7 +184,7 @@ The following hyperparameters were used during training:
 <details>
 <summary>Expand</summary>
 
-- learning_rate: `6e-05`
+- learning_rate: `0.0001`
 - train_batch_size: `8`
 - eval_batch_size: `4`
 - seed: `42`
@@ -181,7 +204,7 @@ The following hyperparameters were used during training:
         weight=0
     )
 )`
-- lr_scheduler: `<torch.optim.lr_scheduler.LambdaLR object at 0x7d824cbaf4f0>`
+- lr_scheduler: `<torch.optim.lr_scheduler.LambdaLR object at 0x766de39d92d0>`
 - student_model_name_or_path: `None`
 - student_config_name_or_path: `None`
 - student_model_config: `{'num_hidden_layers': 15}`
@@ -192,11 +215,11 @@ The following hyperparameters were used during training:
 - teacher_model_name_or_path: `HuggingFaceTB/SmolLM-135M`
 - teacher_load_in_8bit: `False`
 - teacher_load_in_4bit: `False`
-- dataset_uri: `HuggingFaceFW/fineweb-edu`
-- dataset_subset: `sample-10BT`
+- dataset_uri: `wikimedia/wikipedia`
+- dataset_subset: `20231101.en`
 - dataset_split: `train`
 - dataset_column_name: `text`
-- dataset_sample_size: `1000000`
+- dataset_sample_size: `4000000`
 - dataset_max_seq_length: `1024`
 - dataset_test_size: `0.002`
 - dataset_shuffle: `False`

benchmarks.shelve.bak

@@ -5,3 +5,4 @@
 'distily_smollm_dataset_sweep/logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=sample-10BT, dataset_uri=HuggingFaceFW_fineweb, per_device_train_batch_size=8', (2048, 448)
 'distily_smollm_dataset_sweep/logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=sample-10BT, dataset_uri=HuggingFaceFW_fineweb, learning_rate=6e-05, per_device_train_batch_size=8', (2560, 448)
 'distily_smollm_dataset_sweep/logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=sample-10BT, dataset_uri=HuggingFaceFW_fineweb-edu, learning_rate=6e-05, per_device_train_batch_size=8', (3072, 448)
+'distily_smollm_dataset_sweep/logs/dataset_max_seq_length=1024, dataset_sample_size=4000000, dataset_subset=20231101.en, dataset_uri=wikimedia_wikipedia, per_device_train_batch_size=8', (3584, 448)

benchmarks.shelve.dir

@@ -5,3 +5,4 @@
 'distily_smollm_dataset_sweep/logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=sample-10BT, dataset_uri=HuggingFaceFW_fineweb, per_device_train_batch_size=8', (2048, 448)
 'distily_smollm_dataset_sweep/logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=sample-10BT, dataset_uri=HuggingFaceFW_fineweb, learning_rate=6e-05, per_device_train_batch_size=8', (2560, 448)
 'distily_smollm_dataset_sweep/logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=sample-10BT, dataset_uri=HuggingFaceFW_fineweb-edu, learning_rate=6e-05, per_device_train_batch_size=8', (3072, 448)
+'distily_smollm_dataset_sweep/logs/dataset_max_seq_length=1024, dataset_sample_size=4000000, dataset_subset=20231101.en, dataset_uri=wikimedia_wikipedia, per_device_train_batch_size=8', (3584, 448)

logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=20231101.en, dataset_uri=wikimedia_wikipedia, per_device_train_batch_size=8/events.out.tfevents.1727460185.1c1a426a2fee

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ecb25697ef7473b7b8b5667c93f6875e0df7d2d8d51da0be13197fd00326eb29
+size 562

logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=None, dataset_uri=distily_filtered_redpajama_en, per_device_train_batch_size=8/events.out.tfevents.1727460185.1c1a426a2fee

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c2fbbc999fdcdf1468846bc314f79a1c8adccdf843e07328638a3735b3a093cd
+size 562

logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=sample-10BT, dataset_uri=HuggingFaceFW_fineweb, learning_rate=6e-05, per_device_train_batch_size=8/events.out.tfevents.1727460185.1c1a426a2fee

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0ca1b09a384e70ad76ad81e675af99c09aaa0191fd62338bf082102f390bce91
+size 562

logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=sample-10BT, dataset_uri=HuggingFaceFW_fineweb, per_device_train_batch_size=8/events.out.tfevents.1727460185.1c1a426a2fee

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ad1ce823ddaf3c3e06a9a6b7c1dbf7ec6d4f98433cc0de7b677cddc18e3ad5e2
+size 562

logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=sample-10BT, dataset_uri=HuggingFaceFW_fineweb-edu, learning_rate=6e-05, per_device_train_batch_size=8/events.out.tfevents.1727460185.1c1a426a2fee

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8b8e4a1d6b2bc49849d6254e186ae8f0645b868c943c576edebfd67c9efdd853
+size 562

logs/dataset_max_seq_length=1024, dataset_sample_size=1000000, dataset_subset=sample-10BT, dataset_uri=HuggingFaceFW_fineweb-edu, per_device_train_batch_size=8/events.out.tfevents.1727460185.1c1a426a2fee

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6f626ce7e1138affbdf03f1c7c32a4f9935b13ee7d190d3c277b04761c26625b
+size 562

logs/dataset_max_seq_length=1024, dataset_sample_size=4000000, dataset_subset=20231101.en, dataset_uri=wikimedia_wikipedia, per_device_train_batch_size=8/events.out.tfevents.1727459787.1c1a426a2fee

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9db57ae95333d43e007d253ff414f8364bd7568da1de63554ee5e1dcbd09f338
+size 529

logs/dataset_max_seq_length=1024, dataset_sample_size=4000000, dataset_subset=20231101.en, dataset_uri=wikimedia_wikipedia, per_device_train_batch_size=8/events.out.tfevents.1727460185.1c1a426a2fee

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a065d3f14191df20143698661ca8115aecd472b34caf6ef7848f74d8000a707f
+size 562

tokenizer.json

@@ -1,19 +1,7 @@
 {
   "version": "1.0",
-  "truncation": {
-    "direction": "Right",
-    "max_length": 1023,
-    "strategy": "LongestFirst",
-    "stride": 0
-  },
-  "padding": {
-    "strategy": "BatchLongest",
-    "direction": "Right",
-    "pad_to_multiple_of": null,
-    "pad_id": 0,
-    "pad_type_id": 0,
-    "pad_token": "<|endoftext|>"
-  },
+  "truncation": null,
+  "padding": null,
   "added_tokens": [
     {
       "id": 0,