End of training

README.md

@@ -16,14 +16,14 @@ This student model is distilled from the teacher model [gpt2](https://huggingfac
 The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
 
 It achieves the following results on the evaluation set:
-- eval_enwikippl: 1248.0
-- eval_frwikippl: 6560.0
-- eval_zhwikippl: 24448.0
-- eval_tinystoriesppl: 968.0
-- eval_loss: 2.4413
-- eval_runtime: 12.6655
-- eval_samples_per_second: 47.373
-- eval_steps_per_second: 11.843
+- eval_enwikippl: 2160.0
+- eval_frwikippl: 9536.0
+- eval_zhwikippl: 98816.0
+- eval_tinystoriesppl: 1960.0
+- eval_loss: 3.2783
+- eval_runtime: 12.9016
+- eval_samples_per_second: 46.506
+- eval_steps_per_second: 11.626
 
 <!-- This model card has been generated automatically according to the information the Trainer had access to. You
 should probably proofread and complete it, then remove this comment.
@@ -46,7 +46,7 @@ More information needed
 ### Training hyperparameters
 
 The following hyperparameters were used during training:
-- distillation_objective: DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl, layer_mapper=None, projector=None), hs_loss_component=LossComponent(label=hs, weight=0, loss_fn=None, layer_mapper=None, projector=None), attn_loss_component=LossComponent(label=attn, weight=0, loss_fn=None, layer_mapper=None, projector=None))
+- distillation_objective: DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl, layer_mapper=None, projector=None), hs_loss_component=LossComponent(label=hs, weight=1.0, loss_fn=kl, layer_mapper=all, projector=None), attn_loss_component=LossComponent(label=attn, weight=0, loss_fn=None, layer_mapper=None, projector=None))
 - train_embeddings: True
 - learning_rate: 0.0001
 - train_batch_size: 8
@@ -58,23 +58,23 @@ The following hyperparameters were used during training:
 - num_epochs: 1.0
 
 ### Resource Usage
-Peak GPU Memory: 7.9388 GB
+Peak GPU Memory: 8.0905 GB
 
 ### Eval-Phase Metrics
 | step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | tinystoriesppl | zhwikippl |
 | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
 | **teacher eval** |  | 43.75 | 61.75 |  |  |  |  | 11.8125 | 19.125 |
-| 0 | 0 | 1065151889408.0 | 117097988358144.0 | 20.4033 | 12.6433 | 47.456 | 11.864 | 4362076160.0 | 27762668601344.0 |
-| 750 | 0.1010 | 1248.0 | 6560.0 | 2.4413 | 12.6655 | 47.373 | 11.843 | 968.0 | 24448.0 |
-| 1500 | 0.2020 | 500.0 | 3712.0 | 1.8019 | 12.6495 | 47.433 | 11.858 | 352.0 | 820.0 |
-| 2250 | 0.3030 | 342.0 | 1832.0 | 1.5659 | 12.681 | 47.315 | 11.829 | 276.0 | 294.0 |
-| 3000 | 0.4040 | 245.0 | 932.0 | 1.3644 | 12.6925 | 47.272 | 11.818 | 198.0 | 228.0 |
-| 3750 | 0.5051 | 191.0 | 680.0 | 1.2038 | 12.6837 | 47.305 | 11.826 | 159.0 | 219.0 |
-| 4500 | 0.6061 | 148.0 | 596.0 | 1.0541 | 12.6874 | 47.291 | 11.823 | 127.5 | 188.0 |
-| 5250 | 0.7071 | 129.0 | 442.0 | 0.9331 | 12.6692 | 47.359 | 11.84 | 102.5 | 126.5 |
-| 6000 | 0.8081 | 117.0 | 412.0 | 0.8757 | 12.6912 | 47.277 | 11.819 | 94.5 | 124.5 |
-| 6750 | 0.9091 | 112.0 | 398.0 | 0.8483 | 12.8574 | 46.666 | 11.666 | 90.0 | 119.5 |
-| 7425 | 1.0 | 110.5 | 392.0 | 0.8433 | 12.717 | 47.181 | 11.795 | 89.0 | 119.0 |
+| 0 | 0 | 1855425871872.0 | 61297773248512.0 | 26.3692 | 12.8695 | 46.622 | 11.655 | 14495514624.0 | 11338713661440.0 |
+| 750 | 0.1010 | 2160.0 | 9536.0 | 3.2783 | 12.9016 | 46.506 | 11.626 | 1960.0 | 98816.0 |
+| 1500 | 0.2020 | 740.0 | 4640.0 | 2.2764 | 12.9025 | 46.503 | 11.626 | 612.0 | 9728.0 |
+| 2250 | 0.3030 | 450.0 | 2656.0 | 1.9411 | 12.9079 | 46.483 | 11.621 | 344.0 | 564.0 |
+| 3000 | 0.4040 | 312.0 | 1544.0 | 1.6865 | 12.9248 | 46.422 | 11.606 | 272.0 | 304.0 |
+| 3750 | 0.5051 | 237.0 | 964.0 | 1.4813 | 12.9469 | 46.343 | 11.586 | 199.0 | 252.0 |
+| 4500 | 0.6061 | 185.0 | 712.0 | 1.2780 | 13.0095 | 46.12 | 11.53 | 144.0 | 254.0 |
+| 5250 | 0.7071 | 143.0 | 520.0 | 1.1058 | 13.0283 | 46.054 | 11.513 | 114.5 | 212.0 |
+| 6000 | 0.8081 | 130.0 | 478.0 | 1.0297 | 12.9977 | 46.162 | 11.541 | 103.0 | 180.0 |
+| 6750 | 0.9091 | 123.5 | 442.0 | 0.9899 | 12.9475 | 46.341 | 11.585 | 99.0 | 160.0 |
+| 7425 | 1.0 | 122.5 | 438.0 | 0.9819 | 12.9461 | 46.346 | 11.587 | 97.5 | 160.0 |
 
 ### Framework versions
 - Distily 0.2.0

logs/hs_layer_mapper=all, hs_loss_fn=kl, hs_weight=1.0/events.out.tfevents.1724092951.f383272e719b

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5751aee29f11ef263fb2e003455bb13e2c840db1437c1257448b4b4eff93b788
+size 578