End of training

README.md

@@ -16,13 +16,13 @@ This student model is distilled from the teacher model [gpt2](https://huggingfac
 The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
 
 It achieves the following results on the evaluation set:
-- eval_enwikippl: 322.4071
-- eval_frwikippl: 2411.2278
-- eval_zhwikippl: 1802.7419
-- eval_loss: 1569.2800
-- eval_runtime: 21.3907
-- eval_samples_per_second: 46.749
-- eval_steps_per_second: 11.687
+- eval_enwikippl: 400.4869
+- eval_frwikippl: 2969.3420
+- eval_zhwikippl: 6008.7476
+- eval_loss: 1989.5520
+- eval_runtime: 21.3179
+- eval_samples_per_second: 46.909
+- eval_steps_per_second: 11.727
 
 <!-- This model card has been generated automatically according to the information the Trainer had access to. You
 should probably proofread and complete it, then remove this comment.
@@ -45,7 +45,7 @@ More information needed
 ### Training hyperparameters
 
 The following hyperparameters were used during training:
-- distillation_objective: LinearObjective(logits_weight=100, logits_loss_fn=<function kl_divergence_loss at 0x7f57c4b07910>, activations_weight=0, activations_loss_fn=<function mse_loss at 0x7f57c4b07880>, attentions_weight=0, attentions_loss_fn=<function mse_loss at 0x7f57c4b07880>)
+- distillation_objective: LinearObjective(logits_weight=1, logits_loss_fn=<function kl_divergence_loss at 0x7f57c4b07910>, activations_weight=1, activations_loss_fn=<function mse_loss at 0x7f57c4b07880>, attentions_weight=0, attentions_loss_fn=<function mse_loss at 0x7f57c4b07880>)
 - train_embeddings: True
 - learning_rate: 4e-05
 - train_batch_size: 4
@@ -58,26 +58,26 @@ The following hyperparameters were used during training:
 - num_epochs: 1.0
 
 ### Resource Usage
-Peak GPU Memory: 4.4338 GB
+Peak GPU Memory: 4.5067 GB
 
 ### Eval-Phase Metrics
 | step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | zhwikippl |
 | --- | --- | --- | --- | --- | --- | --- | --- | --- |
 | **teacher eval** |  | 30.2385 | 57.2728 |  |  |  |  | 18.1772 |
-| 0 | 0 | 55339.3672 | 57682.5742 | 6157.4399 | 21.587 | 46.324 | 11.581 | 57080.2930 |
-| 500 | 0.0808 | 1033.0840 | 6283.9331 | 2279.6641 | 21.6395 | 46.212 | 11.553 | 52104.6055 |
-| 1000 | 0.1616 | 765.2576 | 4892.3081 | 2097.8081 | 21.3235 | 46.897 | 11.724 | 26096.3535 |
-| 1500 | 0.2424 | 644.3925 | 4229.7388 | 2007.1281 | 21.672 | 46.142 | 11.536 | 19367.6953 |
-| 2000 | 0.3232 | 572.4389 | 3848.9590 | 1927.5200 | 21.3193 | 46.906 | 11.726 | 13467.1152 |
-| 2500 | 0.4040 | 522.0941 | 3661.2842 | 1866.1200 | 21.356 | 46.825 | 11.706 | 8405.3037 |
-| 3000 | 0.4848 | 479.3863 | 3402.9036 | 1813.2240 | 21.551 | 46.402 | 11.6 | 6118.0571 |
-| 3500 | 0.5657 | 438.9093 | 3085.1194 | 1765.5601 | 21.4458 | 46.629 | 11.657 | 5073.1313 |
-| 4000 | 0.6465 | 414.7451 | 2969.3420 | 1719.8400 | 21.3815 | 46.769 | 11.692 | 4076.1655 |
-| 4500 | 0.7273 | 384.2867 | 2787.7578 | 1679.6400 | 21.4967 | 46.519 | 11.63 | 3883.0295 |
-| 5000 | 0.8081 | 357.7559 | 2647.6235 | 1643.3920 | 21.5199 | 46.469 | 11.617 | 2635.5439 |
-| 5500 | 0.8889 | 344.0447 | 2642.6804 | 1612.0720 | 21.5612 | 46.38 | 11.595 | 1745.4163 |
-| 6000 | 0.9697 | 322.2757 | 2468.9441 | 1580.7920 | 21.3421 | 46.856 | 11.714 | 1794.2168 |
-| 6187 | 0.9999 | 322.4071 | 2411.2278 | 1569.2800 | 21.3907 | 46.749 | 11.687 | 1802.7419 |
+| 0 | 0 | 55339.3672 | 57682.5742 | 6775.2959 | 21.2939 | 46.962 | 11.74 | 57080.2930 |
+| 500 | 0.0808 | 1019.9713 | 6165.6597 | 2829.9199 | 21.7268 | 46.026 | 11.507 | 49348.3086 |
+| 1000 | 0.1616 | 776.1801 | 5053.2275 | 2518.8479 | 21.7049 | 46.072 | 11.518 | 43976.7227 |
+| 1500 | 0.2424 | 668.2543 | 4423.4121 | 2381.3120 | 21.5969 | 46.303 | 11.576 | 45161.0352 |
+| 2000 | 0.3232 | 617.0906 | 4036.9917 | 2304.1279 | 21.5749 | 46.35 | 11.588 | 40062.9219 |
+| 2500 | 0.4040 | 572.6278 | 4018.1096 | 2242.3999 | 21.5368 | 46.432 | 11.608 | 24775.3672 |
+| 3000 | 0.4848 | 534.8770 | 3649.6860 | 2192.1919 | 21.7099 | 46.062 | 11.515 | 135085.7969 |
+| 3500 | 0.5657 | 504.0086 | 3549.8198 | 2150.6721 | 21.6613 | 46.165 | 11.541 | 46912.9102 |
+| 4000 | 0.6465 | 483.1796 | 3357.9766 | 2115.4399 | 21.688 | 46.108 | 11.527 | 24034.0039 |
+| 4500 | 0.7273 | 458.0607 | 3258.0415 | 2080.0 | 21.5498 | 46.404 | 11.601 | 44794.6328 |
+| 5000 | 0.8081 | 432.8254 | 3191.2034 | 2050.5439 | 21.3113 | 46.923 | 11.731 | 9512.9375 |
+| 5500 | 0.8889 | 422.7440 | 3201.5676 | 2027.1520 | 21.5677 | 46.366 | 11.591 | 14696.1348 |
+| 6000 | 0.9697 | 406.4080 | 3030.5735 | 2000.3840 | 21.1806 | 47.213 | 11.803 | 3418.1089 |
+| 6187 | 0.9999 | 400.4869 | 2969.3420 | 1989.5520 | 21.3179 | 46.909 | 11.727 | 6008.7476 |
 
 ### Framework versions
 - Distily 0.2.0

logs/distillation_objective=LinearObjective_logits_weight_1__logits_loss_fn__function_kl_divergence_loss_at_0x7f57c4b07910___activations_weight_1__activations_loss_fn__function_mse_loss_at_0x7f57c4b07880__/events.out.tfevents.1723362266.715f24f8d8b8

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7a980bf26bfbda5b2d34c886003cc4829b6bd26a7549f4097c1b06433dfeade4
+size 249