End of training

README.md

@@ -1,6 +1,7 @@
 ---
-license: mit
 base_model: gpt2
+library_name: distily
+license: mit
 tags:
 - generated_from_trainer
 model-index:
@@ -8,14 +9,23 @@ model-index:
   results: []
 ---
 
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
+# distily_bench_gpt2_optim
+
+This student model is distilled from the teacher model [gpt2](https://huggingface.co/gpt2) using the dataset (unspecified).
 
-# distily_bench_gpt2_linear_objectives
+The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
 
-This model is a fine-tuned version of [gpt2](https://huggingface.co/gpt2) on an unknown dataset.
 It achieves the following results on the evaluation set:
-- Loss: 8689.6641
+- eval_enwikippl: 332.3850
+- eval_frwikippl: 2330.5789
+- eval_zhwikippl: 2631.6760
+- eval_loss: 1627.4800
+- eval_runtime: 21.3386
+- eval_samples_per_second: 46.863
+- eval_steps_per_second: 11.716
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment.
 
 ## Model description
 
@@ -28,12 +38,15 @@ More information needed
 ## Training and evaluation data
 
 More information needed
+-->
 
 ## Training procedure
 
 ### Training hyperparameters
 
 The following hyperparameters were used during training:
+- distillation_objective: LinearObjective(logits_weight=1, logits_loss_fn=<function kl_divergence_loss at 0x7f57c4b07910>, activations_weight=0, activations_loss_fn=<function mse_loss at 0x7f57c4b07880>, attentions_weight=0, attentions_loss_fn=<function mse_loss at 0x7f57c4b07880>)
+- train_embeddings: True
 - learning_rate: 4e-05
 - train_batch_size: 4
 - eval_batch_size: 4
@@ -44,28 +57,30 @@ The following hyperparameters were used during training:
 - lr_scheduler_type: constant
 - num_epochs: 1.0
 
-### Training results
-
-| Training Loss | Epoch  | Step | Validation Loss |
-|:-------------:|:------:|:----:|:---------------:|
-| No log        | 0      | 0    | 338272.25       |
-| 14496.0       | 0.0808 | 500  | 11190.9121      |
-| 12080.0       | 0.1616 | 1000 | 10443.1357      |
-| 15856.0       | 0.2424 | 1500 | 10039.6162      |
-| 12192.0       | 0.3232 | 2000 | 9736.1924       |
-| 12784.0       | 0.4040 | 2500 | 9512.1279       |
-| 13216.0       | 0.4848 | 3000 | 9331.4561       |
-| 10856.0       | 0.5657 | 3500 | 9238.5283       |
-| 12256.0       | 0.6465 | 4000 | 9076.0957       |
-| 9856.0        | 0.7273 | 4500 | 8944.4482       |
-| 12512.0       | 0.8081 | 5000 | 8849.7275       |
-| 10224.0       | 0.8889 | 5500 | 8798.5918       |
-| 10688.0       | 0.9697 | 6000 | 8741.2480       |
-
+### Resource Usage
+Peak GPU Memory: 4.4311 GB
+
+### Eval-Phase Metrics
+| step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | zhwikippl |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| **teacher eval** |  | 30.2385 | 57.2728 |  |  |  |  | 18.1772 |
+| 0 | 0 | 56520.6797 | 56906.9609 | 6220.2881 | 20.9383 | 47.759 | 11.94 | 55769.3047 |
+| 500 | 0.0808 | 1258.1868 | 6532.3999 | 3064.5439 | 20.9174 | 47.807 | 11.952 | 374879.2188 |
+| 1000 | 0.1616 | 787.1970 | 4834.1909 | 2175.0400 | 20.938 | 47.76 | 11.94 | 43497.8008 |
+| 1500 | 0.2424 | 665.8845 | 4272.1475 | 2074.2080 | 21.349 | 46.841 | 11.71 | 27341.3633 |
+| 2000 | 0.3232 | 599.6594 | 4003.5479 | 1993.9120 | 21.3465 | 46.846 | 11.711 | 14360.5186 |
+| 2500 | 0.4040 | 540.3354 | 3653.2900 | 1930.3280 | 22.4488 | 44.546 | 11.136 | 10866.1885 |
+| 3000 | 0.4848 | 494.5775 | 3489.5195 | 1875.5520 | 21.8809 | 45.702 | 11.426 | 6831.9717 |
+| 3500 | 0.5657 | 458.0697 | 3287.1162 | 1827.1600 | 21.4169 | 46.692 | 11.673 | 5614.6519 |
+| 4000 | 0.6465 | 423.6559 | 3147.5164 | 1782.9680 | 21.794 | 45.884 | 11.471 | 4302.3994 |
+| 4500 | 0.7273 | 404.1501 | 2901.5630 | 1743.7920 | 21.8359 | 45.796 | 11.449 | 4091.1592 |
+| 5000 | 0.8081 | 376.5035 | 2821.1733 | 1703.9600 | 21.562 | 46.378 | 11.594 | 3270.3157 |
+| 5500 | 0.8889 | 353.2973 | 2519.7690 | 1668.6639 | 21.4165 | 46.693 | 11.673 | 2576.3877 |
+| 6000 | 0.9697 | 331.4378 | 2333.6213 | 1636.2000 | 21.3513 | 46.836 | 11.709 | 2431.3169 |
+| 6187 | 0.9999 | 332.3850 | 2330.5789 | 1627.4800 | 21.3386 | 46.863 | 11.716 | 2631.6760 |
 
 ### Framework versions
-
+- Distily 0.2.0
 - Transformers 4.44.0
 - Pytorch 2.3.0
 - Datasets 2.20.0
-- Tokenizers 0.19.1

logs/distillation_objective=LinearObjective_logits_weight_1__logits_loss_fn__function_kl_divergence_loss_at_0x7f57c4b07910___activations_weight_0__activations_loss_fn__function_mse_loss_at_0x7f57c4b07880__/events.out.tfevents.1723349376.715f24f8d8b8

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f57b8bfe6f209c3d45b23b984694ce0608000b7bcb8840ed40524a66fed6101b
+size 249