metadata
base_model: gpt2
library_name: Distily
license: mit
tags:
- generated_from_trainer
model-index:
- name: distily_bench_gpt2_activation_loss_c
results: []
distily_bench_gpt2_activation_loss_c
This student model is distilled from the teacher model gpt2 using the dataset (unspecified).
The Distily library was used for this distillation.
It achieves the following results on the evaluation set:
- eval_enwikippl: 215.5059
- eval_frwikippl: 1193.6056
- eval_zhwikippl: 627.1483
- eval_loss: 1.2009
- eval_runtime: 85.3591
- eval_samples_per_second: 58.576
- eval_steps_per_second: 7.322
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- distillation_objective: DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl, layer_mapper=None, projector=None), hs_loss_component=LossComponent(label=hs, weight=2.0, loss_fn=mse, layer_mapper=None, projector=None), attn_loss_component=LossComponent(label=attn, weight=0, loss_fn=None, layer_mapper=None, projector=None))
- train_embeddings: True
- learning_rate: 4e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: constant
- num_epochs: 1.0
Resource Usage
Peak GPU Memory: 8.0873 GB
Eval-Phase Metrics
| step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | zhwikippl |
|---|---|---|---|---|---|---|---|---|
| teacher eval | 30.2086 | 57.2728 | 18.1784 | |||||
| 0 | 0 | 56314.7695 | 59887.2773 | 5.8256 | 85.6711 | 58.363 | 7.295 | 59033.8086 |
| 1000 | 0.0162 | 703.3142 | 4236.9004 | 1.8490 | 85.6638 | 58.368 | 7.296 | 11133.5088 |
| 2000 | 0.0323 | 504.8312 | 3192.5520 | 1.6764 | 85.4461 | 58.516 | 7.315 | 1842.1659 |
| 3000 | 0.0485 | 421.6048 | 2827.9453 | 1.5711 | 85.6065 | 58.407 | 7.301 | 841.4271 |
| 4000 | 0.0646 | 359.8385 | 2300.7822 | 1.4898 | 85.5248 | 58.463 | 7.308 | 1321.4115 |
| 5000 | 0.0808 | 320.1989 | 1782.2493 | 1.4134 | 85.6041 | 58.408 | 7.301 | 921.9020 |
| 6000 | 0.0970 | 279.3613 | 1572.2640 | 1.3457 | 85.4507 | 58.513 | 7.314 | 775.6033 |
| 7000 | 0.1131 | 252.3406 | 1452.0632 | 1.2901 | 85.4137 | 58.539 | 7.317 | 675.1237 |
| 8000 | 0.1293 | 230.5502 | 1345.9784 | 1.2423 | 85.9632 | 58.164 | 7.271 | 594.2899 |
| 9000 | 0.1455 | 215.5059 | 1193.6056 | 1.2009 | 85.3591 | 58.576 | 7.322 | 627.1483 |
| 10000 | 0.1616 | 194.5708 | 1147.0729 | 1.1501 | 85.2878 | 58.625 | 7.328 | 681.0092 |
| 11000 | 0.1778 | 179.9636 | 1066.1221 | 1.1062 | 85.2181 | 58.673 | 7.334 | 556.0541 |
| 12000 | 0.1939 | 165.4222 | 900.6642 | 1.0627 | 85.2275 | 58.667 | 7.333 | 517.4376 |
| 13000 | 0.2101 | 155.7605 | 880.5709 | 1.0328 | 85.4983 | 58.481 | 7.31 | 504.9460 |
| 14000 | 0.2263 | 148.5429 | 820.9711 | 1.0057 | 85.4522 | 58.512 | 7.314 | 432.5430 |
| 15000 | 0.2424 | 142.2881 | 752.3494 | 0.9840 | 85.291 | 58.623 | 7.328 | 371.8599 |
| 16000 | 0.2586 | 138.4622 | 756.4453 | 0.9709 | 85.5234 | 58.464 | 7.308 | 645.2426 |
| 17000 | 0.2747 | 136.4131 | 709.4854 | 0.9606 | 85.2257 | 58.668 | 7.333 | 653.3060 |
| 18000 | 0.2909 | 133.8840 | 722.0003 | 0.9493 | 85.2137 | 58.676 | 7.335 | 538.2999 |
| 19000 | 0.3071 | 131.9743 | 726.1355 | 0.9435 | 85.2513 | 58.65 | 7.331 | 595.8792 |
| 20000 | 0.3232 | 129.7892 | 706.8889 | 0.9335 | 85.2873 | 58.625 | 7.328 | 420.4131 |
| 21000 | 0.3394 | 127.3829 | 659.1836 | 0.9238 | 85.0899 | 58.761 | 7.345 | 377.2113 |
| 22000 | 0.3556 | 125.8100 | 627.8823 | 0.9149 | 85.107 | 58.75 | 7.344 | 378.1191 |
| 23000 | 0.3717 | 124.3241 | 675.1288 | 0.9101 | 85.2843 | 58.627 | 7.328 | 407.0987 |
| 24000 | 0.3879 | 121.7446 | 648.3518 | 0.9012 | 85.2715 | 58.636 | 7.33 | 370.7195 |
| 25000 | 0.4040 | 121.8676 | 673.0380 | 0.8998 | 85.4414 | 58.52 | 7.315 | 401.9131 |
| 26000 | 0.4202 | 121.1881 | 598.3207 | 0.8906 | 85.1925 | 58.691 | 7.336 | 455.3015 |
| 27000 | 0.4364 | 119.8778 | 614.9578 | 0.8859 | 85.3813 | 58.561 | 7.32 | 291.9007 |
| 28000 | 0.4525 | 119.7104 | 589.9427 | 0.8831 | 85.3094 | 58.61 | 7.326 | 313.4760 |
| 29000 | 0.4687 | 118.6553 | 652.7549 | 0.8794 | 85.2629 | 58.642 | 7.33 | 299.0819 |
| 30000 | 0.4848 | 118.7475 | 602.5115 | 0.8775 | 85.4036 | 58.546 | 7.318 | 355.6388 |
| 31000 | 0.5010 | 118.1863 | 610.4652 | 0.8759 | 85.1743 | 58.703 | 7.338 | 275.1334 |
| 32000 | 0.5172 | 117.4726 | 628.6798 | 0.8750 | 85.2859 | 58.626 | 7.328 | 301.3671 |
| 33000 | 0.5333 | 115.1694 | 602.0021 | 0.8713 | 85.2629 | 58.642 | 7.33 | 277.0137 |
| 34000 | 0.5495 | 115.8600 | 574.1846 | 0.8689 | 85.1695 | 58.706 | 7.338 | 277.8658 |
| 35000 | 0.5657 | 114.0391 | 537.2504 | 0.8629 | 85.3032 | 58.614 | 7.327 | 307.7109 |
| 36000 | 0.5818 | 115.1694 | 602.9366 | 0.8660 | 85.3327 | 58.594 | 7.324 | 328.2996 |
| 37000 | 0.5980 | 113.9152 | 575.0357 | 0.8590 | 85.7449 | 58.313 | 7.289 | 332.3134 |
| 38000 | 0.6141 | 114.4739 | 573.7802 | 0.8618 | 85.7064 | 58.339 | 7.292 | 270.8683 |
| 39000 | 0.6303 | 112.6310 | 546.8427 | 0.8543 | 85.2884 | 58.625 | 7.328 | 289.1075 |
| 40000 | 0.6465 | 112.8762 | 570.1909 | 0.8537 | 85.2282 | 58.666 | 7.333 | 257.7758 |
| 41000 | 0.6626 | 112.9112 | 548.9287 | 0.8543 | 85.3272 | 58.598 | 7.325 | 325.8972 |
| 42000 | 0.6788 | 111.7424 | 549.7032 | 0.8534 | 85.5416 | 58.451 | 7.306 | 291.7448 |
| 43000 | 0.6949 | 112.2556 | 568.9060 | 0.8524 | 85.2667 | 58.64 | 7.33 | 310.9328 |
| 44000 | 0.7111 | 110.7490 | 603.5746 | 0.8525 | 85.3547 | 58.579 | 7.322 | 269.4612 |
| 45000 | 0.7273 | 112.0378 | 593.0288 | 0.8486 | 85.4267 | 58.53 | 7.316 | 378.8268 |
| 46000 | 0.7434 | 111.5950 | 589.0699 | 0.8492 | 85.0567 | 58.784 | 7.348 | 364.7776 |
| 47000 | 0.7596 | 112.7010 | 588.7380 | 0.8558 | 85.186 | 58.695 | 7.337 | 446.9284 |
| 48000 | 0.7758 | 114.3584 | 519.3724 | 0.8590 | 85.1156 | 58.744 | 7.343 | 2148.5159 |
| 49000 | 0.7919 | 115.1962 | 590.7754 | 0.8648 | 85.1589 | 58.714 | 7.339 | 430.9863 |
| 50000 | 0.8081 | 114.1809 | 614.9578 | 0.8597 | 85.2319 | 58.663 | 7.333 | 309.8965 |
| 51000 | 0.8242 | 112.0117 | 593.5725 | 0.8551 | 85.2128 | 58.677 | 7.335 | 423.6820 |
| 52000 | 0.8404 | 109.5515 | 563.6358 | 0.8457 | 85.2439 | 58.655 | 7.332 | 337.9521 |
| 53000 | 0.8566 | 109.7388 | 550.0908 | 0.8446 | 85.1744 | 58.703 | 7.338 | 412.3510 |
| 54000 | 0.8727 | 111.3959 | 551.1781 | 0.8453 | 85.2832 | 58.628 | 7.329 | 368.3508 |
| 55000 | 0.8889 | 111.1712 | 575.0760 | 0.8450 | 85.1663 | 58.709 | 7.339 | 283.8286 |
| 56000 | 0.9051 | 110.6545 | 557.3918 | 0.8454 | 85.4688 | 58.501 | 7.313 | 360.2272 |
| 57000 | 0.9212 | 110.4055 | 604.0854 | 0.8479 | 85.3777 | 58.563 | 7.32 | 420.6379 |
| 58000 | 0.9374 | 111.5257 | 635.6327 | 0.8466 | 85.6212 | 58.397 | 7.3 | 492.8218 |
| 59000 | 0.9535 | 109.2372 | 581.6412 | 0.8423 | 85.6034 | 58.409 | 7.301 | 366.9761 |
| 60000 | 0.9697 | 108.7379 | 565.1876 | 0.8362 | 85.2814 | 58.629 | 7.329 | 331.8257 |
| 61000 | 0.9859 | 108.9746 | 583.4484 | 0.8370 | 85.1729 | 58.704 | 7.338 | 399.4518 |
| 61875 | 1.0 | 109.4665 | 569.9899 | 0.8351 | 85.5074 | 58.474 | 7.309 | 290.9278 |
Framework versions
- Distily 0.2.0
- Transformers 4.44.0
- Pytorch 2.3.0
- Datasets 2.21.0