Hugging Face's logo

RylanSchaeffer
/
collapse_gemma-2-27b_hs2_accumulate_iter3_sftsd1

Safetensors
gemma2
trl
sft
Generated from Trainer
Model card Files Community
collapse_gemma-2-27b_hs2_accumulate_iter3_sftsd1 / README.md
RylanSchaeffer's picture
RylanSchaeffer
End of training
ad50cdf verified
preview code
|
raw
history blame contribute delete
5.38 kB
metadata
license: gemma
base_model: google/gemma-2-27b
tags:
  - trl
  - sft
  - generated_from_trainer
model-index:
  - name: collapse_gemma-2-27b_hs2_accumulate_iter3_sftsd1
    results: []

collapse_gemma-2-27b_hs2_accumulate_iter3_sftsd1

This model is a fine-tuned version of google/gemma-2-27b on an unknown dataset. It achieves the following results on the evaluation set:

  • Loss: 0.9280
  • Num Input Tokens Seen: 13412700

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 8e-06
  • train_batch_size: 4
  • eval_batch_size: 16
  • seed: 1
  • gradient_accumulation_steps: 32
  • total_train_batch_size: 128
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: constant_with_warmup
  • lr_scheduler_warmup_ratio: 0.05
  • num_epochs: 1

Training results

Training Loss Epoch Step Validation Loss Input Tokens Seen
No log 0 0 1.1282 0
2.1585 0.0187 5 1.0516 253472
2.0366 0.0374 10 0.9878 506396
2.2853 0.0562 15 0.9800 760944
1.9353 0.0749 20 0.9748 1012816
1.7788 0.0936 25 0.9765 1258660
1.5677 0.1123 30 0.9865 1505980
1.6266 0.1310 35 0.9797 1748944
1.3893 0.1498 40 0.9770 1996076
1.3214 0.1685 45 0.9758 2249964
1.2104 0.1872 50 0.9732 2502428
1.1943 0.2059 55 0.9673 2758156
0.9618 0.2246 60 0.9648 3002952
0.9917 0.2434 65 0.9608 3250420
0.9458 0.2621 70 0.9592 3498588
0.8799 0.2808 75 0.9541 3753220
0.9288 0.2995 80 0.9547 4005744
0.9042 0.3182 85 0.9524 4251648
0.7466 0.3370 90 0.9507 4504748
0.802 0.3557 95 0.9492 4759604
0.786 0.3744 100 0.9468 5010224
0.8059 0.3931 105 0.9463 5261388
0.7014 0.4118 110 0.9448 5508984
0.7977 0.4306 115 0.9438 5767344
0.9226 0.4493 120 0.9425 6015220
0.9092 0.4680 125 0.9414 6270096
0.692 0.4867 130 0.9401 6522928
0.7488 0.5054 135 0.9394 6774308
0.6813 0.5242 140 0.9378 7026956
0.9565 0.5429 145 0.9353 7281764
0.7867 0.5616 150 0.9364 7535708
0.6354 0.5803 155 0.9373 7783224
0.8341 0.5990 160 0.9340 8026812
0.834 0.6178 165 0.9358 8276260
0.7364 0.6365 170 0.9338 8529636
0.7822 0.6552 175 0.9329 8787372
0.8144 0.6739 180 0.9337 9033612
0.7588 0.6926 185 0.9321 9283952
0.6757 0.7114 190 0.9320 9528272
0.5925 0.7301 195 0.9327 9775216
0.6711 0.7488 200 0.9321 10031428
0.7888 0.7675 205 0.9301 10287112
0.7551 0.7862 210 0.9322 10539552
0.7367 0.8050 215 0.9328 10786728
0.6682 0.8237 220 0.9318 11033040
0.7802 0.8424 225 0.9310 11281864
0.7423 0.8611 230 0.9317 11537232
0.8502 0.8798 235 0.9309 11791856
0.7691 0.8986 240 0.9283 12041012
0.7173 0.9173 245 0.9318 12291188
0.7158 0.9360 250 0.9296 12542864
0.7733 0.9547 255 0.9307 12794508
0.6864 0.9734 260 0.9298 13055348
0.6458 0.9922 265 0.9288 13306708

Framework versions

  • Transformers 4.44.0
  • Pytorch 2.4.0+cu121
  • Datasets 2.20.0
  • Tokenizers 0.19.1