Edit model card

smollm2-1.7B-8k-mix7-ep2-v2-qlora-r16-a16-lr3e4-mix1-dpo

This model is a fine-tuned version of gabrielmbmb/smollm2-1.7B-8k-mix7-ep2-v2-qlora-r16-a16-lr3e4-mix1 on the HuggingFaceH4/ultrafeedback_binarized dataset. It achieves the following results on the evaluation set:

  • Loss: 0.5156
  • Rewards/chosen: -4.3948
  • Rewards/rejected: -5.4305
  • Rewards/accuracies: 0.7656
  • Rewards/margins: 1.0357
  • Logps/rejected: -849.2994
  • Logps/chosen: -752.2209
  • Logits/rejected: -1.2614
  • Logits/chosen: -1.2341

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0003
  • train_batch_size: 4
  • eval_batch_size: 8
  • seed: 42
  • distributed_type: multi-GPU
  • num_devices: 8
  • gradient_accumulation_steps: 4
  • total_train_batch_size: 128
  • total_eval_batch_size: 64
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_ratio: 0.1
  • num_epochs: 1

Training results

Training Loss Epoch Step Validation Loss Rewards/chosen Rewards/rejected Rewards/accuracies Rewards/margins Logps/rejected Logps/chosen Logits/rejected Logits/chosen
0.5629 0.2093 100 0.5765 -2.2363 -2.7515 0.7227 0.5152 -581.3958 -536.3666 -1.6836 -1.6572
0.5499 0.4186 200 0.5310 -3.6681 -4.4704 0.7734 0.8022 -753.2846 -679.5535 -0.8209 -0.8334
0.511 0.6279 300 0.5225 -4.8153 -5.7685 0.7695 0.9532 -883.0991 -794.2732 -1.1394 -1.1157
0.508 0.8373 400 0.5157 -4.3456 -5.3593 0.7695 1.0137 -842.1772 -747.2964 -1.2109 -1.1871

Framework versions

  • PEFT 0.13.2
  • Transformers 4.45.0
  • Pytorch 2.4.1+cu121
  • Datasets 3.0.2
  • Tokenizers 0.20.1
Downloads last month
23
Inference API
Unable to determine this model’s pipeline type. Check the docs .

Model tree for gabrielmbmb/smollm2-1.7B-8k-mix7-ep2-v2-qlora-r16-a16-lr3e4-mix1-dpo

Adapter
(1)
this model

Dataset used to train gabrielmbmb/smollm2-1.7B-8k-mix7-ep2-v2-qlora-r16-a16-lr3e4-mix1-dpo