dhanishetty's picture
End of training
bbcc84c verified
metadata
base_model: albert/albert-xxlarge-v2
library_name: peft
license: apache-2.0
tags:
  - generated_from_trainer
model-index:
  - name: albert-xxlarge-v2-Adapters_2
    results: []

Visualize in Weights & Biases

albert-xxlarge-v2-Adapters_2

This model is a fine-tuned version of albert/albert-xxlarge-v2 on an unknown dataset. It achieves the following results on the evaluation set:

  • Loss: 0.5145

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 5e-05
  • train_batch_size: 32
  • eval_batch_size: 32
  • seed: 42
  • gradient_accumulation_steps: 4
  • total_train_batch_size: 128
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: inverse_sqrt
  • lr_scheduler_warmup_ratio: 0.1
  • num_epochs: 5
  • mixed_precision_training: Native AMP

Training results

Training Loss Epoch Step Validation Loss
1.1194 0.2291 50 1.1023
1.0831 0.4582 100 1.0770
1.0148 0.6873 150 0.9637
0.9064 0.9164 200 0.8538
0.7693 1.1455 250 0.7700
0.7445 1.3746 300 0.7098
0.7045 1.6037 350 0.6914
0.6733 1.8328 400 0.6374
0.6239 2.0619 450 0.6226
0.6208 2.2910 500 0.5913
0.6153 2.5200 550 0.5811
0.591 2.7491 600 0.5630
0.5894 2.9782 650 0.5500
0.5788 3.2073 700 0.5461
0.5187 3.4364 750 0.5353
0.5271 3.6655 800 0.5386
0.5812 3.8946 850 0.5309
0.5532 4.1237 900 0.5296
0.5602 4.3528 950 0.5204
0.5209 4.5819 1000 0.5144
0.5579 4.8110 1050 0.5145

Framework versions

  • PEFT 0.11.1
  • Transformers 4.42.4
  • Pytorch 2.1.0
  • Datasets 2.20.0
  • Tokenizers 0.19.1