KoModernBERT-chp-04

This model is a fine-tuned version of CocoRoF/KoModernBERT-chp-03 on the None dataset. It achieves the following results on the evaluation set:

• Loss: 2.1738

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• distributed_type: multi-GPU
• num_devices: 8
• gradient_accumulation_steps: 8
• total_train_batch_size: 512
• total_eval_batch_size: 64
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 1.0

Training results

Training Loss	Epoch	Step	Validation Loss
18.7395	0.0904	5000	2.3312
18.4567	0.1808	10000	2.3132
18.0597	0.2712	15000	2.2958
18.0592	0.3616	20000	2.2780
18.1118	0.4520	25000	2.2565
18.1207	0.5424	30000	2.2429
17.3473	0.6329	35000	2.2228
17.8666	0.7233	40000	2.2078
17.909	0.8137	45000	2.1974
17.1812	0.9041	50000	2.1855
17.5599	0.9945	55000	2.1738

Framework versions

• Transformers 4.48.1
• Pytorch 2.5.1+cu124
• Datasets 3.2.0
• Tokenizers 0.21.0