distilbert-base-vietnamese-case

This model is a fine-tuned version of distilbert-base-cased on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 3.1978

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 64
• eval_batch_size: 64
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 1000

Training results

Training Loss	Epoch	Step	Validation Loss
6.6995	1.0	313	5.7838
5.7246	2.0	626	5.5341
5.4565	3.0	939	5.3280
5.271	4.0	1252	5.1409
5.0514	5.0	1565	4.9143
4.874	6.0	1878	4.7130
4.7083	7.0	2191	4.5682
4.5677	8.0	2504	4.3724
4.4244	9.0	2817	4.3262
4.3013	10.0	3130	4.1231
4.2077	11.0	3443	4.1388
4.1103	12.0	3756	3.8696
4.0141	13.0	4069	3.8849
3.9435	14.0	4382	3.7311
3.8604	15.0	4695	3.7155
3.804	16.0	5008	3.6445
3.7076	17.0	5321	3.5784
3.6807	18.0	5634	3.5516
3.6239	19.0	5947	3.4008
3.5729	20.0	6260	3.4827
3.5308	21.0	6573	3.3921
3.4707	22.0	6886	3.3729
3.4341	23.0	7199	3.3543
3.3989	24.0	7512	3.2836
3.3505	25.0	7825	3.3003
3.3256	26.0	8138	3.1750
3.2892	27.0	8451	3.1930
3.2614	28.0	8764	3.2089
3.2387	29.0	9077	3.1978

Framework versions

• Transformers 4.33.2
• Pytorch 2.0.1+cu117
• Datasets 2.14.5
• Tokenizers 0.13.3