GIZ/SECTOR-multilabel-bge_f

SECTOR-multilabel-bge

This model is a fine-tuned version of BAAI/bge-base-en-v1.5 on the Policy-Classification dataset.

The loss function BCEWithLogitsLoss is modified with pos_weight to focus on recall, therefore instead of loss the evaluation metrics are used to assess the model performance during training It achieves the following results on the evaluation set:

• Loss: 0.6114
• Precision-micro: 0.6428
• Precision-samples: 0.7488
• Precision-weighted: 0.6519
• Recall-micro: 0.7855
• Recall-samples: 0.8627
• Recall-weighted: 0.7855
• F1-micro: 0.7071
• F1-samples: 0.7638
• F1-weighted: 0.7109

Model description

The purpose of this model is to predict multiple labels simultaneously from a given input data. Specifically, the model will predict Sector labels - Agriculture,Buildings, Coastal Zone,Cross-Cutting Area,Disaster Risk Management (DRM),Economy-wide,Education,Energy,Environment,Health,Industries,LULUCF/Forestry,Social Development,Tourism, Transport,Urban,Waste,Water

Intended uses & limitations

More information needed

Training and evaluation data

• Training Dataset: 10123

  Class	Positive Count of Class
  Agriculture	2235
  Buildings	169
  Coastal Zone	698
  Cross-Cutting Area	1853
  Disaster Risk Management (DRM)	814
  Economy-wide	873
  Education	180
  Energy	2847
  Environment	905
  Health	662
  Industries	419
  LULUCF/Forestry	1861
  Social Development	507
  Tourism	192
  Transport	1173
  Urban	558
  Waste	714
  Water	1207

• Validation Dataset: 936

  Class	Positive Count of Class
  Agriculture	200
  Buildings	18
  Coastal Zone	71
  Cross-Cutting Area	180
  Disaster Risk Management (DRM)	85
  Economy-wide	85
  Education	23
  Energy	254
  Environment	91
  Health	68
  Industries	41
  LULUCF/Forestry	193
  Social Development	56
  Tourism	28
  Transport	107
  Urban	51
  Waste	59
  Water	106

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 7.04e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 300
• num_epochs: 7

Training results

Training Loss	Epoch	Step	Validation Loss	Precision-micro	Precision-samples	Precision-weighted	Recall-micro	Recall-samples	Recall-weighted	F1-micro	F1-samples	F1-weighted
0.7077	1.0	633	0.5490	0.4226	0.5465	0.4954	0.8211	0.8908	0.8211	0.5580	0.6243	0.5977
0.4546	2.0	1266	0.5009	0.4899	0.6127	0.5202	0.8438	0.9023	0.8438	0.6199	0.6822	0.6366
0.3105	3.0	1899	0.4947	0.5005	0.6593	0.5317	0.8508	0.8970	0.8508	0.6303	0.7125	0.6474
0.2044	4.0	2532	0.5430	0.5757	0.7044	0.5970	0.8106	0.8801	0.8106	0.6733	0.7379	0.6834
0.1314	5.0	3165	0.5633	0.6132	0.7385	0.6271	0.8065	0.8772	0.8065	0.6967	0.7606	0.7032
0.0892	6.0	3798	0.6073	0.6425	0.7499	0.6545	0.7844	0.8610	0.7844	0.7064	0.7634	0.7113
0.0721	7.0	4431	0.6114	0.6428	0.7488	0.6519	0.7855	0.8627	0.7855	0.7071	0.7638	0.7109

label	precision	recall	f1-score	support
Agriculture	0.740	0.840	0.786	200
Buildings	0.535	0.833	0.652	18
Coastal Zone	0.579	0.718	0.641	71
Cross-Cutting Area	0.551	0.738	0.631	180
Disaster Risk Management (DRM)	0.642	0.717	0.67	85
Economy-wide	0.401	0.600	0.481	85
Education	0.652	0.652	0.652	23
Energy	0.771	0.862	0.814	254
Environment	0.539	0.747	0.626	91
Health	0.743	0.808	0.774	68
Industries	0.648	0.853	0.736	41
LULUCF/Forestry	0.728	0.849	0.784	193
Social Development	0.661	0.767	0.710	56
Tourism	0.586	0.607	0.596	28
Transport	0.715	0.822	0.765	107
Urban	0.414	0.568	0.479	51
Waste	0.662	0.898	0.762	59
Water	0.601	.783	0.680	106

Environmental Impact

Carbon emissions were measured using CodeCarbon.

• Carbon Emitted: 0.05819 kg of CO2
• Hours Used: 1.43 hours

Training Hardware

• On Cloud: yes
• GPU Model: 1 x Tesla T4
• CPU Model: Intel(R) Xeon(R) CPU @ 2.00GHz
• RAM Size: 12.67 GB

Framework versions

• Transformers 4.38.1
• Pytorch 2.1.0+cu121
• Datasets 2.18.0
• Tokenizers 0.15.2