GIZ/SECTOR-multilabel-climatebert_f

SECTOR-multilabel-climatebert

This model is a fine-tuned version of climatebert/distilroberta-base-climate-f on the Policy-Classification dataset.

The loss function BCEWithLogitsLoss is modified with pos_weight to focus on recall, therefore instead of loss the evaluation metrics are used to assess the model performance during training It achieves the following results on the evaluation set:

• Loss: 0.6028
• Precision-micro: 0.6395
• Precision-samples: 0.7543
• Precision-weighted: 0.6475
• Recall-micro: 0.7762
• Recall-samples: 0.8583
• Recall-weighted: 0.7762
• F1-micro: 0.7012
• F1-samples: 0.7655
• F1-weighted: 0.7041

Model description

The purpose of this model is to predict multiple labels simultaneously from a given input data. Specifically, the model will predict Sector labels - Agriculture,Buildings, Coastal Zone,Cross-Cutting Area,Disaster Risk Management (DRM),Economy-wide,Education,Energy,Environment,Health,Industries,LULUCF/Forestry,Social Development,Tourism, Transport,Urban,Waste,Water

Intended uses & limitations

More information needed

Training and evaluation data

• Training Dataset: 10123

  Class	Positive Count of Class
  Agriculture	2235
  Buildings	169
  Coastal Zone	698
  Cross-Cutting Area	1853
  Disaster Risk Management (DRM)	814
  Economy-wide	873
  Education	180
  Energy	2847
  Environment	905
  Health	662
  Industries	419
  LULUCF/Forestry	1861
  Social Development	507
  Tourism	192
  Transport	1173
  Urban	558
  Waste	714
  Water	1207

• Validation Dataset: 936

  Class	Positive Count of Class
  Agriculture	200
  Buildings	18
  Coastal Zone	71
  Cross-Cutting Area	180
  Disaster Risk Management (DRM)	85
  Economy-wide	85
  Education	23
  Energy	254
  Environment	91
  Health	68
  Industries	41
  LULUCF/Forestry	193
  Social Development	56
  Tourism	28
  Transport	107
  Urban	51
  Waste	59
  Water	106

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 9.07e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 300
• num_epochs: 7

Training results

Training Loss	Epoch	Step	Validation Loss	Precision-micro	Precision-samples	Precision-weighted	Recall-micro	Recall-samples	Recall-weighted	F1-micro	F1-samples	F1-weighted
0.6978	1.0	633	0.5968	0.3948	0.5274	0.4982	0.7873	0.8675	0.7873	0.5259	0.5996	0.5793
0.485	2.0	1266	0.5255	0.5089	0.6365	0.5469	0.7984	0.8749	0.7984	0.6216	0.6907	0.6384
0.3657	3.0	1899	0.5248	0.4984	0.6617	0.5397	0.8141	0.8769	0.8141	0.6183	0.7066	0.6393
0.2585	4.0	2532	0.5457	0.5807	0.7148	0.5992	0.8007	0.8752	0.8007	0.6732	0.7449	0.6813
0.1841	5.0	3165	0.5551	0.6016	0.7426	0.6192	0.7937	0.8677	0.7937	0.6844	0.7590	0.6917
0.1359	6.0	3798	0.5913	0.6349	0.7506	0.6449	0.7844	0.8676	0.7844	0.7018	0.7667	0.7057
0.1133	7.0	4431	0.6028	0.6395	0.7543	0.6475	0.7762	0.8583	0.7762	0.7012	0.7655	0.7041

label	precision	recall	f1-score	support
Agriculture	0.720	0.850	0.780	200
Buildings	0.636	0.777	0.700	18
Coastal Zone	0.562	0.760	0.646	71
Cross-Cutting Area	0.569	0.777	0.657	180
Disaster Risk Management (DRM)	0.567	0.694	0.624	85
Economy-wide	0.461	0.635	0.534	85
Education	0.608	0.608	0.608	23
Energy	0.816	0.838	0.827	254
Environment	0.561	0.703	0.624	91
Health	0.708	0.750	0.728	68
Industries	0.660	0.902	0.762	41
LULUCF/Forestry	0.676	0.844	0.751	193
Social Development	0.593	0.678	0.633	56
Tourism	0.551	0.571	0.561	28
Transport	0.700	0.766	0.732	107
Urban	0.414	0.568	0.479	51
Waste	0.658	0.881	0.753	59
Water	0.602	0.773	0.677	106

Environmental Impact

Carbon emissions were measured using CodeCarbon.

• Carbon Emitted: 0.02867 kg of CO2
• Hours Used: 0.706 hours

Training Hardware

• On Cloud: yes
• GPU Model: 1 x Tesla T4
• CPU Model: Intel(R) Xeon(R) CPU @ 2.00GHz
• RAM Size: 12.67 GB

Framework versions

• Transformers 4.38.1
• Pytorch 2.1.0+cu121
• Datasets 2.18.0
• Tokenizers 0.15.2