GePaBERT

This model is a fine-tuned version of deepset/gbert-large on a corpus of parliamentary speeches held in the German Bundestag. It was specifically designed for the KONVENS 2023 shared task on speaker attribution. It achieves the following results on the evaluation set:

  • Loss: 0.7997
  • Accuracy: 0.8020

Training and evaluation data

The corpus of parliamentary speeches covers speeches held in the German Bundestag during the 9th-20th legislative period, from 1980 to April 2023. (757 MB) The speeches were automatically prepared from the publicly available plenary protocols, using the extraction pipeline Open Discourse (GitHub code). Evaluation was done on a randomly-sampled 5% held-out dataset.

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 2e-05
  • train_batch_size: 8
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • num_epochs: 5

Training results

Training Loss Epoch Step Accuracy Validation Loss
1.0697 0.1 3489 0.7697 0.9802
1.0339 0.2 6978 0.7727 0.9562
1.0203 0.3 10467 0.7739 0.9463
1.0215 0.4 13956 0.7743 0.9477
1.0046 0.5 17445 0.7779 0.9299
1.0036 0.6 20934 0.7764 0.9372
1.2439 0.7 24423 0.7352 1.2473
1.4382 0.8 27912 0.6947 1.5782
1.1744 0.9 31401 0.7764 0.9360
0.9718 1.0 34890 0.7799 0.9179
0.9557 1.1 38379 0.7824 0.9038
0.947 1.2 41868 0.7830 0.9000
0.9487 1.3 45357 0.7833 0.8982
0.9457 1.4 48846 0.7851 0.8862
0.9442 1.5 52335 0.7863 0.8839
0.9473 1.6 55824 0.7850 0.8855
0.9388 1.7 59313 0.7865 0.8771
0.9293 1.8 62802 0.7868 0.8805
0.9242 1.9 66291 0.7873 0.8738
0.9241 2.0 69780 0.7872 0.8757
0.9127 2.1 73269 0.7896 0.8641
0.9114 2.2 76758 0.7900 0.8627
0.9095 2.3 80247 0.7913 0.8540
0.9042 2.4 83736 0.7920 0.8518
0.8999 2.5 87225 0.7919 0.8514
0.899 2.6 90714 0.7918 0.8543
0.8945 2.7 94203 0.7935 0.8418
0.8867 2.8 97692 0.7934 0.8437
0.893 2.9 101181 0.7938 0.8414
0.8798 3.0 104670 0.7951 0.8359
0.868 3.1 108159 0.7943 0.8375
0.8736 3.2 111648 0.7956 0.8323
0.8756 3.3 115137 0.7959 0.8315
0.8681 3.4 118626 0.7964 0.8258
0.8726 3.5 122115 0.7966 0.8266
0.8594 3.6 125604 0.7967 0.8246
0.8515 3.7 129093 0.7973 0.8227
0.8568 3.8 132582 0.7979 0.8195
0.8626 3.9 136071 0.7983 0.8173
0.8585 4.0 139560 0.7978 0.8190
0.8497 4.1 143049 0.7991 0.8127
0.8383 4.2 146538 0.7992 0.8154
0.8457 4.3 150027 0.8002 0.8080
0.8353 4.4 153516 0.8005 0.8077
0.8393 4.5 157005 0.8009 0.8027
0.8417 4.6 160494 0.8050 0.8007
0.836 4.7 163983 0.8004 0.8017
0.8317 4.8 167472 0.7993 0.8021
0.832 4.9 170961 0.8011 0.8013

Framework versions

  • Transformers 4.30.2
  • Pytorch 2.0.1+cu117
  • Datasets 2.13.1
  • Tokenizers 0.13.3
Downloads last month
27
Safetensors
Model size
336M params
Tensor type
I64
·
F32
·
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.