fedcsis-slot_baseline-xlm_r-pl

This model is a fine-tuned version of xlm-roberta-base on the leyzer-fedcsis dataset.

Results on test set:

• Precision: 0.9621
• Recall: 0.9583
• F1: 0.9602
• Accuracy: 0.9857

It achieves the following results on the evaluation set:

• Loss: 0.1009
• Precision: 0.9579
• Recall: 0.9512
• F1: 0.9546
• Accuracy: 0.9860

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 10

Training results

Training Loss	Epoch	Step	Validation Loss	Precision	Recall	F1	Accuracy
1.1608	1.0	798	0.2575	0.8881	0.8916	0.8898	0.9532
0.1561	2.0	1596	0.1188	0.9459	0.9389	0.9424	0.9806
0.0979	3.0	2394	0.1060	0.9507	0.9486	0.9497	0.9838
0.0579	4.0	3192	0.0916	0.9573	0.9475	0.9524	0.9851
0.0507	5.0	3990	0.1109	0.9527	0.9506	0.9516	0.9839
0.0344	6.0	4788	0.0987	0.9575	0.9488	0.9531	0.9855
0.0266	7.0	5586	0.1010	0.9584	0.9501	0.9542	0.9854
0.0211	8.0	6384	0.1051	0.9575	0.9498	0.9536	0.9855
0.0168	9.0	7182	0.1009	0.9577	0.9516	0.9546	0.9861
0.016	10.0	7980	0.1009	0.9579	0.9512	0.9546	0.9860

Per slot evaluation

slot_name	precision	recall	f1	tc_size
album	0.2000	0.3333	0.2500	9
all_lang	1.0000	1.0000	1.0000	5
artist	0.9341	0.9444	0.9392	90
av_alias	0.6667	0.8000	0.7273	5
caption	0.9651	0.9432	0.9540	88
category	0.0000	0.0000	0.0000	1
category_a	1.0000	0.9167	0.9565	12
category_b	1.0000	1.0000	1.0000	25
channel	0.9492	0.9333	0.9412	60
channel_id	0.9701	0.9644	0.9673	337
count	1.0000	0.9167	0.9565	12
date	0.9764	0.9841	0.9802	126
date_day	1.0000	0.9500	0.9744	20
date_month	0.9677	1.0000	0.9836	30
device_name	0.9091	1.0000	0.9524	10
email	1.0000	0.9913	0.9956	115
event_name	0.8788	0.9355	0.9063	31
file_name	0.9778	0.9778	0.9778	45
file_size	1.0000	1.0000	1.0000	12
filename	0.9722	0.9589	0.9655	73
filter	1.0000	1.0000	1.0000	35
from	0.9811	0.9123	0.9455	57
hashtag	1.0000	1.0000	1.0000	28
img_query	0.9707	0.9678	0.9693	342
label	1.0000	1.0000	1.0000	5
location	0.9766	0.9728	0.9747	257
mail	1.0000	1.0000	1.0000	3
message	0.9250	0.9487	0.9367	117
mime_type	0.9375	1.0000	0.9677	15
name	0.9412	0.9796	0.9600	49
pathname	0.8889	0.8889	0.8889	18
percent	1.0000	1.0000	1.0000	3
phone_number	0.9774	0.9774	0.9774	177
phone_type	1.0000	1.0000	1.0000	21
picture_url	0.9846	0.9412	0.9624	68
playlist	0.9516	0.9672	0.9593	122
portal	0.9869	0.9869	0.9869	153
priority	0.7500	1.0000	0.8571	6
purpose	0.0000	0.0000	0.0000	5
query	0.9663	0.9690	0.9677	355
rating	0.9630	0.9286	0.9455	28
review_count	1.0000	1.0000	1.0000	20
section	0.9730	0.9730	0.9730	74
seek_time	1.0000	1.0000	1.0000	3
sender	1.0000	1.0000	1.0000	6
sender_address	1.0000	0.9444	0.9714	18
song	0.8824	0.8898	0.8861	118
src_lang_de	0.9880	0.9762	0.9820	84
src_lang_en	0.9455	0.9630	0.9541	54
src_lang_es	0.9853	0.9306	0.9571	72
src_lang_fr	0.9733	0.9733	0.9733	75
src_lang_it	0.9872	0.9506	0.9686	81
src_lang_pl	0.9818	1.0000	0.9908	54
status	0.8810	0.9487	0.9136	39
subject	0.9636	0.9725	0.9680	109
text_de	0.9762	0.9762	0.9762	84
text_en	0.9796	0.9697	0.9746	99
text_es	0.8734	0.9583	0.9139	72
text_fr	0.9733	0.9733	0.9733	75
text_it	0.9872	0.9506	0.9686	81
text_multi	0.0000	0.0000	0.0000	4
text_pl	0.9310	1.0000	0.9643	54
time	0.9063	0.8788	0.8923	33
to	0.9648	0.9648	0.9648	199
topic	0.0000	0.0000	0.0000	3
translator	0.9838	0.9838	0.9838	185
trg_lang_de	0.9474	0.9730	0.9600	37
trg_lang_en	1.0000	0.9565	0.9778	46
trg_lang_es	0.9792	0.9792	0.9792	48
trg_lang_fr	0.9808	1.0000	0.9903	51
trg_lang_general	0.9500	0.9500	0.9500	20
trg_lang_it	0.9825	0.9492	0.9655	59
trg_lang_pl	0.9302	0.9756	0.9524	41
txt_query	0.9375	0.9146	0.9259	82
username	0.9615	0.8929	0.9259	28
value	0.8750	0.8750	0.8750	8
weight	1.0000	1.0000	1.0000	3

Framework versions

• Transformers 4.27.4
• Pytorch 1.13.1+cu116
• Datasets 2.11.0
• Tokenizers 0.13.2

Citation

If you use this model, please cite the following:

@inproceedings{kubis2023caiccaic,
    author={Marek Kubis and Paweł Skórzewski and Marcin Sowański and Tomasz Ziętkiewicz},
    pages={1319–1324},
    title={Center for Artificial Intelligence Challenge on Conversational AI Correctness},
    booktitle={Proceedings of the 18th Conference on Computer Science and Intelligence Systems},
    year={2023},
    doi={10.15439/2023B6058},
    url={http://dx.doi.org/10.15439/2023B6058},
    volume={35},
    series={Annals of Computer Science and Information Systems}
}