|
--- |
|
license: mit |
|
tags: |
|
- generated_from_trainer |
|
metrics: |
|
- accuracy |
|
model-index: |
|
- name: deberta-v3-large |
|
results: [] |
|
--- |
|
|
|
# deberta-v3-large-sentiment |
|
|
|
This model is a fine-tuned version of [microsoft/deberta-v3-large](https://huggingface.co/microsoft/deberta-v3-large) on an [tweet_eval](https://huggingface.co/datasets/tweet_eval) dataset. |
|
|
|
## Model description |
|
|
|
Test set results: |
|
|
|
| Model | Emotion | Hate | Irony | Offensive | Sentiment | |
|
| ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | |
|
| deberta-v3-large | **86.3** | **61.3** | **87.1** | **86.4** | **73.9** | |
|
| BERTweet | 79.3 | - | 82.1 | 79.5 | 73.4 | |
|
| RoB-RT | 79.5 | 52.3 | 61.7 | 80.5 | 69.3 | |
|
|
|
[source:papers_with_code](https://paperswithcode.com/sota/sentiment-analysis-on-tweeteval) |
|
|
|
|
|
## Intended uses & limitations |
|
|
|
Classifying attributes of interest on tweeter like data. |
|
|
|
## Training and evaluation data |
|
|
|
[tweet_eval](https://huggingface.co/datasets/tweet_eval) dataset. |
|
|
|
## Training procedure |
|
|
|
Fine tuned and evaluated with [run_glue.py]() |
|
|
|
### Training hyperparameters |
|
|
|
The following hyperparameters were used during training: |
|
- learning_rate: 5e-06 |
|
- train_batch_size: 16 |
|
- eval_batch_size: 16 |
|
- seed: 42 |
|
- gradient_accumulation_steps: 2 |
|
- total_train_batch_size: 32 |
|
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 |
|
- lr_scheduler_type: linear |
|
- lr_scheduler_warmup_steps: 50 |
|
- num_epochs: 10.0 |
|
|
|
### Training results |
|
|
|
| Training Loss | Epoch | Step | Validation Loss | Accuracy | |
|
|:-------------:|:-----:|:-----:|:---------------:|:--------:| |
|
| 1.0614 | 0.07 | 100 | 1.0196 | 0.4345 | |
|
| 0.8601 | 0.14 | 200 | 0.7561 | 0.6460 | |
|
| 0.734 | 0.21 | 300 | 0.6796 | 0.6955 | |
|
| 0.6753 | 0.28 | 400 | 0.6521 | 0.7000 | |
|
| 0.6408 | 0.35 | 500 | 0.6119 | 0.7440 | |
|
| 0.5991 | 0.42 | 600 | 0.6034 | 0.7370 | |
|
| 0.6069 | 0.49 | 700 | 0.5976 | 0.7375 | |
|
| 0.6122 | 0.56 | 800 | 0.5871 | 0.7425 | |
|
| 0.5908 | 0.63 | 900 | 0.5935 | 0.7445 | |
|
| 0.5884 | 0.7 | 1000 | 0.5792 | 0.7520 | |
|
| 0.5839 | 0.77 | 1100 | 0.5780 | 0.7555 | |
|
| 0.5772 | 0.84 | 1200 | 0.5727 | 0.7570 | |
|
| 0.5895 | 0.91 | 1300 | 0.5601 | 0.7550 | |
|
| 0.5757 | 0.98 | 1400 | 0.5613 | 0.7525 | |
|
| 0.5121 | 1.05 | 1500 | 0.5867 | 0.7600 | |
|
| 0.5254 | 1.12 | 1600 | 0.5595 | 0.7630 | |
|
| 0.5074 | 1.19 | 1700 | 0.5594 | 0.7585 | |
|
| 0.4947 | 1.26 | 1800 | 0.5697 | 0.7575 | |
|
| 0.5019 | 1.33 | 1900 | 0.5665 | 0.7580 | |
|
| 0.5005 | 1.4 | 2000 | 0.5484 | 0.7655 | |
|
| 0.5125 | 1.47 | 2100 | 0.5626 | 0.7605 | |
|
| 0.5241 | 1.54 | 2200 | 0.5561 | 0.7560 | |
|
| 0.5198 | 1.61 | 2300 | 0.5602 | 0.7600 | |
|
| 0.5124 | 1.68 | 2400 | 0.5654 | 0.7490 | |
|
| 0.5096 | 1.75 | 2500 | 0.5803 | 0.7515 | |
|
| 0.4885 | 1.82 | 2600 | 0.5889 | 0.75 | |
|
| 0.5111 | 1.89 | 2700 | 0.5508 | 0.7665 | |
|
| 0.4868 | 1.96 | 2800 | 0.5621 | 0.7635 | |
|
| 0.4599 | 2.04 | 2900 | 0.5995 | 0.7615 | |
|
| 0.4147 | 2.11 | 3000 | 0.6202 | 0.7530 | |
|
| 0.4233 | 2.18 | 3100 | 0.5875 | 0.7625 | |
|
| 0.4324 | 2.25 | 3200 | 0.5794 | 0.7610 | |
|
| 0.4141 | 2.32 | 3300 | 0.5902 | 0.7460 | |
|
| 0.4306 | 2.39 | 3400 | 0.6053 | 0.7545 | |
|
| 0.4266 | 2.46 | 3500 | 0.5979 | 0.7570 | |
|
| 0.4227 | 2.53 | 3600 | 0.5920 | 0.7650 | |
|
| 0.4226 | 2.6 | 3700 | 0.6166 | 0.7455 | |
|
| 0.3978 | 2.67 | 3800 | 0.6126 | 0.7560 | |
|
| 0.3954 | 2.74 | 3900 | 0.6152 | 0.7550 | |
|
| 0.4209 | 2.81 | 4000 | 0.5980 | 0.75 | |
|
| 0.3982 | 2.88 | 4100 | 0.6096 | 0.7490 | |
|
| 0.4016 | 2.95 | 4200 | 0.6541 | 0.7425 | |
|
| 0.3966 | 3.02 | 4300 | 0.6377 | 0.7545 | |
|
| 0.3074 | 3.09 | 4400 | 0.6860 | 0.75 | |
|
| 0.3551 | 3.16 | 4500 | 0.6160 | 0.7550 | |
|
| 0.3323 | 3.23 | 4600 | 0.6714 | 0.7520 | |
|
| 0.3171 | 3.3 | 4700 | 0.6538 | 0.7535 | |
|
| 0.3403 | 3.37 | 4800 | 0.6774 | 0.7465 | |
|
| 0.3396 | 3.44 | 4900 | 0.6726 | 0.7465 | |
|
| 0.3259 | 3.51 | 5000 | 0.6465 | 0.7480 | |
|
| 0.3392 | 3.58 | 5100 | 0.6860 | 0.7460 | |
|
| 0.3251 | 3.65 | 5200 | 0.6697 | 0.7495 | |
|
| 0.3253 | 3.72 | 5300 | 0.6770 | 0.7430 | |
|
| 0.3455 | 3.79 | 5400 | 0.7177 | 0.7360 | |
|
| 0.3323 | 3.86 | 5500 | 0.6943 | 0.7400 | |
|
| 0.3335 | 3.93 | 5600 | 0.6507 | 0.7555 | |
|
| 0.3368 | 4.0 | 5700 | 0.6580 | 0.7485 | |
|
| 0.2479 | 4.07 | 5800 | 0.7667 | 0.7430 | |
|
| 0.2613 | 4.14 | 5900 | 0.7513 | 0.7505 | |
|
| 0.2557 | 4.21 | 6000 | 0.7927 | 0.7485 | |
|
| 0.243 | 4.28 | 6100 | 0.7792 | 0.7450 | |
|
| 0.2473 | 4.35 | 6200 | 0.8107 | 0.7355 | |
|
| 0.2447 | 4.42 | 6300 | 0.7851 | 0.7370 | |
|
| 0.2515 | 4.49 | 6400 | 0.7529 | 0.7465 | |
|
| 0.274 | 4.56 | 6500 | 0.7390 | 0.7465 | |
|
| 0.2674 | 4.63 | 6600 | 0.7658 | 0.7460 | |
|
| 0.2416 | 4.7 | 6700 | 0.7915 | 0.7485 | |
|
| 0.2432 | 4.77 | 6800 | 0.7989 | 0.7435 | |
|
| 0.2595 | 4.84 | 6900 | 0.7850 | 0.7380 | |
|
| 0.2736 | 4.91 | 7000 | 0.7577 | 0.7395 | |
|
| 0.2783 | 4.98 | 7100 | 0.7650 | 0.7405 | |
|
| 0.2304 | 5.05 | 7200 | 0.8542 | 0.7385 | |
|
| 0.1937 | 5.12 | 7300 | 0.8390 | 0.7345 | |
|
| 0.1878 | 5.19 | 7400 | 0.9150 | 0.7330 | |
|
| 0.1921 | 5.26 | 7500 | 0.8792 | 0.7405 | |
|
| 0.1916 | 5.33 | 7600 | 0.8892 | 0.7410 | |
|
| 0.2011 | 5.4 | 7700 | 0.9012 | 0.7325 | |
|
| 0.211 | 5.47 | 7800 | 0.8608 | 0.7420 | |
|
| 0.2194 | 5.54 | 7900 | 0.8852 | 0.7320 | |
|
| 0.205 | 5.61 | 8000 | 0.8803 | 0.7385 | |
|
| 0.1981 | 5.68 | 8100 | 0.8681 | 0.7330 | |
|
| 0.1908 | 5.75 | 8200 | 0.9020 | 0.7435 | |
|
| 0.1942 | 5.82 | 8300 | 0.8780 | 0.7410 | |
|
| 0.1958 | 5.89 | 8400 | 0.8937 | 0.7345 | |
|
| 0.1883 | 5.96 | 8500 | 0.9121 | 0.7360 | |
|
| 0.1819 | 6.04 | 8600 | 0.9409 | 0.7430 | |
|
| 0.145 | 6.11 | 8700 | 1.1390 | 0.7265 | |
|
| 0.1696 | 6.18 | 8800 | 0.9189 | 0.7430 | |
|
| 0.1488 | 6.25 | 8900 | 0.9718 | 0.7400 | |
|
| 0.1637 | 6.32 | 9000 | 0.9702 | 0.7450 | |
|
| 0.1547 | 6.39 | 9100 | 1.0033 | 0.7410 | |
|
| 0.1605 | 6.46 | 9200 | 0.9973 | 0.7355 | |
|
| 0.1552 | 6.53 | 9300 | 1.0491 | 0.7290 | |
|
| 0.1731 | 6.6 | 9400 | 1.0271 | 0.7335 | |
|
| 0.1738 | 6.67 | 9500 | 0.9575 | 0.7430 | |
|
| 0.1669 | 6.74 | 9600 | 0.9614 | 0.7350 | |
|
| 0.1347 | 6.81 | 9700 | 1.0263 | 0.7365 | |
|
| 0.1593 | 6.88 | 9800 | 1.0173 | 0.7360 | |
|
| 0.1549 | 6.95 | 9900 | 1.0398 | 0.7350 | |
|
| 0.1675 | 7.02 | 10000 | 0.9975 | 0.7380 | |
|
| 0.1182 | 7.09 | 10100 | 1.1059 | 0.7350 | |
|
| 0.1351 | 7.16 | 10200 | 1.0933 | 0.7400 | |
|
| 0.1496 | 7.23 | 10300 | 1.0731 | 0.7355 | |
|
| 0.1197 | 7.3 | 10400 | 1.1089 | 0.7360 | |
|
| 0.1111 | 7.37 | 10500 | 1.1381 | 0.7405 | |
|
| 0.1494 | 7.44 | 10600 | 1.0252 | 0.7425 | |
|
| 0.1235 | 7.51 | 10700 | 1.0906 | 0.7360 | |
|
| 0.133 | 7.58 | 10800 | 1.1796 | 0.7375 | |
|
| 0.1248 | 7.65 | 10900 | 1.1332 | 0.7420 | |
|
| 0.1268 | 7.72 | 11000 | 1.1304 | 0.7415 | |
|
| 0.1368 | 7.79 | 11100 | 1.1345 | 0.7380 | |
|
| 0.1228 | 7.86 | 11200 | 1.2018 | 0.7320 | |
|
| 0.1281 | 7.93 | 11300 | 1.1884 | 0.7350 | |
|
| 0.1449 | 8.0 | 11400 | 1.1571 | 0.7345 | |
|
| 0.1025 | 8.07 | 11500 | 1.1538 | 0.7345 | |
|
| 0.1199 | 8.14 | 11600 | 1.2113 | 0.7390 | |
|
| 0.1016 | 8.21 | 11700 | 1.2882 | 0.7370 | |
|
| 0.114 | 8.28 | 11800 | 1.2872 | 0.7390 | |
|
| 0.1019 | 8.35 | 11900 | 1.2876 | 0.7380 | |
|
| 0.1142 | 8.42 | 12000 | 1.2791 | 0.7385 | |
|
| 0.1135 | 8.49 | 12100 | 1.2883 | 0.7380 | |
|
| 0.1139 | 8.56 | 12200 | 1.2829 | 0.7360 | |
|
| 0.1107 | 8.63 | 12300 | 1.2698 | 0.7365 | |
|
| 0.1183 | 8.7 | 12400 | 1.2660 | 0.7345 | |
|
| 0.1064 | 8.77 | 12500 | 1.2889 | 0.7365 | |
|
| 0.0895 | 8.84 | 12600 | 1.3480 | 0.7330 | |
|
| 0.1244 | 8.91 | 12700 | 1.2872 | 0.7325 | |
|
| 0.1209 | 8.98 | 12800 | 1.2681 | 0.7375 | |
|
| 0.1144 | 9.05 | 12900 | 1.2711 | 0.7370 | |
|
| 0.1034 | 9.12 | 13000 | 1.2801 | 0.7360 | |
|
| 0.113 | 9.19 | 13100 | 1.2801 | 0.7350 | |
|
| 0.0994 | 9.26 | 13200 | 1.2920 | 0.7360 | |
|
| 0.0966 | 9.33 | 13300 | 1.2761 | 0.7335 | |
|
| 0.0939 | 9.4 | 13400 | 1.2909 | 0.7365 | |
|
| 0.0975 | 9.47 | 13500 | 1.2953 | 0.7360 | |
|
| 0.0842 | 9.54 | 13600 | 1.3179 | 0.7335 | |
|
| 0.0871 | 9.61 | 13700 | 1.3149 | 0.7385 | |
|
| 0.1162 | 9.68 | 13800 | 1.3124 | 0.7350 | |
|
| 0.085 | 9.75 | 13900 | 1.3207 | 0.7355 | |
|
| 0.0966 | 9.82 | 14000 | 1.3248 | 0.7335 | |
|
| 0.1064 | 9.89 | 14100 | 1.3261 | 0.7335 | |
|
| 0.1046 | 9.96 | 14200 | 1.3255 | 0.7360 | |
|
|
|
|
|
### Framework versions |
|
|
|
- Transformers 4.20.0.dev0 |
|
- Pytorch 1.9.0 |
|
- Datasets 2.2.2 |
|
- Tokenizers 0.11.6 |
|
|
