File size: 64,781 Bytes
82f0150 6d5161a 82f0150 6d5161a 82f0150 6d5161a 09b73ae 6d5161a 09b73ae 6d5161a 4cb5953 6d5161a 82f0150 6d5161a 82f0150 6d5161a 82f0150 6d5161a 82f0150 6d5161a aa16a20 6d5161a 613e8c5 6d5161a 82f0150 6d5161a 82f0150 c927baa 3e8cb87 c927baa 3e8cb87 c927baa 3e8cb87 c927baa 3e8cb87 6d5161a 82f0150 6d5161a 82f0150 6d5161a 09b73ae 82f0150 6d5161a 893885d 6d5161a 893885d 6d5161a 82f0150 6d5161a 82f0150 6d5161a 8414521 82f0150 6d5161a 82f0150 6d5161a 82f0150 6d5161a 82f0150 6d5161a |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 |
---
language:
- en
tags:
- text-classification
- zero-shot-classification
pipeline_tag: zero-shot-classification
library_name: transformers
license: mit
---
# Model description: deberta-v3-base-zeroshot-v1.1-all-33
The model is designed for zero-shot classification with the Hugging Face pipeline.
The model can do one universal classification task: determine whether a hypothesis is "true" or "not true" given a text
(`entailment` vs. `not_entailment`).
This task format is based on the Natural Language Inference task (NLI).
The task is so universal that any classification task can be reformulated into this task.
A detailed description of how the model was trained and how it can be used is available in this [paper](https://arxiv.org/pdf/2312.17543.pdf).
## Training data
The model was trained on a mixture of __33 datasets and 387 classes__ that have been reformatted into this universal format.
1. Five NLI datasets with ~885k texts: "mnli", "anli", "fever", "wanli", "ling"
2. 28 classification tasks reformatted into the universal NLI format. ~51k cleaned texts were used to avoid overfitting:
'amazonpolarity', 'imdb', 'appreviews', 'yelpreviews', 'rottentomatoes',
'emotiondair', 'emocontext', 'empathetic',
'financialphrasebank', 'banking77', 'massive',
'wikitoxic_toxicaggregated', 'wikitoxic_obscene', 'wikitoxic_threat', 'wikitoxic_insult', 'wikitoxic_identityhate',
'hateoffensive', 'hatexplain', 'biasframes_offensive', 'biasframes_sex', 'biasframes_intent',
'agnews', 'yahootopics',
'trueteacher', 'spam', 'wellformedquery',
'manifesto', 'capsotu'.
See details on each dataset here: https://github.com/MoritzLaurer/zeroshot-classifier/blob/main/datasets_overview.csv
Note that compared to other NLI models, this model predicts two classes (`entailment` vs. `not_entailment`)
as opposed to three classes (entailment/neutral/contradiction)
The model was only trained on English data. For __multilingual use-cases__,
I recommend machine translating texts to English with libraries like [EasyNMT](https://github.com/UKPLab/EasyNMT).
English-only models tend to perform better than multilingual models and
validation with English data can be easier if you don't speak all languages in your corpus.
### How to use the model
#### Simple zero-shot classification pipeline
```python
#!pip install transformers[sentencepiece]
from transformers import pipeline
text = "Angela Merkel is a politician in Germany and leader of the CDU"
hypothesis_template = "This example is about {}"
classes_verbalized = ["politics", "economy", "entertainment", "environment"]
zeroshot_classifier = pipeline("zero-shot-classification", model="MoritzLaurer/deberta-v3-base-zeroshot-v1.1-all-33")
output = zeroshot_classifier(text, classes_verbalized, hypothesis_template=hypothesis_template, multi_label=False)
print(output)
```
### Details on data and training
The code for preparing the data and training & evaluating the model is fully open-source here: https://github.com/MoritzLaurer/zeroshot-classifier/tree/main
Hyperparameters and other details are available in this Weights & Biases repo: https://wandb.ai/moritzlaurer/deberta-v3-base-zeroshot-v1-1-all-33/table?workspace=user-
## Metrics
Balanced accuracy is reported for all datasets.
`deberta-v3-base-zeroshot-v1.1-all-33` was trained on all datasets, with only maximum 500 texts per class to avoid overfitting.
The metrics on these datasets are therefore not strictly zeroshot, as the model has seen some data for each task during training.
`deberta-v3-base-zeroshot-v1.1-heldout` indicates zeroshot performance on the respective dataset.
To calculate these zeroshot metrics, the pipeline was run 28 times, each time with one dataset held out from training to simulate a zeroshot setup.
| | deberta-v3-base-mnli-fever-anli-ling-wanli-binary | deberta-v3-base-zeroshot-v1.1-heldout | deberta-v3-base-zeroshot-v1.1-all-33 |
|:---------------------------|---------------------------:|----------------------------------------:|---------------------------------------:|
| datasets mean (w/o nli) | 62 | 70.7 | 84 |
| amazonpolarity (2) | 91.7 | 95.7 | 96 |
| imdb (2) | 87.3 | 93.6 | 94.5 |
| appreviews (2) | 91.3 | 92.2 | 94.4 |
| yelpreviews (2) | 95.1 | 97.4 | 98.3 |
| rottentomatoes (2) | 83 | 88.7 | 90.8 |
| emotiondair (6) | 46.5 | 42.6 | 74.5 |
| emocontext (4) | 58.5 | 57.4 | 81.2 |
| empathetic (32) | 31.3 | 37.3 | 52.7 |
| financialphrasebank (3) | 78.3 | 68.9 | 91.2 |
| banking77 (72) | 18.9 | 46 | 73.7 |
| massive (59) | 44 | 56.6 | 78.9 |
| wikitoxic_toxicaggreg (2) | 73.7 | 82.5 | 90.5 |
| wikitoxic_obscene (2) | 77.3 | 91.6 | 92.6 |
| wikitoxic_threat (2) | 83.5 | 95.2 | 96.7 |
| wikitoxic_insult (2) | 79.6 | 91 | 91.6 |
| wikitoxic_identityhate (2) | 83.9 | 88 | 94.4 |
| hateoffensive (3) | 55.2 | 66.1 | 86 |
| hatexplain (3) | 44.1 | 57.6 | 76.9 |
| biasframes_offensive (2) | 56.8 | 85.4 | 87 |
| biasframes_sex (2) | 85.4 | 87 | 91.8 |
| biasframes_intent (2) | 56.3 | 85.2 | 87.8 |
| agnews (4) | 77.3 | 80 | 90.5 |
| yahootopics (10) | 53.6 | 57.7 | 72.8 |
| trueteacher (2) | 51.4 | 49.5 | 82.4 |
| spam (2) | 51.8 | 50 | 97.2 |
| wellformedquery (2) | 49.9 | 52.5 | 77.2 |
| manifesto (56) | 5.8 | 18.9 | 39.1 |
| capsotu (21) | 25.2 | 64 | 72.5 |
| mnli_m (2) | 92.4 | nan | 92.7 |
| mnli_mm (2) | 92.4 | nan | 92.5 |
| fevernli (2) | 89 | nan | 89.1 |
| anli_r1 (2) | 79.4 | nan | 80 |
| anli_r2 (2) | 68.4 | nan | 68.4 |
| anli_r3 (2) | 66.2 | nan | 68 |
| wanli (2) | 81.6 | nan | 81.8 |
| lingnli (2) | 88.4 | nan | 88.4 |
## Limitations and bias
The model can only do text classification tasks.
Please consult the original DeBERTa paper and the papers for the different datasets for potential biases.
## License
The base model (DeBERTa-v3) is published under the MIT license.
The datasets the model was fine-tuned on are published under a diverse set of licenses.
The following table provides an overview of the non-NLI datasets used for fine-tuning,
information on licenses, the underlying papers etc.: https://github.com/MoritzLaurer/zeroshot-classifier/blob/main/datasets_overview.csv
## Citation
If you use this model academically, please cite:
```
@misc{laurer_building_2023,
title = {Building {Efficient} {Universal} {Classifiers} with {Natural} {Language} {Inference}},
url = {http://arxiv.org/abs/2312.17543},
doi = {10.48550/arXiv.2312.17543},
abstract = {Generative Large Language Models (LLMs) have become the mainstream choice for fewshot and zeroshot learning thanks to the universality of text generation. Many users, however, do not need the broad capabilities of generative LLMs when they only want to automate a classification task. Smaller BERT-like models can also learn universal tasks, which allow them to do any text classification task without requiring fine-tuning (zeroshot classification) or to learn new tasks with only a few examples (fewshot), while being significantly more efficient than generative LLMs. This paper (1) explains how Natural Language Inference (NLI) can be used as a universal classification task that follows similar principles as instruction fine-tuning of generative LLMs, (2) provides a step-by-step guide with reusable Jupyter notebooks for building a universal classifier, and (3) shares the resulting universal classifier that is trained on 33 datasets with 389 diverse classes. Parts of the code we share has been used to train our older zeroshot classifiers that have been downloaded more than 55 million times via the Hugging Face Hub as of December 2023. Our new classifier improves zeroshot performance by 9.4\%.},
urldate = {2024-01-05},
publisher = {arXiv},
author = {Laurer, Moritz and van Atteveldt, Wouter and Casas, Andreu and Welbers, Kasper},
month = dec,
year = {2023},
note = {arXiv:2312.17543 [cs]},
keywords = {Computer Science - Artificial Intelligence, Computer Science - Computation and Language},
}
```
### Ideas for cooperation or questions?
If you have questions or ideas for cooperation, contact me at m{dot}laurer{at}vu{dot}nl or [LinkedIn](https://www.linkedin.com/in/moritz-laurer/)
### Debugging and issues
Note that DeBERTa-v3 was released on 06.12.21 and older versions of HF Transformers can have issues running the model (e.g. resulting in an issue with the tokenizer). Using Transformers>=4.13 might solve some issues.
Also make sure to install sentencepiece to avoid tokenizer errors. Run: `pip install transformers[sentencepiece]` or `pip install sentencepiece`
### Hypotheses used for classification
The hypotheses in the tables below were used to fine-tune the model.
Inspecting them can help users get a feeling for which type of hypotheses and tasks the model was trained on.
You can formulate your own hypotheses by changing the `hypothesis_template` of the zeroshot pipeline. For example:
```python
from transformers import pipeline
text = "Angela Merkel is a politician in Germany and leader of the CDU"
hypothesis_template = "Merkel is the leader of the party: {}"
classes_verbalized = ["CDU", "SPD", "Greens"]
zeroshot_classifier = pipeline("zero-shot-classification", model="MoritzLaurer/deberta-v3-base-zeroshot-v1.1-all-33")
output = zeroshot_classifier(text, classes_verbalised, hypothesis_template=hypothesis_template, multi_label=False)
print(output)
```
Note that a few rows in the `massive` and `banking77` datasets contain `nan` because some classes were so ambiguous/unclear that I excluded them from the data.
#### wellformedquery
| label | hypothesis |
|:----------------|:-----------------------------------------------|
| not_well_formed | This example is not a well formed Google query |
| well_formed | This example is a well formed Google query. |
#### biasframes_sex
| label | hypothesis |
|:--------|:-----------------------------------------------------------|
| not_sex | This example does not contain allusions to sexual content. |
| sex | This example contains allusions to sexual content. |
#### biasframes_intent
| label | hypothesis |
|:-----------|:-----------------------------------------------------------------|
| intent | The intent of this example is to be offensive/disrespectful. |
| not_intent | The intent of this example is not to be offensive/disrespectful. |
#### biasframes_offensive
| label | hypothesis |
|:--------------|:-------------------------------------------------------------------------|
| not_offensive | This example could not be considered offensive, disrespectful, or toxic. |
| offensive | This example could be considered offensive, disrespectful, or toxic. |
#### financialphrasebank
| label | hypothesis |
|:---------|:--------------------------------------------------------------------------|
| negative | The sentiment in this example is negative from an investor's perspective. |
| neutral | The sentiment in this example is neutral from an investor's perspective. |
| positive | The sentiment in this example is positive from an investor's perspective. |
#### rottentomatoes
| label | hypothesis |
|:---------|:-----------------------------------------------------------------------|
| negative | The sentiment in this example rotten tomatoes movie review is negative |
| positive | The sentiment in this example rotten tomatoes movie review is positive |
#### amazonpolarity
| label | hypothesis |
|:---------|:----------------------------------------------------------------|
| negative | The sentiment in this example amazon product review is negative |
| positive | The sentiment in this example amazon product review is positive |
#### imdb
| label | hypothesis |
|:---------|:------------------------------------------------------------|
| negative | The sentiment in this example imdb movie review is negative |
| positive | The sentiment in this example imdb movie review is positive |
#### appreviews
| label | hypothesis |
|:---------|:------------------------------------------------------|
| negative | The sentiment in this example app review is negative. |
| positive | The sentiment in this example app review is positive. |
#### yelpreviews
| label | hypothesis |
|:---------|:-------------------------------------------------------|
| negative | The sentiment in this example yelp review is negative. |
| positive | The sentiment in this example yelp review is positive. |
#### wikitoxic_toxicaggregated
| label | hypothesis |
|:--------------------|:----------------------------------------------------------------|
| not_toxicaggregated | This example wikipedia comment does not contain toxic language. |
| toxicaggregated | This example wikipedia comment contains toxic language. |
#### wikitoxic_obscene
| label | hypothesis |
|:------------|:------------------------------------------------------------------|
| not_obscene | This example wikipedia comment does not contain obscene language. |
| obscene | This example wikipedia comment contains obscene language. |
#### wikitoxic_threat
| label | hypothesis |
|:-----------|:----------------------------------------------------------|
| not_threat | This example wikipedia comment does not contain a threat. |
| threat | This example wikipedia comment contains a threat. |
#### wikitoxic_insult
| label | hypothesis |
|:-----------|:-----------------------------------------------------------|
| insult | This example wikipedia comment contains an insult. |
| not_insult | This example wikipedia comment does not contain an insult. |
#### wikitoxic_identityhate
| label | hypothesis |
|:-----------------|:---------------------------------------------------------------|
| identityhate | This example wikipedia comment contains identity hate. |
| not_identityhate | This example wikipedia comment does not contain identity hate. |
#### hateoffensive
| label | hypothesis |
|:------------|:------------------------------------------------------------------------|
| hate_speech | This example tweet contains hate speech. |
| neither | This example tweet contains neither offensive language nor hate speech. |
| offensive | This example tweet contains offensive language without hate speech. |
#### hatexplain
| label | hypothesis |
|:------------|:-------------------------------------------------------------------------------------------|
| hate_speech | This example text from twitter or gab contains hate speech. |
| neither | This example text from twitter or gab contains neither offensive language nor hate speech. |
| offensive | This example text from twitter or gab contains offensive language without hate speech. |
#### spam
| label | hypothesis |
|:---------|:------------------------------|
| not_spam | This example sms is not spam. |
| spam | This example sms is spam. |
#### emotiondair
| label | hypothesis |
|:---------|:---------------------------------------------------|
| anger | This example tweet expresses the emotion: anger |
| fear | This example tweet expresses the emotion: fear |
| joy | This example tweet expresses the emotion: joy |
| love | This example tweet expresses the emotion: love |
| sadness | This example tweet expresses the emotion: sadness |
| surprise | This example tweet expresses the emotion: surprise |
#### emocontext
| label | hypothesis |
|:--------|:--------------------------------------------------------------------------------------|
| angry | This example tweet expresses the emotion: anger |
| happy | This example tweet expresses the emotion: happiness |
| others | This example tweet does not express any of the emotions: anger, sadness, or happiness |
| sad | This example tweet expresses the emotion: sadness |
#### empathetic
| label | hypothesis |
|:-------------|:-----------------------------------------------------------|
| afraid | The main emotion of this example dialogue is: afraid |
| angry | The main emotion of this example dialogue is: angry |
| annoyed | The main emotion of this example dialogue is: annoyed |
| anticipating | The main emotion of this example dialogue is: anticipating |
| anxious | The main emotion of this example dialogue is: anxious |
| apprehensive | The main emotion of this example dialogue is: apprehensive |
| ashamed | The main emotion of this example dialogue is: ashamed |
| caring | The main emotion of this example dialogue is: caring |
| confident | The main emotion of this example dialogue is: confident |
| content | The main emotion of this example dialogue is: content |
| devastated | The main emotion of this example dialogue is: devastated |
| disappointed | The main emotion of this example dialogue is: disappointed |
| disgusted | The main emotion of this example dialogue is: disgusted |
| embarrassed | The main emotion of this example dialogue is: embarrassed |
| excited | The main emotion of this example dialogue is: excited |
| faithful | The main emotion of this example dialogue is: faithful |
| furious | The main emotion of this example dialogue is: furious |
| grateful | The main emotion of this example dialogue is: grateful |
| guilty | The main emotion of this example dialogue is: guilty |
| hopeful | The main emotion of this example dialogue is: hopeful |
| impressed | The main emotion of this example dialogue is: impressed |
| jealous | The main emotion of this example dialogue is: jealous |
| joyful | The main emotion of this example dialogue is: joyful |
| lonely | The main emotion of this example dialogue is: lonely |
| nostalgic | The main emotion of this example dialogue is: nostalgic |
| prepared | The main emotion of this example dialogue is: prepared |
| proud | The main emotion of this example dialogue is: proud |
| sad | The main emotion of this example dialogue is: sad |
| sentimental | The main emotion of this example dialogue is: sentimental |
| surprised | The main emotion of this example dialogue is: surprised |
| terrified | The main emotion of this example dialogue is: terrified |
| trusting | The main emotion of this example dialogue is: trusting |
#### agnews
| label | hypothesis |
|:---------|:-------------------------------------------------------|
| Business | This example news text is about business news |
| Sci/Tech | This example news text is about science and technology |
| Sports | This example news text is about sports |
| World | This example news text is about world news |
#### yahootopics
| label | hypothesis |
|:-----------------------|:---------------------------------------------------------------------------------------------------|
| Business & Finance | This example question from the Yahoo Q&A forum is categorized in the topic: Business & Finance |
| Computers & Internet | This example question from the Yahoo Q&A forum is categorized in the topic: Computers & Internet |
| Education & Reference | This example question from the Yahoo Q&A forum is categorized in the topic: Education & Reference |
| Entertainment & Music | This example question from the Yahoo Q&A forum is categorized in the topic: Entertainment & Music |
| Family & Relationships | This example question from the Yahoo Q&A forum is categorized in the topic: Family & Relationships |
| Health | This example question from the Yahoo Q&A forum is categorized in the topic: Health |
| Politics & Government | This example question from the Yahoo Q&A forum is categorized in the topic: Politics & Government |
| Science & Mathematics | This example question from the Yahoo Q&A forum is categorized in the topic: Science & Mathematics |
| Society & Culture | This example question from the Yahoo Q&A forum is categorized in the topic: Society & Culture |
| Sports | This example question from the Yahoo Q&A forum is categorized in the topic: Sports |
#### massive
| label | hypothesis |
|:-------------------------|:------------------------------------------------------------------------------------------|
| alarm_query | The example utterance is a query about alarms. |
| alarm_remove | The intent of this example utterance is to remove an alarm. |
| alarm_set | The intent of the example utterance is to set an alarm. |
| audio_volume_down | The intent of the example utterance is to lower the volume. |
| audio_volume_mute | The intent of this example utterance is to mute the volume. |
| audio_volume_other | The example utterance is related to audio volume. |
| audio_volume_up | The intent of this example utterance is turning the audio volume up. |
| calendar_query | The example utterance is a query about a calendar. |
| calendar_remove | The intent of the example utterance is to remove something from a calendar. |
| calendar_set | The intent of this example utterance is to set something in a calendar. |
| cooking_query | The example utterance is a query about cooking. |
| cooking_recipe | This example utterance is about cooking recipies. |
| datetime_convert | The example utterance is related to date time changes or conversion. |
| datetime_query | The intent of this example utterance is a datetime query. |
| email_addcontact | The intent of this example utterance is adding an email address to contacts. |
| email_query | The example utterance is a query about emails. |
| email_querycontact | The intent of this example utterance is to query contact details. |
| email_sendemail | The intent of the example utterance is to send an email. |
| general_greet | This example utterance is a general greet. |
| general_joke | The intent of the example utterance is to hear a joke. |
| general_quirky | nan |
| iot_cleaning | The intent of the example utterance is for an IoT device to start cleaning. |
| iot_coffee | The intent of this example utterance is for an IoT device to make coffee. |
| iot_hue_lightchange | The intent of this example utterance is changing the light. |
| iot_hue_lightdim | The intent of the example utterance is to dim the lights. |
| iot_hue_lightoff | The example utterance is related to turning the lights off. |
| iot_hue_lighton | The example utterance is related to turning the lights on. |
| iot_hue_lightup | The intent of this example utterance is to brighten lights. |
| iot_wemo_off | The intent of this example utterance is turning an IoT device off. |
| iot_wemo_on | The intent of the example utterance is to turn an IoT device on. |
| lists_createoradd | The example utterance is related to creating or adding to lists. |
| lists_query | The example utterance is a query about a list. |
| lists_remove | The intent of this example utterance is to remove a list or remove something from a list. |
| music_dislikeness | The intent of this example utterance is signalling music dislike. |
| music_likeness | The example utterance is related to liking music. |
| music_query | The example utterance is a query about music. |
| music_settings | The intent of the example utterance is to change music settings. |
| news_query | The example utterance is a query about the news. |
| play_audiobook | The example utterance is related to playing audiobooks. |
| play_game | The intent of this example utterance is to start playing a game. |
| play_music | The intent of this example utterance is for an IoT device to play music. |
| play_podcasts | The example utterance is related to playing podcasts. |
| play_radio | The intent of the example utterance is to play something on the radio. |
| qa_currency | This example utteranceis about currencies. |
| qa_definition | The example utterance is a query about a definition. |
| qa_factoid | The example utterance is a factoid question. |
| qa_maths | The example utterance is a question about maths. |
| qa_stock | This example utterance is about stocks. |
| recommendation_events | This example utterance is about event recommendations. |
| recommendation_locations | The intent of this example utterance is receiving recommendations for good locations. |
| recommendation_movies | This example utterance is about movie recommendations. |
| social_post | The example utterance is about social media posts. |
| social_query | The example utterance is a query about a social network. |
| takeaway_order | The intent of this example utterance is to order takeaway food. |
| takeaway_query | This example utterance is about takeaway food. |
| transport_query | The example utterance is a query about transport or travels. |
| transport_taxi | The intent of this example utterance is to get a taxi. |
| transport_ticket | This example utterance is about transport tickets. |
| transport_traffic | This example utterance is about transport or traffic. |
| weather_query | This example utterance is a query about the wheather. |
#### banking77
| label | hypothesis |
|:-------------------------------------------------|:----------------------------------------------------------------------------------------------------------|
| Refund_not_showing_up | This customer example message is about a refund not showing up. |
| activate_my_card | This banking customer example message is about activating a card. |
| age_limit | This banking customer example message is related to age limits. |
| apple_pay_or_google_pay | This banking customer example message is about apple pay or google pay |
| atm_support | This banking customer example message requests ATM support. |
| automatic_top_up | This banking customer example message is about automatic top up. |
| balance_not_updated_after_bank_transfer | This banking customer example message is about a balance not updated after a transfer. |
| balance_not_updated_after_cheque_or_cash_deposit | This banking customer example message is about a balance not updated after a cheque or cash deposit. |
| beneficiary_not_allowed | This banking customer example message is related to a beneficiary not being allowed or a failed transfer. |
| cancel_transfer | This banking customer example message is related to the cancellation of a transfer. |
| card_about_to_expire | This banking customer example message is related to the expiration of a card. |
| card_acceptance | This banking customer example message is related to the scope of acceptance of a card. |
| card_arrival | This banking customer example message is about the arrival of a card. |
| card_delivery_estimate | This banking customer example message is about a card delivery estimate or timing. |
| card_linking | nan |
| card_not_working | This banking customer example message is about a card not working. |
| card_payment_fee_charged | This banking customer example message is about a card payment fee. |
| card_payment_not_recognised | This banking customer example message is about a payment the customer does not recognise. |
| card_payment_wrong_exchange_rate | This banking customer example message is about a wrong exchange rate. |
| card_swallowed | This banking customer example message is about a card swallowed by a machine. |
| cash_withdrawal_charge | This banking customer example message is about a cash withdrawal charge. |
| cash_withdrawal_not_recognised | This banking customer example message is about an unrecognised cash withdrawal. |
| change_pin | This banking customer example message is about changing a pin code. |
| compromised_card | This banking customer example message is about a compromised card. |
| contactless_not_working | This banking customer example message is about contactless not working |
| country_support | This banking customer example message is about country-specific support. |
| declined_card_payment | This banking customer example message is about a declined card payment. |
| declined_cash_withdrawal | This banking customer example message is about a declined cash withdrawal. |
| declined_transfer | This banking customer example message is about a declined transfer. |
| direct_debit_payment_not_recognised | This banking customer example message is about an unrecognised direct debit payment. |
| disposable_card_limits | This banking customer example message is about the limits of disposable cards. |
| edit_personal_details | This banking customer example message is about editing personal details. |
| exchange_charge | This banking customer example message is about exchange rate charges. |
| exchange_rate | This banking customer example message is about exchange rates. |
| exchange_via_app | nan |
| extra_charge_on_statement | This banking customer example message is about an extra charge. |
| failed_transfer | This banking customer example message is about a failed transfer. |
| fiat_currency_support | This banking customer example message is about fiat currency support |
| get_disposable_virtual_card | This banking customer example message is about getting a disposable virtual card. |
| get_physical_card | nan |
| getting_spare_card | This banking customer example message is about getting a spare card. |
| getting_virtual_card | This banking customer example message is about getting a virtual card. |
| lost_or_stolen_card | This banking customer example message is about a lost or stolen card. |
| lost_or_stolen_phone | This banking customer example message is about a lost or stolen phone. |
| order_physical_card | This banking customer example message is about ordering a card. |
| passcode_forgotten | This banking customer example message is about a forgotten passcode. |
| pending_card_payment | This banking customer example message is about a pending card payment. |
| pending_cash_withdrawal | This banking customer example message is about a pending cash withdrawal. |
| pending_top_up | This banking customer example message is about a pending top up. |
| pending_transfer | This banking customer example message is about a pending transfer. |
| pin_blocked | This banking customer example message is about a blocked pin. |
| receiving_money | This banking customer example message is about receiving money. |
| request_refund | This banking customer example message is about a refund request. |
| reverted_card_payment? | This banking customer example message is about reverting a card payment. |
| supported_cards_and_currencies | nan |
| terminate_account | This banking customer example message is about terminating an account. |
| top_up_by_bank_transfer_charge | nan |
| top_up_by_card_charge | This banking customer example message is about the charge for topping up by card. |
| top_up_by_cash_or_cheque | This banking customer example message is about topping up by cash or cheque. |
| top_up_failed | This banking customer example message is about top up issues or failures. |
| top_up_limits | This banking customer example message is about top up limitations. |
| top_up_reverted | This banking customer example message is about issues with topping up. |
| topping_up_by_card | This banking customer example message is about topping up by card. |
| transaction_charged_twice | This banking customer example message is about a transaction charged twice. |
| transfer_fee_charged | This banking customer example message is about an issue with a transfer fee charge. |
| transfer_into_account | This banking customer example message is about transfers into the customer's own account. |
| transfer_not_received_by_recipient | This banking customer example message is about a transfer that has not arrived yet. |
| transfer_timing | This banking customer example message is about transfer timing. |
| unable_to_verify_identity | This banking customer example message is about an issue with identity verification. |
| verify_my_identity | This banking customer example message is about identity verification. |
| verify_source_of_funds | This banking customer example message is about the source of funds. |
| verify_top_up | This banking customer example message is about verification and top ups |
| virtual_card_not_working | This banking customer example message is about a virtual card not working |
| visa_or_mastercard | This banking customer example message is about types of bank cards. |
| why_verify_identity | This banking customer example message questions why identity verification is necessary. |
| wrong_amount_of_cash_received | This banking customer example message is about a wrong amount of cash received. |
| wrong_exchange_rate_for_cash_withdrawal | This banking customer example message is about a wrong exchange rate for a cash withdrawal. |
#### trueteacher
| label | hypothesis |
|:-----------------------|:---------------------------------------------------------------------|
| factually_consistent | The example summary is factually consistent with the full article. |
| factually_inconsistent | The example summary is factually inconsistent with the full article. |
#### capsotu
| label | hypothesis |
|:----------------------|:----------------------------------------------------------------------------------------------------------|
| Agriculture | This example text from a US presidential speech is about agriculture |
| Civil Rights | This example text from a US presidential speech is about civil rights or minorities or civil liberties |
| Culture | This example text from a US presidential speech is about cultural policy |
| Defense | This example text from a US presidential speech is about defense or military |
| Domestic Commerce | This example text from a US presidential speech is about banking or finance or commerce |
| Education | This example text from a US presidential speech is about education |
| Energy | This example text from a US presidential speech is about energy or electricity or fossil fuels |
| Environment | This example text from a US presidential speech is about the environment or water or waste or pollution |
| Foreign Trade | This example text from a US presidential speech is about foreign trade |
| Government Operations | This example text from a US presidential speech is about government operations or administration |
| Health | This example text from a US presidential speech is about health |
| Housing | This example text from a US presidential speech is about community development or housing issues |
| Immigration | This example text from a US presidential speech is about migration |
| International Affairs | This example text from a US presidential speech is about international affairs or foreign aid |
| Labor | This example text from a US presidential speech is about employment or labour |
| Law and Crime | This example text from a US presidential speech is about law, crime or family issues |
| Macroeconomics | This example text from a US presidential speech is about macroeconomics |
| Public Lands | This example text from a US presidential speech is about public lands or water management |
| Social Welfare | This example text from a US presidential speech is about social welfare |
| Technology | This example text from a US presidential speech is about space or science or technology or communications |
| Transportation | This example text from a US presidential speech is about transportation |
#### manifesto
| label | hypothesis |
|:-------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Agriculture and Farmers: Positive | This example text from a political party manifesto is positive towards policies for agriculture and farmers |
| Anti-Growth Economy: Positive | This example text from a political party manifesto is in favour of anti-growth politics |
| Anti-Imperialism | This example text from a political party manifesto is anti-imperialistic, for example against controlling other countries and for greater self-government of colonies |
| Centralisation | This example text from a political party manifesto is in favour of political centralisation |
| Civic Mindedness: Positive | This example text from a political party manifesto is positive towards national solidarity, civil society or appeals for public spiritedness or against anti-social attitudes |
| Constitutionalism: Negative | This example text from a political party manifesto is positive towards constitutionalism |
| Constitutionalism: Positive | This example text from a political party manifesto is positive towards constitutionalism and the status quo of the constitution |
| Controlled Economy | This example text from a political party manifesto is supportive of direct government control of the economy, e.g. price control or minimum wages |
| Corporatism/Mixed Economy | This example text from a political party manifesto is positive towards cooperation of government, employers, and trade unions simultaneously |
| Culture: Positive | This example text from a political party manifesto is in favour of cultural policies or leisure facilities, for example museus, libraries or public sport clubs |
| Decentralization | This example text from a political party manifesto is for decentralisation or federalism |
| Democracy | This example text from a political party manifesto favourably mentions democracy or democratic procedures or institutions |
| Economic Goals | This example text from a political party manifesto is a broad/general statement on economic goals without specifics |
| Economic Growth: Positive | This example text from a political party manifesto is supportive of economic growth, for example facilitation of more production or government aid for growth |
| Economic Orthodoxy | This example text from a political party manifesto is for economic orthodoxy, for example reduction of budget deficits, thrift or a strong currency |
| Economic Planning | This example text from a political party manifesto is positive towards government economic planning, e.g. policy plans or strategies |
| Education Expansion | This example text from a political party manifesto is about the need to expand/improve policy on education |
| Education Limitation | This example text from a political party manifesto is sceptical towards state expenditure on education, for example in favour of study fees or private schools |
| Environmental Protection | This example text from a political party manifesto is in favour of environmental protection, e.g. fighting climate change or 'green' policies or preservation of natural resources or animal rights |
| Equality: Positive | This example text from a political party manifesto is positive towards equality or social justice, e.g. protection of underprivileged groups or fair distribution of resources |
| European Community/Union: Negative | This example text from a political party manifesto negatively mentions the EU or European Community |
| European Community/Union: Positive | This example text from a political party manifesto is positive towards the EU or European Community, for example EU expansion and integration |
| Foreign Special Relationships: Negative | This example text from a political party manifesto is negative towards particular countries |
| Foreign Special Relationships: Positive | This example text from a political party manifesto is positive towards particular countries |
| Free Market Economy | This example text from a political party manifesto is in favour of a free market economy and capitalism |
| Freedom and Human Rights | This example text from a political party manifesto is in favour of freedom and human rights, for example freedom of speech, assembly or against state coercion or for individualism |
| Governmental and Administrative Efficiency | This example text from a political party manifesto is in favour of efficiency in government/administration, for example by restructuring civil service or improving bureaucracy |
| Incentives: Positive | This example text from a political party manifesto is favourable towards supply side economic policies supporting businesses, for example for incentives like subsidies or tax breaks |
| Internationalism: Negative | This example text from a political party manifesto is sceptical of internationalism, for example negative towards international cooperation, in favour of national sovereignty and unilaterialism |
| Internationalism: Positive | This example text from a political party manifesto is in favour of international cooperation with other countries, for example mentions the need for aid to developing countries, or global governance |
| Keynesian Demand Management | This example text from a political party manifesto is for keynesian demand management and demand side economic policies |
| Labour Groups: Negative | This example text from a political party manifesto is negative towards labour groups and unions |
| Labour Groups: Positive | This example text from a political party manifesto is positive towards labour groups, for example for good working conditions, fair wages or unions |
| Law and Order: Positive | This example text from a political party manifesto is positive towards law and order and strict law enforcement |
| Market Regulation | This example text from a political party manifesto is supports market regulation for a fair and open market, for example for consumer protection or for increased competition or for social market economy |
| Marxist Analysis | This example text from a political party manifesto is positive towards Marxist-Leninist ideas or uses specific Marxist terminology |
| Middle Class and Professional Groups | This example text from a political party manifesto favourably references the middle class, e.g. white colar groups or the service sector |
| Military: Negative | This example text from a political party manifesto is negative towards the military, for example for decreasing military spending or disarmament |
| Military: Positive | This example text from a political party manifesto is positive towards the military, for example for military spending or rearmament or military treaty obligations |
| Multiculturalism: Negative | This example text from a political party manifesto is sceptical towards multiculturalism, or for cultural integration or appeals to cultural homogeneity in society |
| Multiculturalism: Positive | This example text from a political party manifesto favourably mentions cultural diversity, for example for freedom of religion or linguistic heritages |
| National Way of Life: Negative | This example text from a political party manifesto unfavourably mentions a country's nation and history, for example sceptical towards patriotism or national pride |
| National Way of Life: Positive | This example text from a political party manifesto is positive towards the national way of life and history, for example pride of citizenship or appeals to patriotism |
| Nationalisation | This example text from a political party manifesto is positive towards government ownership of industries or land or for economic nationalisation |
| Non-economic Demographic Groups | This example text from a political party manifesto favourably mentions non-economic demographic groups like women, students or specific age groups |
| Peace | This example text from a political party manifesto is positive towards peace and peaceful means of solving crises, for example in favour of negotiations and ending wars |
| Political Authority | This example text from a political party manifesto mentions the speaker's competence to govern or other party's lack of such competence, or favourably mentions a strong/stable government |
| Political Corruption | This example text from a political party manifesto is negative towards political corruption or abuse of political/bureaucratic power |
| Protectionism: Negative | This example text from a political party manifesto is negative towards protectionism, in favour of free trade |
| Protectionism: Positive | This example text from a political party manifesto is in favour of protectionism, for example tariffs, export subsidies |
| Technology and Infrastructure: Positive | This example text from a political party manifesto is about technology and infrastructure, e.g. the importance of modernisation of industry, or supportive of public spending on infrastructure/tech |
| Traditional Morality: Negative | This example text from a political party manifesto is negative towards traditional morality, for example against religious moral values, for divorce or abortion, for modern families or separation of church and state |
| Traditional Morality: Positive | This example text from a political party manifesto is favourable towards traditional or religious values, for example for censorship of immoral behavour, for traditional family values or religious institutions |
| Underprivileged Minority Groups | This example text from a political party manifesto favourably mentions underprivileged minorities, for example handicapped, homosexuals or immigrants |
| Welfare State Expansion | This example text from a political party manifesto is positive towards the welfare state, e.g. health care, pensions or social housing |
| Welfare State Limitation | This example text from a political party manifesto is for limiting the welfare state, for example public funding for social services or social security, e.g. private care before state care | |