Update README.md

README.md

@@ -1,71 +1,173 @@
 ---
-license: mit
+language:
+- en
 tags:
-- generated_from_trainer
-base_model: FacebookAI/roberta-large
-metrics:
-- accuracy
-model-index:
-- name: roberta-large-zeroshot-v2.0-2024-03-22-14-58
-  results: []
+- text-classification
+- zero-shot-classification
+pipeline_tag: zero-shot-classification
+library_name: transformers
+license: mit
+datasets:
+- nyu-mll/multi_nli
+- fever
 ---
 
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
-
-# roberta-large-zeroshot-v2.0-2024-03-22-14-58
-
-This model is a fine-tuned version of [FacebookAI/roberta-large](https://huggingface.co/FacebookAI/roberta-large) on an unknown dataset.
-It achieves the following results on the evaluation set:
-- Loss: 0.1357
-- F1 Macro: 0.5228
-- F1 Micro: 0.5294
-- Accuracy Balanced: 0.5577
-- Accuracy: 0.5294
-- Precision Macro: 0.6624
-- Recall Macro: 0.5577
-- Precision Micro: 0.5294
-- Recall Micro: 0.5294
-
-## Model description
-
-More information needed
-
-## Intended uses & limitations
-
-More information needed
-
-## Training and evaluation data
-
-More information needed
-
-## Training procedure
-
-### Training hyperparameters
-
-The following hyperparameters were used during training:
-- learning_rate: 9e-06
-- train_batch_size: 4
-- eval_batch_size: 32
-- seed: 42
-- gradient_accumulation_steps: 8
-- total_train_batch_size: 32
-- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
-- lr_scheduler_type: linear
-- lr_scheduler_warmup_ratio: 0.06
-- num_epochs: 2
+# Model description:  roberta-large-zeroshot-v2.0
+The model is designed for zero-shot classification with the Hugging Face pipeline. 
+
+The main advantage of this `zeroshot-v2.0` series of models is that they are trained on commercially-friendly data
+and are fully commercially usable, while my older `zeroshot-v1.1` models included training data with non-commercially licenses data. 
+An overview of the latest zeroshot classifiers with different sizes and licenses is available in my [Zeroshot Classifier Collection](https://huggingface.co/collections/MoritzLaurer/zeroshot-classifiers-6548b4ff407bb19ff5c3ad6f).
+
+The model can do one universal classification task: determine whether a hypothesis is "true" or "not true" given a text
+(`entailment` vs. `not_entailment`).  
+This task format is based on the Natural Language Inference task (NLI).
+The task is so universal that any classification task can be reformulated into this task.
+Note that compared to other NLI models, this model predicts two classes (`entailment` vs. `not_entailment`)
+as opposed to three classes (entailment/neutral/contradiction).
 
-### Training results
 
-| Training Loss | Epoch | Step  | Validation Loss | F1 Macro | F1 Micro | Accuracy Balanced | Accuracy | Precision Macro | Recall Macro | Precision Micro | Recall Micro |
-|:-------------:|:-----:|:-----:|:---------------:|:--------:|:--------:|:-----------------:|:--------:|:---------------:|:------------:|:---------------:|:------------:|
-| 0.2319        | 1.0   | 27331 | 0.6148          | 0.7746   | 0.7904   | 0.7810            | 0.7904   | 0.7704          | 0.7810       | 0.7904          | 0.7904       |
-| 0.1646        | 2.0   | 54662 | 0.6324          | 0.7822   | 0.7999   | 0.7847            | 0.7999   | 0.7800          | 0.7847       | 0.7999          | 0.7999       |
+## Training data
+The model is trained on two types of fully commercially-friendly data: 
+1. Synthetic data generated with [Mixtral-8x7B-Instruct-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1).
+I first created a list of 500+ diverse text classification tasks for 25 professions in conversations with Mistral-large. The data was manually curated.
+I then used this as seed data to generate several hundred thousand texts for the different tasks with Mixtral-8x7B-Instruct-v0.1.
+The final dataset used is available in the [synthetic_zeroshot_mixtral_v0.1](https://huggingface.co/datasets/MoritzLaurer/synthetic_zeroshot_mixtral_v0.1) dataset
+in the subset `mixtral_written_text_for_tasks_v4`. Data curation was done in multiple iterations and I will release more information on this process soon. 
+2. Two commercially-friendly NLI datasets: ([MNLI](https://huggingface.co/datasets/nyu-mll/multi_nli), [FEVER-NLI](https://huggingface.co/datasets/fever)).
+These datasets were added to increase generalization. Datasets like ANLI were excluded due to their non-commercial license. 
 
+The model was only trained on English data. I will release a multilingual version of this model soon.
+For __multilingual use-cases__, 
+I alternatively recommend machine translating texts to English with libraries like [EasyNMT](https://github.com/UKPLab/EasyNMT).
+English-only models tend to perform better than multilingual models and
+validation with English data can be easier if you don't speak all languages in your corpus.
+
+
+
+## How to use the model
+```python
+#!pip install transformers[sentencepiece]
+from transformers import pipeline
+text = "Angela Merkel is a politician in Germany and leader of the CDU"
+hypothesis_template = "This example is about {}"
+classes_verbalized = ["politics", "economy", "entertainment", "environment"]
+zeroshot_classifier = pipeline("zero-shot-classification", model="MoritzLaurer/roberta-large-zeroshot-v2.0")
+output = zeroshot_classifier(text, classes_verbalised, hypothesis_template=hypothesis_template, multi_label=False)
+print(output)
+```
+
+`multi_label=False` forces the model to decide on only one class. `multi_label=True` enables the model to choose multiple classes. 
+
+
+## Metrics 
+
+The model was evaluated on 28 different text classification tasks with the [balanced_accuracy](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.balanced_accuracy_score.html) metric.
+The main reference point is `facebook/bart-large-mnli` which is at the time of writing (27.03.24) the most used commercially-friendly 0-shot classifier.
+The different `zeroshot-v2.0` models were all trained with the same data and the only difference is the underlying foundation model. 
+
+![results_aggreg_v2.0](https://raw.githubusercontent.com/MoritzLaurer/zeroshot-classifier/e859471dd183ad44b705c047130433301386aab8/v2_synthetic_data/results/zeroshot-v2.0-aggreg.png)
+
+|                            |   facebook/bart-large-mnli |   roberta-base-zeroshot-v2.0 |   roberta-large-zeroshot-v2.0 |   deberta-v3-base-zeroshot-v2.0 |   deberta-v3-large-zeroshot-v2.0 |
+|:---------------------------|---------------------------:|-----------------------------:|------------------------------:|--------------------------------:|---------------------------------:|
+| all datasets mean          |                      0.566 |                        0.612 |                         0.65  |                           0.647 |                            0.697 |
+| amazonpolarity (2)         |                      0.937 |                        0.924 |                         0.951 |                           0.937 |                            0.952 |
+| imdb (2)                   |                      0.892 |                        0.871 |                         0.904 |                           0.893 |                            0.923 |
+| appreviews (2)             |                      0.934 |                        0.913 |                         0.937 |                           0.938 |                            0.943 |
+| yelpreviews (2)            |                      0.948 |                        0.953 |                         0.977 |                           0.979 |                            0.988 |
+| rottentomatoes (2)         |                      0.831 |                        0.803 |                         0.841 |                           0.841 |                            0.87  |
+| emotiondair (6)            |                      0.495 |                        0.523 |                         0.514 |                           0.487 |                            0.495 |
+| emocontext (4)             |                      0.605 |                        0.535 |                         0.609 |                           0.566 |                            0.687 |
+| empathetic (32)            |                      0.366 |                        0.386 |                         0.417 |                           0.388 |                            0.455 |
+| financialphrasebank (3)    |                      0.673 |                        0.521 |                         0.445 |                           0.678 |                            0.656 |
+| banking77 (72)             |                      0.327 |                        0.138 |                         0.297 |                           0.433 |                            0.542 |
+| massive (59)               |                      0.454 |                        0.481 |                         0.599 |                           0.533 |                            0.599 |
+| wikitoxic_toxicaggreg (2)  |                      0.609 |                        0.752 |                         0.768 |                           0.752 |                            0.751 |
+| wikitoxic_obscene (2)      |                      0.728 |                        0.818 |                         0.854 |                           0.853 |                            0.884 |
+| wikitoxic_threat (2)       |                      0.531 |                        0.796 |                         0.874 |                           0.861 |                            0.876 |
+| wikitoxic_insult (2)       |                      0.514 |                        0.738 |                         0.802 |                           0.768 |                            0.778 |
+| wikitoxic_identityhate (2) |                      0.567 |                        0.776 |                         0.801 |                           0.774 |                            0.801 |
+| hateoffensive (3)          |                      0.41  |                        0.497 |                         0.484 |                           0.539 |                            0.634 |
+| hatexplain (3)             |                      0.373 |                        0.423 |                         0.385 |                           0.441 |                            0.446 |
+| biasframes_offensive (2)   |                      0.499 |                        0.571 |                         0.587 |                           0.546 |                            0.648 |
+| biasframes_sex (2)         |                      0.503 |                        0.703 |                         0.845 |                           0.794 |                            0.877 |
+| biasframes_intent (2)      |                      0.635 |                        0.541 |                         0.635 |                           0.562 |                            0.696 |
+| agnews (4)                 |                      0.722 |                        0.765 |                         0.764 |                           0.694 |                            0.824 |
+| yahootopics (10)           |                      0.303 |                        0.55  |                         0.621 |                           0.575 |                            0.605 |
+| trueteacher (2)            |                      0.492 |                        0.488 |                         0.501 |                           0.505 |                            0.515 |
+| spam (2)                   |                      0.523 |                        0.537 |                         0.528 |                           0.531 |                            0.698 |
+| wellformedquery (2)        |                      0.528 |                        0.5   |                         0.5   |                           0.5   |                            0.476 |
+| manifesto (56)             |                      0.088 |                        0.111 |                         0.206 |                           0.198 |                            0.277 |
+| capsotu (21)               |                      0.375 |                        0.525 |                         0.558 |                           0.543 |                            0.631 |
+
+
+
+## When to use which model
+
+- deberta-v3-zeroshot vs. roberta-zeroshot: deberta-v3 performs clearly better than roberta, but it is slower.
+roberta is directly compatible with Hugging Face's production inference TEI containers and flash attention.
+These containers are a good choice for production use-cases. tl;dr: For accuracy, use a deberta-v3 model.
+If production inference speed is a concern, you can consider a roberta model (e.g. in a TEI container and [HF Inference Endpoints](https://ui.endpoints.huggingface.co/catalog)).
+- `zeroshot-v1.1` vs. `zeroshot-v2.0` models: My `zeroshot-v1.1` models (see [Zeroshot Classifier Collection](https://huggingface.co/collections/MoritzLaurer/zeroshot-classifiers-6548b4ff407bb19ff5c3ad6f)))
+perform better on these 28 datasets, but they are trained on several datasets with non-commercial licenses. 
+For commercial users, I therefore recommend using a v2.0 model and non-commercial users might get better performance with a v1.1 model.
+- The latest updates on new models are always available in the [Zeroshot Classifier Collection](https://huggingface.co/collections/MoritzLaurer/zeroshot-classifiers-6548b4ff407bb19ff5c3ad6f).
+
+
+## Reproduction
+
+Reproduction code is available in the `v2_synthetic_data` directory here: https://github.com/MoritzLaurer/zeroshot-classifier/tree/main
+
+
+
+## Limitations and bias
+The model can only do text classification tasks. 
+
+Biases can come from the underlying foundation model, the human NLI training data and the synthetic data generated by Mixtral.
+
+
+
+## License
+The foundation model (DeBERTa-v3) is published under the MIT license.
+The training data is released under different, permissive, commercially-friendly licenses 
+([MNLI](https://huggingface.co/datasets/nyu-mll/multi_nli), [FEVER-NLI](https://huggingface.co/datasets/fever), [synthetic_zeroshot_mixtral_v0.1](https://huggingface.co/datasets/MoritzLaurer/synthetic_zeroshot_mixtral_v0.1))
+
+## Citation
+
+This model is an extension of the research described in this [paper](https://arxiv.org/pdf/2312.17543.pdf).
+
+If you use this model academically, please cite: 
+```
+@misc{laurer_building_2023,
+	title = {Building {Efficient} {Universal} {Classifiers} with {Natural} {Language} {Inference}},
+	url = {http://arxiv.org/abs/2312.17543},
+	doi = {10.48550/arXiv.2312.17543},
+	abstract = {Generative Large Language Models (LLMs) have become the mainstream choice for fewshot and zeroshot learning thanks to the universality of text generation. Many users, however, do not need the broad capabilities of generative LLMs when they only want to automate a classification task. Smaller BERT-like models can also learn universal tasks, which allow them to do any text classification task without requiring fine-tuning (zeroshot classification) or to learn new tasks with only a few examples (fewshot), while being significantly more efficient than generative LLMs. This paper (1) explains how Natural Language Inference (NLI) can be used as a universal classification task that follows similar principles as instruction fine-tuning of generative LLMs, (2) provides a step-by-step guide with reusable Jupyter notebooks for building a universal classifier, and (3) shares the resulting universal classifier that is trained on 33 datasets with 389 diverse classes. Parts of the code we share has been used to train our older zeroshot classifiers that have been downloaded more than 55 million times via the Hugging Face Hub as of December 2023. Our new classifier improves zeroshot performance by 9.4\%.},
+	urldate = {2024-01-05},
+	publisher = {arXiv},
+	author = {Laurer, Moritz and van Atteveldt, Wouter and Casas, Andreu and Welbers, Kasper},
+	month = dec,
+	year = {2023},
+	note = {arXiv:2312.17543 [cs]},
+	keywords = {Computer Science - Artificial Intelligence, Computer Science - Computation and Language},
+}
+```
+
+### Ideas for cooperation or questions?
+If you have questions or ideas for cooperation, contact me at moritz{at}huggingface{dot}co or [LinkedIn](https://www.linkedin.com/in/moritz-laurer/)
 
-### Framework versions
 
-- Transformers 4.37.2
-- Pytorch 2.2.1+cu121
-- Datasets 2.17.1
-- Tokenizers 0.15.2
+
+### Flexible usage and "prompting"
+You can formulate your own hypotheses by changing the `hypothesis_template` of the zeroshot pipeline. 
+Similar to "prompt engineering" for LLMs, you can test different formulations of your `hypothesis_template` and verbalized classes to improve performance.
+
+```python
+from transformers import pipeline
+text = "Angela Merkel is a politician in Germany and leader of the CDU"
+hypothesis_template = "Merkel is the leader of the party: {}"
+classes_verbalized = ["CDU", "SPD", "Greens"]
+zeroshot_classifier = pipeline("zero-shot-classification", model="MoritzLaurer/roberta-large-zeroshot-v2.0")
+output = zeroshot_classifier(text, classes_verbalized, hypothesis_template=hypothesis_template, multi_label=False)
+print(output)
+```