|
--- |
|
|
|
language: de |
|
|
|
tags: |
|
|
|
- text-classification |
|
|
|
- pytorch |
|
|
|
- nli |
|
|
|
- de |
|
|
|
|
|
pipeline_tag: zero-shot-classification |
|
|
|
widget: |
|
|
|
- text: "Ich habe ein Problem mit meinem Iphone das so schnell wie möglich gelöst werden muss." |
|
|
|
candidate_labels: "Computer, Handy, Tablet, dringend, nicht dringend" |
|
|
|
hypothesis_template: "In diesem Satz geht es um das Thema {}." |
|
|
|
--- |
|
|
|
|
|
# SVALabs - Gbert Large Zeroshot Nli |
|
|
|
In this repository, we present our German zeroshot classification model. |
|
|
|
This model was trained on the basis of the German BERT large model from [deepset.ai](https://huggingface.co/deepset/gbert-large) and finetuned for natural language inference based on 847.862 machine-translated nli sentence pairs, using the [mnli](https://huggingface.co/datasets/multi_nli), [anli](https://huggingface.co/datasets/anli) and [snli](https://huggingface.co/datasets/snli) datasets. For this purpose, we translated the sentence pairs in these datasets to German. |
|
|
|
If you are a German speaker you may also have a look at our [Blog post](https://focus.sva.de/zeroshot-klassifikation/) about this model and about Zeroshot Classification. |
|
|
|
### Model Details |
|
|
|
| | Description or Link | |
|
|---|---| |
|
|**Base model** | [```gbert-large```](https://huggingface.co/deepset/gbert-large) | |
|
|**Finetuning task**| Text Pair Classification / Natural Language Inference | |
|
|**Source datasets**| [```mnli```](https://huggingface.co/datasets/multi_nli); [```anli```](https://huggingface.co/datasets/anli); [```snli```](https://huggingface.co/datasets/snli) | |
|
|
|
### Performance |
|
|
|
We evaluated our model for the nli task using the TEST set of the German part of the [xnli](https://huggingface.co/datasets/xnli) dataset. |
|
|
|
XNLI TEST-Set Accuracy: 85.6% |
|
|
|
|
|
### Zeroshot Text Classification Task Benchmark |
|
|
|
We further tested our model for a zeroshot text classification task using a part of the [10kGNAD Dataset](https://tblock.github.io/10kGNAD/). |
|
Specifically, we used all articles that were labeled "Kultur", "Sport", "Web", "Wirtschaft" and "Wissenschaft". |
|
|
|
The next table shows the results as well as a comparison with other German language and multilanguage zeroshot options performing the same task: |
|
|
|
| Model | Accuracy | |
|
|:-------------------:|:------:| |
|
| Svalabs/gbert-large-zeroshot-nli | 0.81 | |
|
| Sahajtomar/German_Zeroshot | 0.76 | |
|
| Symanto/xlm-roberta-base-snli-mnli-anli-xnli | 0.16 | |
|
| Deepset/gbert-base | 0.65 | |
|
|
|
|
|
### How to use |
|
|
|
The simplest way to use the model is the huggingface transformers pipeline tool. |
|
Just initialize the pipeline specifying the task as "zero-shot-classification" |
|
and select "svalabs/gbert-large-zeroshot-nli" as model. |
|
|
|
The model requires you to specify labels, |
|
a sequence (or list of sequences) to classify and a hypothesis template. |
|
In our tests, if the labels comprise only single words, |
|
"In diesem Satz geht es um das Thema {}" performed the best. |
|
|
|
However, for multiple words, especially when they combine nouns and verbs, |
|
simple hypothesis such as "Weil {}" or "Daher {}" may work better. |
|
|
|
Here is an example of how to use the model: |
|
|
|
```python |
|
|
|
from transformers import pipeline |
|
|
|
zershot_pipeline = pipeline("zero-shot-classification", |
|
model="svalabs/gbert-large-zeroshot-nli") |
|
|
|
sequence = "Ich habe ein Problem mit meinem Iphone das so schnell wie möglich gelöst werden muss" |
|
labels = ["Computer", "Handy", "Tablet", "dringend", "nicht dringend"] |
|
hypothesis_template = "In diesem Satz geht es um das Thema {}." |
|
|
|
|
|
zershot_pipeline(sequence, labels, hypothesis_template=hypothesis_template) |
|
|
|
``` |
|
|
|
### Contact |
|
- Daniel Ehnes, daniel.ehnes@sva.de |
|
- Baran Avinc, baran.avinc@sva.de |
|
|
