|
--- |
|
library_name: transformers |
|
base_model: |
|
- answerdotai/ModernBERT-base |
|
license: apache-2.0 |
|
language: |
|
- en |
|
pipeline_tag: zero-shot-classification |
|
datasets: |
|
- nyu-mll/glue |
|
- facebook/anli |
|
tags: |
|
- instruct |
|
- natural-language-inference |
|
- nli |
|
--- |
|
|
|
# Model Card for Model ID |
|
|
|
ModernBERT multi-task fine-tuned on tasksource NLI tasks, including MNLI, ANLI, SICK, WANLI, doc-nli, LingNLI, FOLIO, FOL-NLI, LogicNLI, Label-NLI and all datasets in the below table). |
|
This is the equivalent of an "instruct" version. |
|
|
|
Test accuracy at 100k training steps. 215k steps version coming december 24th. |
|
|
|
| test_name | test_accuracy | |
|
|:-------------------------------------|----------------:| |
|
| glue/mnli | 0.91 | |
|
| glue/qnli | 0.93 | |
|
| glue/rte | 0.86 | |
|
| super_glue/cb | 0.89 | |
|
| anli/a1 | 0.62 | |
|
| anli/a2 | 0.47 | |
|
| anli/a3 | 0.42 | |
|
| sick/label | 0.92 | |
|
| sick/entailment_AB | 0.84 | |
|
| snli | 0.91 | |
|
| scitail/snli_format | 0.95 | |
|
| hans | 1 | |
|
| WANLI | 0.71 | |
|
| recast/recast_sentiment | 0.98 | |
|
| recast/recast_verbcorner | 0.94 | |
|
| recast/recast_ner | 0.87 | |
|
| recast/recast_factuality | 0.93 | |
|
| recast/recast_puns | 0.93 | |
|
| recast/recast_kg_relations | 0.94 | |
|
| recast/recast_verbnet | 0.88 | |
|
| recast/recast_megaveridicality | 0.87 | |
|
| probability_words_nli/usnli | 0.77 | |
|
| probability_words_nli/reasoning_1hop | 0.99 | |
|
| probability_words_nli/reasoning_2hop | 0.9 | |
|
| nan-nli | 0.85 | |
|
| nli_fever | 0.72 | |
|
| breaking_nli | 1 | |
|
| conj_nli | 0.71 | |
|
| fracas | 0.86 | |
|
| dialogue_nli | 0.88 | |
|
| mpe | 0.73 | |
|
| dnc | 0.9 | |
|
| recast_white/fnplus | 0.81 | |
|
| recast_white/sprl | 0.92 | |
|
| recast_white/dpr | 0.61 | |
|
| robust_nli/IS_CS | 0.76 | |
|
| robust_nli/LI_LI | 0.98 | |
|
| robust_nli/ST_WO | 0.85 | |
|
| robust_nli/PI_SP | 0.74 | |
|
| robust_nli/PI_CD | 0.8 | |
|
| robust_nli/ST_SE | 0.78 | |
|
| robust_nli/ST_NE | 0.86 | |
|
| robust_nli/ST_LM | 0.81 | |
|
| robust_nli_is_sd | 1 | |
|
| robust_nli_li_ts | 0.91 | |
|
| add_one_rte | 0.91 | |
|
| cycic_classification | 0.83 | |
|
| lingnli | 0.82 | |
|
| monotonicity-entailment | 0.95 | |
|
| scinli | 0.79 | |
|
| naturallogic | 0.91 | |
|
| syntactic-augmentation-nli | 0.95 | |
|
| autotnli | 0.92 | |
|
| defeasible-nli/atomic | 0.76 | |
|
| defeasible-nli/snli | 0.79 | |
|
| help-nli | 0.91 | |
|
| nli-veridicality-transitivity | 0.99 | |
|
| lonli | 0.99 | |
|
| dadc-limit-nli | 0.67 | |
|
| folio | 0.59 | |
|
| tomi-nli | 0.53 | |
|
| temporal-nli | 0.92 | |
|
| counterfactually-augmented-snli | 0.74 | |
|
| cnli | 0.81 | |
|
| logiqa-2.0-nli | 0.57 | |
|
| mindgames | 0.94 | |
|
| ConTRoL-nli | 0.65 | |
|
| logical-fallacy | 0.31 | |
|
| conceptrules_v2 | 0.99 | |
|
| zero-shot-label-nli | 0.74 | |
|
| scone | 0.97 | |
|
| monli | 0.98 | |
|
| SpaceNLI | 1 | |
|
| propsegment/nli | 0.91 | |
|
| SDOH-NLI | 1 | |
|
| scifact_entailment | 0.78 | |
|
| AdjectiveScaleProbe-nli | 0.99 | |
|
| resnli | 0.99 | |
|
| semantic_fragments_nli | 0.99 | |
|
| dataset_train_nli | 0.88 | |
|
| ruletaker | 0.91 | |
|
| PARARULE-Plus | 1 | |
|
| logical-entailment | 0.73 | |
|
| nope | 0.54 | |
|
| LogicNLI | 0.65 | |
|
| contract-nli/contractnli_a/seg | 0.87 | |
|
| contract-nli/contractnli_b/full | 0.78 | |
|
| nli4ct_semeval2024 | 0.6 | |
|
| biosift-nli | 0.88 | |
|
| SIGA-nli | 0.54 | |
|
| FOL-nli | 0.71 | |
|
| doc-nli | 0.82 | |
|
| mctest-nli | 0.89 | |
|
| idioms-nli | 0.86 | |
|
| lifecycle-entailment | 0.71 | |
|
| MSciNLI | 0.82 | |
|
| hover-3way/nli | 0.9 | |
|
| seahorse_summarization_evaluation | 0.82 | |
|
| babi_nli | 0.94 | |
|
| gen_debiased_nli | 0.9 | |
|
|
|
# Usage |
|
|
|
## [ZS] Zero-shot classification pipeline |
|
```python |
|
from transformers import pipeline |
|
classifier = pipeline("zero-shot-classification",model="tasksource/ModernBERT-base-nli") |
|
|
|
text = "one day I will see the world" |
|
candidate_labels = ['travel', 'cooking', 'dancing'] |
|
classifier(text, candidate_labels) |
|
``` |
|
NLI training data of this model includes [label-nli](https://huggingface.co/datasets/tasksource/zero-shot-label-nli), a NLI dataset specially constructed to improve this kind of zero-shot classification. |
|
|
|
## [NLI] Natural language inference pipeline |
|
|
|
```python |
|
from transformers import pipeline |
|
pipe = pipeline("text-classification",model="tasksource/ModernBERT-base-nli") |
|
pipe([dict(text='there is a cat', |
|
text_pair='there is a black cat')]) #list of (premise,hypothesis) |
|
``` |
|
|
|
## Backbone for further fune-tuning |
|
|
|
This checkpoint has stronger reasoning and fine-grained abilities than the base version and can be used for further fine-tuning. |
|
|
|
# Citation |
|
|
|
``` |
|
@inproceedings{sileo-2024-tasksource, |
|
title = "tasksource: A Large Collection of {NLP} tasks with a Structured Dataset Preprocessing Framework", |
|
author = "Sileo, Damien", |
|
booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)", |
|
month = may, |
|
year = "2024", |
|
address = "Torino, Italia", |
|
publisher = "ELRA and ICCL", |
|
url = "https://aclanthology.org/2024.lrec-main.1361", |
|
pages = "15655--15684", |
|
} |
|
``` |
