metadata

library_name: transformers
base_model:
  - answerdotai/ModernBERT-base
license: apache-2.0
language:
  - en
pipeline_tag: zero-shot-classification
datasets:
  - nyu-mll/glue
  - facebook/anli
tags:
  - instruct
  - natural-language-inference
  - nli

Model Card for Model ID

ModernBERT mul-itask fine-tuned on tasksource NLI tasks, including MNLI, ANLI, SICK, WANLI, doc-nli, LingNLI, FOLIO, FOL-NLI, LogicNLI, Label-NLI...).

Test accuracy at 10k training steps (current version, 100k steps incoming at the end of the week).

test_name	test_accuracy
glue/mnli	0.82
glue/qnli	0.84
glue/rte	0.78
super_glue/cb	0.75
anli/a1	0.51
anli/a2	0.39
anli/a3	0.38
sick/label	0.91
sick/entailment_AB	0.81
snli	0.82
scitail/snli_format	0.94
hans	0.99
WANLI	0.7
recast/recast_ner	0.84
recast/recast_kg_relations	0.89
recast/recast_puns	0.78
recast/recast_verbcorner	0.87
recast/recast_sentiment	0.97
recast/recast_verbnet	0.74
recast/recast_factuality	0.88
recast/recast_megaveridicality	0.86
probability_words_nli/reasoning_2hop	0.76
probability_words_nli/reasoning_1hop	0.84
probability_words_nli/usnli	0.7
nan-nli	0.62
nli_fever	0.71
breaking_nli	0.98
conj_nli	0.66
fracas	0
dialogue_nli	0.84
mpe	0.69
dnc	0.81
recast_white/fnplus	0.6
recast_white/sprl	0.83
recast_white/dpr	0.57
robust_nli/IS_CS	0.45
robust_nli/LI_LI	0.92
robust_nli/ST_WO	0.66
robust_nli/PI_SP	0.53
robust_nli/PI_CD	0.54
robust_nli/ST_SE	0.58
robust_nli/ST_NE	0.52
robust_nli/ST_LM	0.47
robust_nli_is_sd	0.99
robust_nli_li_ts	0.81
add_one_rte	0.87
cycic_classification	0.62
lingnli	0.73
monotonicity-entailment	0.84
scinli	0.65
naturallogic	0.77
syntactic-augmentation-nli	0.87
autotnli	0.83
defeasible-nli/atomic	0.72
defeasible-nli/snli	0.67
help-nli	0.72
nli-veridicality-transitivity	0.92
lonli	0.88
dadc-limit-nli	0.59
folio	0.44
tomi-nli	0.52
temporal-nli	0.62
counterfactually-augmented-snli	0.69
cnli	0.71
logiqa-2.0-nli	0.51
mindgames	0.83
ConTRoL-nli	0.49
logical-fallacy	0.13
conceptrules_v2	0.97
zero-shot-label-nli	0.67
scone	0.79
monli	0.76
SpaceNLI	0.89
propsegment/nli	0.82
SDOH-NLI	0.98
scifact_entailment	0.52
AdjectiveScaleProbe-nli	0.91
resnli	0.97
semantic_fragments_nli	0.91
dataset_train_nli	0.81
ruletaker	0.69
PARARULE-Plus	1
logical-entailment	0.53
nope	0.36
LogicNLI	0.34
contract-nli/contractnli_a/seg	0.79
contract-nli/contractnli_b/full	0.67
nli4ct_semeval2024	0.53
biosift-nli	0.85
SIGA-nli	0.46
FOL-nli	0.49
doc-nli	0.81
mctest-nli	0.84
idioms-nli	0.77
lifecycle-entailment	0.57
MSciNLI	0.65
babi_nli	0.77
gen_debiased_nli	0.82

Usage

[ZS] Zero-shot classification pipeline

from transformers import pipeline
classifier = pipeline("zero-shot-classification",model="tasksource/ModernBERT-base-nli")

text = "one day I will see the world"
candidate_labels = ['travel', 'cooking', 'dancing']
classifier(text, candidate_labels)

NLI training data of this model includes label-nli, a NLI dataset specially constructed to improve this kind of zero-shot classification.

[NLI] Natural language inference pipeline

from transformers import pipeline
pipe = pipeline("text-classification",model="tasksource/ModernBERT-base-nli")
pipe([dict(text='there is a cat',
  text_pair='there is a black cat')]) #list of (premise,hypothesis)

Citation

@inproceedings{sileo-2024-tasksource,
    title = "tasksource: A Large Collection of {NLP} tasks with a Structured Dataset Preprocessing Framework",
    author = "Sileo, Damien",
    booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.lrec-main.1361",
    pages = "15655--15684",
}