metadata

library_name: transformers
base_model:
  - answerdotai/ModernBERT-base
license: apache-2.0
language:
  - en
pipeline_tag: zero-shot-classification
datasets:
  - nyu-mll/glue
  - facebook/anli
tags:
  - instruct
  - natural-language-inference
  - nli

Model Card for Model ID

ModernBERT multi-task fine-tuned on tasksource NLI tasks, including MNLI, ANLI, SICK, WANLI, doc-nli, LingNLI, FOLIO, FOL-NLI, LogicNLI, Label-NLI and all datasets in the below table). This is the equivalent of an "instruct" version.

Test accuracy at 10k training steps (current version, 100k steps incoming at the end of the week).

test_name	test_accuracy
glue/mnli	0.82
glue/qnli	0.84
glue/rte	0.78
super_glue/cb	0.75
anli/a1	0.51
anli/a2	0.39
anli/a3	0.38
sick/label	0.91
sick/entailment_AB	0.81
snli	0.82
scitail/snli_format	0.94
hans	0.99
WANLI	0.7
recast/recast_ner	0.84
recast/recast_kg_relations	0.89
recast/recast_puns	0.78
recast/recast_verbcorner	0.87
recast/recast_sentiment	0.97
recast/recast_verbnet	0.74
recast/recast_factuality	0.88
recast/recast_megaveridicality	0.86
probability_words_nli/reasoning_2hop	0.76
probability_words_nli/reasoning_1hop	0.84
probability_words_nli/usnli	0.7
nan-nli	0.62
nli_fever	0.71
breaking_nli	0.98
conj_nli	0.66
fracas	0
dialogue_nli	0.84
mpe	0.69
dnc	0.81
recast_white/fnplus	0.6
recast_white/sprl	0.83
recast_white/dpr	0.57
robust_nli/IS_CS	0.45
robust_nli/LI_LI	0.92
robust_nli/ST_WO	0.66
robust_nli/PI_SP	0.53
robust_nli/PI_CD	0.54
robust_nli/ST_SE	0.58
robust_nli/ST_NE	0.52
robust_nli/ST_LM	0.47
robust_nli_is_sd	0.99
robust_nli_li_ts	0.81
add_one_rte	0.87
cycic_classification	0.62
lingnli	0.73
monotonicity-entailment	0.84
scinli	0.65
naturallogic	0.77
syntactic-augmentation-nli	0.87
autotnli	0.83
defeasible-nli/atomic	0.72
defeasible-nli/snli	0.67
help-nli	0.72
nli-veridicality-transitivity	0.92
lonli	0.88
dadc-limit-nli	0.59
folio	0.44
tomi-nli	0.52
temporal-nli	0.62
counterfactually-augmented-snli	0.69
cnli	0.71
logiqa-2.0-nli	0.51
mindgames	0.83
ConTRoL-nli	0.49
logical-fallacy	0.13
conceptrules_v2	0.97
zero-shot-label-nli	0.67
scone	0.79
monli	0.76
SpaceNLI	0.89
propsegment/nli	0.82
SDOH-NLI	0.98
scifact_entailment	0.52
AdjectiveScaleProbe-nli	0.91
resnli	0.97
semantic_fragments_nli	0.91
dataset_train_nli	0.81
ruletaker	0.69
PARARULE-Plus	1
logical-entailment	0.53
nope	0.36
LogicNLI	0.34
contract-nli/contractnli_a/seg	0.79
contract-nli/contractnli_b/full	0.67
nli4ct_semeval2024	0.53
biosift-nli	0.85
SIGA-nli	0.46
FOL-nli	0.49
doc-nli	0.81
mctest-nli	0.84
idioms-nli	0.77
lifecycle-entailment	0.57
MSciNLI	0.65
babi_nli	0.77
gen_debiased_nli	0.82

Usage

[ZS] Zero-shot classification pipeline

from transformers import pipeline
classifier = pipeline("zero-shot-classification",model="tasksource/ModernBERT-base-nli")

text = "one day I will see the world"
candidate_labels = ['travel', 'cooking', 'dancing']
classifier(text, candidate_labels)

NLI training data of this model includes label-nli, a NLI dataset specially constructed to improve this kind of zero-shot classification.

[NLI] Natural language inference pipeline

from transformers import pipeline
pipe = pipeline("text-classification",model="tasksource/ModernBERT-base-nli")
pipe([dict(text='there is a cat',
  text_pair='there is a black cat')]) #list of (premise,hypothesis)

Backbone for further fune-tuning

This checkpoint has stronger reasoning and fine-grained abilities than the base version and can be used for further fine-tuning.

Citation

@inproceedings{sileo-2024-tasksource,
    title = "tasksource: A Large Collection of {NLP} tasks with a Structured Dataset Preprocessing Framework",
    author = "Sileo, Damien",
    booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.lrec-main.1361",
    pages = "15655--15684",
}