ModernBERT-base / tokenizer_config.json

Commit History

Add `"add_prefix_space": true,`; this allows for much stronger token-level performance (e.g. NER, ColBERT)
f6df40d
verified

tomaarsen HF staff commited on

Update tokenizer: Set lstrip=True for [MASK]
51c9b61

Tom Aarsen commited on

Upload ModernBERT-base model
cb3183f

Tom Aarsen commited on