HuggingFaceTB

/

nanowhale-100m-base

like 17

Follow

Hugging Face Smol Models Research 3.69k

The bos token is defined but doesn't seem to be used in the training code? It's in the chat template of the final model though. Also the tokenizer doesn't add it by default.

So I'm assuming the bos_token was not used in training and should not be used in inference. But that sounds weird

gpt answer via github copilot in case it's useful

Probably yes, implicitly — but this repo does not manually prepend a BOS token in the base-model pretraining script.

Why I say that:

• The pretraining script uses trl.SFTTrainer on raw text with processing_class=tokenizer, and there is no explicit code that adds a BOS token before each sample.
• The tokenizer/config defines a BOS token:
  • bos_token_id=0 in the model config
  • bos_token is set in tokenizer/tokenizer_config.json
• So whether BOS is actually inserted during pretraining depends on the tokenizer/TR L tokenization path, not on custom repo code. In this repo, there is no custom BOS-prepend logic for base-model training.

Relevant spots:

def build_tokenizer(tokenizer_path="tokenizer"):
    """Load the DeepSeek-V4 tokenizer."""
    tok = PreTrainedTokenizerFast.from_pretrained(tokenizer_path)
    if tok.pad_token is None:
        tok.pad_token = tok.eos_token
    return tok

# SFT Config for pretraining (raw text, no chat template)
sft_config = SFTConfig(
    output_dir=output_dir,
    max_length=max_seq_length,
    packing=True,
    dataset_text_field="text",
    ...
)

trainer = SFTTrainer(
    model=model,
    args=sft_config,
    train_dataset=dataset,
    processing_class=tokenizer,
)

pad_token_id=None,
bos_token_id=0,
eos_token_id=1,
tie_word_embeddings=False,
rope_theta=10000.0,

{
  "backend": "tokenizers",
  "bos_token": "<｜begin▁of▁sentence｜>",
  "eos_token": "<｜end▁of▁sentence｜>",
  "model_max_length": 1000000000000000019884624838656,
  "pad_token": "<｜end▁of▁sentence｜>"
}

Bottom line:

• Repo code: no explicit BOS insertion for base pretraining.
• Tokenizer behavior: BOS token exists, so it may still be added automatically by the tokenizer pipeline/TR L defaults.

If you want, I can check the tokenizer files more closely to determine whether PreTrainedTokenizerFast here is configured to automatically add special tokens during encoding.