ArlowGPT Tokenizer

Overview

The ArlowGPT Tokenizer is a byte pair encoding (BPE) tokenizer developed from scratch, optimized for large-scale language modeling and text generation tasks. It features a vocabulary size of 59,575 tokens and supports a maximum context length of 131,072 tokens, making it suitable for handling extremely long documents and sequences.

Key Features

Vocabulary Size: 59,575 tokens
Maximum Context Length: 131,072 tokens
Tokenizer Type: Byte Pair Encoding (BPE)
Special Tokens:
- <pad>: Padding token used for sequence alignment.
- <mask>: Special token for masked language modeling tasks.
- <eos>: End-of-sequence token.
- <bos>: Beginning-of-sequence token.
Trained From Scratch: The tokenizer was trained from scratch using a large corpus of English and multilingual text.

Training Data

The tokenizer was trained on Wikipedia, ensuring high coverage of general knowledge and domain-specific terms. Although primarily optimized for English, it also includes some multilingual capability due to the nature of the training dataset.

Intended Use Cases

This tokenizer is designed for general-purpose language modeling and is suitable for tasks such as:

Autoregressive text generation
Long-context summarization
Conversational AI
Information retrieval over large documents
General NLP tasks requiring long context processing

Supported Languages

Primary Language: English
Secondary Support: Some multilingual content

Performance & Benchmarks

No formal benchmarks have been conducted yet, but the tokenizer has been designed for efficiency in both tokenization speed and memory usage, with a focus on handling extremely long contexts up to 131,072 tokens.

Limitations

Multilingual Coverage: While the tokenizer includes some multilingual tokens, it is primarily optimized for English text, and performance on non-English languages may vary.
No Benchmarked Metrics: The tokenizer has not undergone formal benchmarking for speed or performance across various tasks.

Citation

If you use the ArlowGPT Tokenizer in your work, please cite it as:

@misc{arlowgpt_tokenizer,
  title={ArlowGPT Tokenizer},
  author={yuchenxie},
  year={2025},
  howpublished={\url{https://huggingface.co/yuchenxie/ArlowGPT-Tokenizer}}
}