Tokenization

Common Tokenizers

BPE - Byte Pair Encoding

• Understanding Multilingual Token Compression in GPT-o Family Models | njkumarr

WordPiece

SentPiece

TokenMonster

• GitHub - alasdairforsythe/tokenmonster: Ungreedy subword tokenizer and vocabulary trainer for Python, Go & Javascript

Over-Tokenization

• [2501.16975] Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling

Length-MAX

• [2511.20849] Length-MAX Tokenizer for Language Models

Getting Rid of Tokenization

Byte Latent Transformer

• Byte Latent Transformer: Patches Scale Better Than Tokens | Research - AI at Meta

• [2401.13660] MambaByte: Token-free Selective State Space Model

• EvaByte: Efficient Byte-level Language Models at Scale

Links

• ChatGPT

• Tiktoken with ARM64 SIMD | Alpin’s Blog

• SentencePiece with ARM64 SIMD | Alpin’s Blog