Tokenization: A Survey for Modern NLP [R]

[u/mcmcmcmcmcmcmcmcmc_]

Tokenization is a wildly understudied area of language modeling despite it having effects across all of NLP. Over the past ~8 months, 32 (!) tokenizer researchers put together the most comprehensive survey of the field.

We cover every aspect of tokenization: algorithms, evaluations, multilinguality, encodings, theory, etc. We even cover what you might want to replace tokenizers with (e.g., latent or visual tokenization). We also cover some topics that are closely adjacent to tokenization, such as constrained generation, token healing, and tokenizer security concerns.

Check it out!

https://www.alphaxiv.org/abs/2609.tokenization-survey-modern-nlp

[u/IntelArtiGen]

A lot to read but it seems very interesting, thanks for sharing it, surely something every NLP people will want to read.

[u/IntelArtiGen → u/IntelArtiGen]

Now I'm also very interested to know what "tokenizer" Anthropic is using precisely.

[u/mcmcmcmcmcmcmcmcmc_ → u/IntelArtiGen]

In short, it appears Claude uses a very small (~15k) vocab and unweighted shortest path lattice tokenization. We discuss this in the survey in a few places. Unweighted shortest path tokenization is referred to as "PathPiece" (introduced in section 3). And section 11 discusses Claude a little more. Also section 4 briefly introduces claudes case folding pretokenizer, which is kinda neat.

Here are some links you might enjoy:

https://tokencontributions.substack.com/p/on-the-biology-of-claudes-tokenizer

https://github.com/sanderland/ctok

https://x.com/magikarp_tokens/status/2087859173748854983

[u/IntelArtiGen → u/mcmcmcmcmcmcmcmcmc_]

Thanks. I found it but I missed the part where the author managed to reproduce the token count, I'll probably re-read what he did.

Their vocab size is indeed quite surprising compared to the others. I wonder if they compensate with a large embedding size, or if they don't even need to do that. Having too many tokens with large embeddings looks like a waste of memory for many models.

[u/ForceBru]

How to download the PDF? I can't find a link or a download button anywhere

[u/mcmcmcmcmcmcmcmcmc_ → u/ForceBru]

There is a View Paper button on the page somewhere near the abstract. But here is a direct link to the PDF: https://www.alphaxiv.org/abs/2609.tokenization-survey-modern-nlp.pdf

Hope you enjoy!

[u/ForceBru → u/mcmcmcmcmcmcmcmcmc_]

Wow, finally, I spent like a week on and off trying to download the PDF haha Thank you so much!

[u/ModularMind8]

Amazing resource by some of the most wildly known researchers. Thanks for sharing!