Tokenization: how text becomes tokens.
A model does not read letters or words the way a person does. Before any text reaches it, the text is chopped into small pieces called tokens. Here is how that works, in plain language.
Tokenization is the very first step in everything a language model does: text in, a sequence of tokens out. We walk it the way the field is built. What a token is, why a fixed word list does not work, how the common recipes split text, what good tokenization unlocks, and where the idea is heading. The papers behind each idea are linked, so you can read the source yourself.
A token is a small piece of text from a fixed list.
That fixed list is called the vocabulary. The model only ever sees sequences of these pieces, each mapped to an integer id.
Tokens are not the same as words. A token can be a whole short word, a fragment of a longer word, a single character, a space, or a piece of punctuation. The word "pricing" might be one token, while "renegotiation" might be three or four. Turning raw text into a sequence of these pieces is tokenization, and at the end the reverse step, detokenization, turns the ids back into readable text.
Take a line of sales notes as a running example: "Renegotiate the Q3 contract with Acme." Watch how common words stay whole while names and rarer words get built from familiar parts. The same fallback is why a model can read a CRM field full of unusual company names or part numbers without choking. In the worst case it spells a word out from very small pieces, so nothing is ever truly unknown.
␣ marks the space before “Acme”.
- Look up each token in a vocabulary
Each piece maps to a numerical token ID.
- Read the corresponding embedding
The model works with vectors for those token IDs.
Why a fixed word list does not work.
The real problem is a trade-off. Bigger pieces make shorter, cheaper sequences but break on rare input. Smaller pieces are robust but make sequences long.
| Unit | Typical trade-off |
|---|---|
| Character or byte | Broad coverage, but longer sequences. |
| Subword piece | Balances coverage and sequence length. |
| Whole word | Shorter sequences for known words; brittle with an unknown vocabulary. |
As the Mielke et al. survey frames it: tokenization is a spectrum. Subword pieces sit between single characters and whole words.
Language has effectively unlimited words.
New names, typos, hyphenated compounds, account codes, emoji, and dozens of human languages mean any fixed word list will meet words it never saw. A pure word vocabulary forces an "unknown" token that throws information away.
Characters alone are too small.
Splitting into single characters never hits an unknown word, but it makes sequences very long. Long sequences cost more compute and make learning harder, so pure character splitting trades one problem for another.
Many scripts, no guaranteed spaces.
A method that assumes spaces between words breaks on languages that do not use them, and on mixed scripts. A general tokenizer cannot assume the words arrive pre-split.
The same text can split many ways.
For a given vocabulary there is often more than one valid segmentation of a word, and the choice affects what the model learns. Some methods make this explicit and even sample alternatives during training.
Tokens are the unit of counting.
Context windows, throughput, and usage-based pricing are all measured in tokens, not words or characters. A method that produces more tokens for the same text raises cost and can disadvantage some languages.
So subwords sit in the middle.
The field's answer is to work with subword pieces. Common words get their own single token, rarer words are composed from familiar parts, and very small pieces are always available as a fallback.
How the common recipes build a vocabulary.
A handful of methods dominate. They answer the same question in different ways: which pieces should be in the list, and how should a word be split into them?
1Byte-Pair Encoding
Merge the frequent pairs.
Byte-Pair Encoding starts from characters, then repeatedly finds the most frequent adjacent pair in the training corpus and merges it into a new token, recording each merge as a rule. It repeats until the vocabulary reaches a target size. Common sequences like "ing," "tion," and "pric" become single tokens, while rare words remain a few pieces.
It was introduced for open-vocabulary translation by Sennrich, Haddow, and Birch (2016), who adapted a data-compression algorithm from the 1990s to language work.
Starting symbols
Apply merge l + o
Apply later merges
2WordPiece
Merge by likelihood, not raw count.
WordPiece is a close cousin of BPE that grew out of work on Japanese and Korean voice search by Schuster and Nakajima (2012). Instead of merging the most frequent pair, it merges the pair that most increases the likelihood of the training data under a simple model. In practice it behaves much like BPE, with a different scoring rule for which merge to make next.
BPE
- Common formulation
- Select a frequent adjacent symbol pair.
WordPiece
- Training idea
- Select pieces that improve a likelihood-based objective.
3Unigram language model
Start big, then prune down.
Kudo (2018) flips the direction. Begin with a large candidate set of pieces, assign each a probability, and iteratively remove the pieces that hurt the corpus likelihood least, until the vocabulary reaches its target size. At inference it picks a high-probability segmentation.
Because it is probabilistic, it can also sample alternative segmentations during training, an idea called subword regularization. That makes a model more robust to how a given word happens to be split, which matters for the noisy, free-text input found in call transcripts and CRM notes.
- Start large
Create many possible subword pieces.
- Score pieces
Estimate their contribution to the corpus likelihood.
- Prune
Remove less useful candidates.
- Repeat
Reach the desired vocabulary size.
4SentencePiece
Tokenize raw text, no pre-splitting.
Kudo and Richardson (2018) packaged BPE and the unigram model into a tool that trains directly on raw sentences and treats the space as a normal symbol, often shown as a visible marker. This removes the assumption that words are already separated by spaces, so the same method works across languages that do and do not use them. Detokenization is exact and reversible, which is what lets the model's output be written back out as precise text.
contract with Acme
Illustrative segmentation. The ▁ marker represents a word boundary or space convention.
5Byte-level BPE
Never meet an unknown character.
Radford et al. (2019, the GPT-2 work) run BPE over raw bytes rather than Unicode characters. The base vocabulary is just the 256 byte values, so any character in any language, plus emoji and odd symbols, can always be represented, with no "unknown" token ever needed. This is a common backbone for modern large models.
- Unicode text
Encode characters as bytes.
- Byte vocabulary
All 256 byte values are representable.
- Merge common sequences
Use learned multi-byte pieces where available.
6Putting it together
A model and its tokenizer are a matched pair.
Train a vocabulary once on a large corpus. At run time, split incoming text into the known pieces, map those pieces to integer ids, and feed the model. To read the output back, reverse the map. The same vocabulary has to be used on both ends, which is why a model and its tokenizer travel together as a set.
- Encode
Map input text to IDs in the model’s vocabulary.
- Model
Read and predict IDs using that vocabulary.
- Decode
Map the predicted IDs back to text.
What good tokenization unlocks.
At the field level: an open vocabulary, one model across many languages, predictable cost, and robustness to messy real-world input.
Tokens matter beyond the plumbing, because they are the unit almost everything is counted in. A handy rule of thumb for English is that a token is roughly four characters, so a paragraph of meeting notes is a few hundred tokens. That rule does not hold across languages or for code, which is one reason tokenization choices have real consequences for cost and fairness.
- 01Context
Capacity is counted in tokens.
- 02Throughput
Generation rate is counted in tokens.
- 03Usage
Some services bill by token usage.
- 04Variation
Characters per token depend on language and tokenizer.
Context, throughput, and price are all measured in tokens, not words or characters.
An open vocabulary.
A model can process words it never saw in training, including new company names, product codes, and misspellings, by composing them from familiar pieces instead of dropping them.
One model, many languages.
Byte-level and SentencePiece methods let a single model read mixed languages and unusual symbols without a separate word list per language.
Predictable cost.
Subword pieces keep sequences far shorter than character-level splitting, which lowers compute. Because tokens are the billing and context unit, a sensible tokenizer keeps usage cost and context limits predictable.
A clean, reversible interface.
Exact detokenization means what a model emits can be turned back into precise text. As an illustration, that matters when an output has to be written into a structured CRM field rather than just shown on screen.
The sales and CRM references here, like keeping one client's data isolated from another's, or adapting a vocabulary to how a company sells, are illustrations of why tokenization matters. They describe the field, not the internals of any AI REV system.
Where the idea is heading.
We describe the direction of the field, not a roadmap of our own.
Fairer tokenization across languages.
The same sentence can take far more tokens in some languages than in English, which raises cost and shrinks effective context for those users. Active work aims to narrow that gap, including morphology-aware byte encodings.
Tokenizer-free and byte models.
A research thread tries to remove the separate tokenizer and let models operate on raw bytes or characters directly, learning their own grouping inside the model, trading tokenizer brittleness for more compute.
Domain-aware vocabularies.
Vocabularies tuned for code, math, or a specific domain can shorten sequences and improve quality there. As an illustration, an enterprise corpus heavy in product names and account codes tokenizes differently from general prose. The vocabulary is a design choice: what works well for one domain may split another domain’s language less efficiently.
Tokens as a first-class unit.
Because context windows, throughput, and pricing are all denominated in tokens, the unit itself is becoming a concept that buyers and builders reason about directly, not just an implementation detail.
The papers behind this page.
- Sennrich, Haddow, Birch (2016). Neural Machine Translation of Rare Words with Subword Units (BPE). ACL Anthology, P16-1162.
- Schuster, Nakajima (2012). Japanese and Korean Voice Search (WordPiece). IEEE ICASSP 2012.
- Kudo (2018). Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates (unigram model). ACL Anthology, P18-1007.
- Kudo, Richardson (2018). SentencePiece: A Simple and Language Independent Subword Tokenizer and Detokenizer for Neural Text Processing. ACL Anthology, D18-2012.
- Radford, Wu, Child, Luan, Amodei, Sutskever (2019). Language Models are Unsupervised Multitask Learners (GPT-2, byte-level BPE).
- Mielke et al. (2021). Between Words and Characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP. arXiv:2112.10508.
Keep reading.
Sibling explainers in the Research section.
Talk. Done.
Tokens are the first step in a chain that ends with clean, complete notes. See the whole thing on a real call.
No app required. Just your phone number.