Learn
What is a tokenizer?
Last reviewed July 16, 2026
A tokenizer is the component that converts text into the sequence of token IDs a language model processes, and converts generated IDs back into text. It splits text into subword pieces from a fixed vocabulary learned during training. Because each model family trains its own vocabulary, the same text produces different token counts on different models, which changes both what fits in context and what a request costs.
How subword tokenization works
Modern tokenizers use byte-pair encoding or similar schemes: frequent character sequences merge into single tokens, rare ones split into pieces. A common word is one token; an unusual identifier may be four or five. This is why prose tokenizes efficiently while code, URLs, and non-English scripts consume more tokens for the same amount of text.
Why counts differ across models
Each model family ships its own vocabulary, so a prompt that is 1,000 tokens on one model can be 1,200 on another. Budgets, context fit, and cost comparisons should be computed with the target model's tokenizer, or with a stated approximation when the exact one is not published.
idapt's token counter uses exact tokenizers for model families that publish them and marks the rest as estimates with a tilde, so a count is never silently wrong.
Frequently asked
See it in practice
The fastest way to understand it is to use it. Start free, no credit card.
Try idapt free