Tokens, simply
- foundations
- costs and limits
A token is the unit of text a model actually processes — a small chunk, roughly a word piece. “Cat” is usually one token; “internationalization” may be several; a single emoji can be two or more. Models never see your letters the way you do: everything is converted to tokens first.
How the pieces are cut
Tokenizers split text according to patterns learned from data: common words stay whole, rarer words get sliced into fragments (international + ization), and unusual characters — accents, code punctuation, emojis — often cost extra pieces.
For everyday English, a rule of thumb is about ¾ of a word per token — roughly 100 tokens per 75 words. It is only an estimate: the count depends on the model, the language (many non-English languages cost more) and the content itself. Code and tables are token-hungry.
Why it matters
Tokens are the currency of everything around the model:
- Limits are counted in tokens, not pages. A “10-page document” is not a fixed amount — dense, code-heavy text fills the context window much faster.
- Costs scale with tokens — input and output both. Verbose prompts and rambling answers are not free.
- Overflow means truncation. When input exceeds the window, the oldest content is dropped first — this is the mechanical reason long chats “forget”.
A bad example
Assume: “A page is about 500 words, so 200 pages is 100,000 words — should fit.”
If the limit is counted in tokens and the document is dense (tables, code, another language), the real count can be far higher. The system may silently drop the beginning — and the model answers from what remains, confidently.
A better example
Ask: “Summarize this report section by section. After each section, give me three bullets I can carry into a new chat.”
Long work becomes a series of short, verifiable steps — and the carried bullets become your compressed context.
Why it works
Once you know the unit, the tactics follow:
- Compress instead of hoarding. Carry summaries forward, not full transcripts.
- Chunk big jobs. Section-by-section beats one giant paste.
- Estimate in tokens. ¾-word rule for quick math; a token counter when precision matters.
A practical takeaway: short, precise input beats padded input. Every word you paste gets processed, and your instruction competes with the padding for attention. State the ask plainly, supply only the material that matters, and both the limits and the focus work in your favor.
Practice
Which statement about tokens is accurate?
You are budgeting an AI feature and want to reason correctly about limits and costs.
Which statement is the most accurate?
Hint
The honest answer contains the word “depends”.
Why this is the answer
Tokens are slices, not words: the count depends on the model's tokenizer, the language and the content. Practical consequences: estimate with the ¾-word rule, expect code and emojis to cost more, and remember that both input and output count.
Transfer
- Budgeting a subscription or API by “messages”? Check what is counted — tokens in, tokens out.
- Long meeting transcript: summarize per segment, keep the decision bullets.
- Dense PDF with tables: expect the token count to be much higher than the word count suggests.
Next
Next: Context windows — what happens when all those tokens run out of room. The token glossary entry is the one-line version.