Tokens, simply

Beginner · 5 min · Explain tokens and their effect on limits, costs and truncation.

  • foundations
  • costs and limits

A token is the unit of text a model actually processes — a small chunk, roughly a word piece. “Cat” is usually one token; “internationalization” may be several; a single emoji can be two or more. Models never see your letters the way you do: everything is converted to tokens first.

How the pieces are cut

Tokenizers split text according to patterns learned from data: common words stay whole, rarer words get sliced into fragments (international + ization), and unusual characters — accents, code punctuation, emojis — often cost extra pieces.

Here is one sentence, cut the way a model reads it:

  1. Un
  2. believ
  3. ably
  4. ,
  5. international
  6. ization
  7. breaks
  8. into
  9. pieces
  10. .
The sentence “Unbelievably, internationalization breaks into pieces.” is read as 10 tokens — chunks, not letters. Punctuation, rare words and word-fragments each take their own piece (example tokenisation; exact cuts vary by model).

For everyday English, a rule of thumb is about ¾ of a word per token — roughly 100 tokens per 75 words. It is only an estimate: the count depends on the model, the language (many non-English languages cost more) and the content itself. Code and tables are token-hungry.

Why it matters

Tokens are the currency of everything around the model:

  1. Limits are counted in tokens, not pages. A “10-page document” is not a fixed amount — dense, code-heavy text fills the context window much faster.
  2. Costs scale with tokens — input and output both. Verbose prompts and rambling answers are not free.
  3. Overflow means truncation. When input exceeds the window, the oldest content is dropped first — this is the mechanical reason long chats “forget”.

A bad example

Assume: “A page is about 500 words, so 200 pages is 100,000 words — should fit.”

If the limit is counted in tokens and the document is dense (tables, code, another language), the real count can be far higher. The system may silently drop the beginning — and the model answers from what remains, confidently.

A better example

Ask: “Summarize this report section by section. After each section, give me three bullets I can carry into a new chat.”

Long work becomes a series of short, verifiable steps — and the carried bullets become your compressed context.

Why it works

Once you know the unit, the tactics follow:

  • Compress instead of hoarding. Carry summaries forward, not full transcripts.
  • Chunk big jobs. Section-by-section beats one giant paste.
  • Estimate in tokens. ¾-word rule for quick math; a token counter when precision matters.

A practical takeaway: short, precise input beats padded input. Every word you paste gets processed, and your instruction competes with the padding for attention. State the ask plainly, supply only the material that matters, and both the limits and the focus work in your favor.

You paste a dense 40-page PDF full of tables and code into a chat. What should you expect?

Practice

Which statement about tokens is accurate?

You are budgeting an AI feature and want to reason correctly about limits and costs.

Which statement is the most accurate?

Choose one option
Hint

The honest answer contains the word “depends”.

Transfer

  • Budgeting a subscription or API by “messages”? Check what is counted — tokens in, tokens out.
  • Long meeting transcript: summarize per segment, keep the decision bullets.
  • Dense PDF with tables: expect the token count to be much higher than the word count suggests.

Key takeaways

  • Models read tokens — roughly ¾ of an English word each — not letters, words or pages.
  • Limits, costs and truncation are all counted in tokens.
  • Short, precise input beats padded input; chunk big jobs section by section.

Next

Next: Context windows — what happens when all those tokens run out of room. The token glossary entry is the one-line version.

Finished this lesson?