← Explainers
2 min read

What is a token?

AI models don't read words. They read tokens. Here's what that means, and why it changes how you use them.

BuzzwordsAI basics

When you type a message to ChatGPT or Claude, the model never sees your words the way you do. Before anything else happens, your text is cut into small pieces called tokens. Everything the model does, it does with those pieces.

Term
Token
The unit of text a language model reads and writes

A small chunk of text: sometimes a whole word, sometimes part of a word, a space, or a punctuation mark.

Think of it like this

Think of LEGO. You don’t build with a whole house; you build with bricks. Tokens are the bricks an AI model uses to read and build sentences.

What a token looks like

Common words are usually a single token. Longer or rarer words get split into pieces the model has seen many times before.

One word, two tokens
tokenization
Illustrative split. The exact pieces depend on each model's tokenizer.

Where tokens fit in

How a prompt becomes an answer
  1. 01Your text
  2. 02Tokens
  3. 03Model
  4. 04Next token
  5. 05Answer
What happens when you hit send
  1. 1Split — your text is cut into tokens
  2. 2Predict — the model picks a likely next token
  3. 3Repeat — that token is added, and it predicts again, one at a time

That last step is why answers appear word by word. The model is writing one token after another.

A useful rule of thumb

~4
characters per token in English

Roughly 100 tokens is about 75 words. Other languages often need more tokens for the same meaning.

Why it matters

  • Limits. Every model can only hold a certain number of tokens at once. That’s its context window.
  • Cost. AI APIs charge per token, for both what you send and what you get back.
  • Odd mistakes. Because the model sees chunks, not letters, simple tasks like counting the letters in a word can trip it up.

A common misunderstanding

“One token = one word.” Not quite. Short, common words often are one token, but long words, names, numbers and emojis can take several.

In the real world

AI APIs such as OpenAI’s and Anthropic’s are priced per million tokens, with what goes in and what comes out counted separately. A chat app that resends a long conversation with every message pays for that history again each time, so many apps trim or summarise it.

LLM · Context window · Embeddings