What is a token?
LLMs do not process text as characters or words. They process tokens: subword units produced by a tokenizer, typically using byte-pair encoding (BPE). A tokenizer splits raw text into pieces from a fixed vocabulary. A common word like "the" is usually one token. A rare word gets split into several smaller pieces. Every prompt you send and every response you receive is measured in these units, and providers bill per token, with input and output priced separately.
The 0.75-words rule of thumb
For ordinary English prose, one token is roughly 0.75 English words. That makes 1,000 tokens about 750 words. It is a useful mental estimate, and it breaks in two common cases:
- Code. Source code usually produces more tokens per word than prose. Punctuation, identifiers, and whitespace all fragment into extra tokens.
- Non-English text. Text outside English also tends to produce more tokens per word, because tokenizer vocabularies are weighted toward English.
If you are estimating costs for a codebase or a multilingual workload, the 0.75 rule undercounts. Count the actual text instead of estimating.
Why providers bill per token
Tokens are the unit the model actually consumes and produces, so they are the unit of compute. That is why every provider's price sheet is per token, why input and output carry different rates, and why discounts like cached input are expressed against per-token prices.
Why counts differ per provider
Each provider has its own tokenizer, so the same text produces different counts on different platforms. Some of those tokenizers are public and some are not:
- OpenAI's o200k_base tokenizer is open source, so exact GPT counting can run entirely in a browser.
- Anthropic publishes no tokenizer for current Claude models. The only exact count is the free count_tokens API endpoint.
- Gemini has a free countTokens API endpoint.
- Kimi K2's tokenizer is open, a tiktoken-style BPE with a 163,840-token vocabulary. DeepSeek's tokenizer is open on Hugging Face.
Counts differ per Claude generation too
Claude's tokenizer differs by model generation, not just by provider. Opus 4.7 and later, including Opus 5 and Fable 5, use a newer tokenizer that produces roughly 1x to 1.35x the tokens of the 4.6-and-older family on identical text. Sonnet 5 counts about 30% more tokens than Sonnet 4.6 on identical text. Verified 2026-08-10.
A practical consequence: a per-token price comparison between Claude generations is incomplete without a token-count comparison. The same document costs a different number of tokens on different Claude models before pricing even enters.
Related
Count your own text with the token calculator or convert with the tokens-to-words converter. For exactly how our counters produce their numbers per provider, see How we count.