What is Token?
The unit LLMs read and bill in — roughly 3/4 of a word. A 1000-word page is ~1300 tokens. Agent platforms charge per input + output token, so token discipline (batching, caching, targeted reads) is the main cost lever.
What a token is
Tokens are the smallest units a language model works with. They can correspond to words, subwords, characters or even bytes. Text you send is encoded into a series of tokens for the model to process, and the model generates its reply as tokens too. Tokens are usually hidden at the text level but matter whenever you look at exact inputs, outputs, limits or prices.
How many characters is a token?
It depends on the model and the language. Anthropic notes that Claude 4.7 and later models use a newer tokenizer that produces roughly 30 percent more tokens for the same text than earlier models, on which a token was about 3.5 English characters. For an exact count, use the provider's token-counting tool with the model you plan to use; rules of thumb drift between model generations.
Why they matter
Tokens are the unit for the context window, for rate limits (requests, input tokens and output tokens per minute) and for price, which is quoted per million tokens. That is why the same prompt can cost a different amount on a newer model: the text did not change, the token count did.
Example
A 10,000-character document is about 2,900 tokens on a tokenizer that averages 3.5 characters per token, and about 30 percent more on a newer Claude tokenizer, so check it with the token counter before you budget a run.
Sources
See also
← Back to the full AI agent glossary.