Token

Models read text as tokens. Common words are often a single token, while rare words, names, and code get split into several pieces, and spaces and punctuation count too. The splits in the demo are illustrative: every model family uses its own tokenizer, so the same sentence can produce different counts.

Almost everything about using a model is measured in tokens: how much fits in the context window, how long a reply can be, and what a request costs. For English, a rough rule of thumb is about four characters, or three quarters of a word, per token.

Code, languages other than English, and long runs of numbers usually take more tokens than plain English prose. Large files, logs, and screenshots use tokens too, so pointing the AI at only the files it needs keeps a conversation cheaper and more focused.