How It Works

Tokenization / Token

The process of breaking text into smaller units (“tokens”) that a model processes — often close to whole words, sometimes parts of words. Usage-based pricing for AI tools is typically measured in tokens.

Origin · no single documented coiner

Inherited from computational linguistics and compiler theory; no AI-specific coining event.

No single primary source link for this term.