# Tokenization > [!note] Disambiguation > This page is about how AI language models split text into tokens. For financial tokenization (stocks, bonds, funds or deposits recorded on a blockchain), see [[Tokenized Assets]]. **Tokenization** is the process of converting raw text into the numerical units — **tokens** — that AI models actually process. Language models don't read words; they read tokens. A token might be a whole word ("the"), a subword fragment ("un" + "believ" + "able"), a single character, or a punctuation mark. English text averages roughly 1 token per 0.75 words, though this varies significantly by language, vocabulary, and content type. Tokenization determines how the model "sees" text, how much text fits in the [[Context Window]], and how much each query costs (pricing is per-token). It is a foundational but often invisible layer of AI infrastructure. ## How It Works Modern LLMs typically use **Byte-Pair Encoding (BPE)** or similar subword tokenization algorithms. The process: start with individual characters, then iteratively merge the most frequent pairs until you reach a desired vocabulary size (typically 32K–100K+ tokens). Common words become single tokens; rare words are split into subword pieces. This approach balances efficiency (common text is compressed into fewer tokens) with coverage (any text, including misspellings, code, and foreign languages, can be represented as a sequence of subword tokens). The trade-off is that languages with different character systems (Chinese, Japanese, Arabic, Hindi) are often tokenized less efficiently than English — requiring more tokens to represent the same semantic content. ## The Language Inequality Tokenization creates a structural bias: English text is tokenized most efficiently (fewest tokens per concept), while other languages — especially those with non-Latin scripts — require more tokens for equivalent content. This means non-English users consume more of the context window per query, pay more per API call, and receive less information per interaction. This is not a design choice so much as a training-data artifact — tokenizers are trained on corpora dominated by English web text, so English vocabulary is better compressed. But the effect is real: AI systems are literally more expensive and less capable for non-English speakers, reinforcing the English-language dominance of the digital information economy. ## KB Significance 1. **The hidden cost structure.** Token-based pricing is how AI companies charge for their products. Understanding tokenization is prerequisite to understanding the economics of AI deployment — including why long-document analysis is expensive, why non-English applications cost more, and why the "price per token" metric that drives AI industry competition is not as straightforward as it appears. 2. **The language-equity issue.** Tokenization's bias toward English is a technical mechanism through which AI systems perpetuate linguistic hierarchies. For the KB's analysis of how technology reinforces existing power structures, tokenization is a concrete example: the most powerful information technology in history works best in the language of its creators. ## See Also - [[Context Window]] — Measured in tokens - [[Large Language Model]] — The systems that use tokenization - [[Inference vs Training]] — Where per-token costs are incurred