Tokenizer
The model-specific converter that splits text into numbered token pieces and turns generated token IDs back into text.
See it
What it is
A tokenizer converts text into the integer token IDs a model consumes, then converts generated IDs back into text. Its vocabulary and splitting algorithm decide whether a string is one token or many. Byte pair encoding, WordPiece, and unigram are the common algorithms, often applied through a library such as SentencePiece, each with rules for punctuation, whitespace, Unicode, and special control tokens.
Reach for the tokenizer whenever you enforce a context limit, estimate cost, truncate chat history, or split documents into chunks. Use the exact tokenizer paired with the selected model, because the count is part of that model's contract rather than a universal property of the text.
Gotcha: counting words or characters is only a rough estimate. Code, URLs, numbers, and different writing systems split very differently, and two model families can assign different IDs and counts to the same sentence. It is also why letter puzzles go wrong: a model asked how many r's are in strawberry never receives ten letters, it receives a couple of tokens. Special tokens can also appear in the serialized request even when they were never visible in the text box.
Ask AI for it
Build a Python token-inspector CLI with tiktoken. Accept a model name and UTF-8 text, select its tokenizer with encoding_for_model, and print a table containing each token ID, decode_single_token_bytes output, display-safe text, byte length, and cumulative token count. Show totals for characters, UTF-8 bytes, words, and tokens, catch the KeyError that encoding_for_model raises for an unknown model name and warn before falling back to get_encoding('o200k_base'), and add tests covering English, Hindi, emoji, JSON, source code, and a long URL.