Tokens, Embeddings, and Context

Key jargon

Term	Plain-language meaning
Token	A model-specific unit of text or bytes represented by an integer ID.
Tokenizer	The reversible rules that convert input into token IDs and IDs back into text.
Embedding	A learned vector representation used to encode meaning or model state.
Context window	The bounded sequence of tokens available to a model for one generation.

Key concepts

Models operate on tokens, not directly on words or ideas.
Embedding proximity can indicate similarity, but it does not prove truth, identity, or authorization.

Concept map

flowchart LR
    A["Raw input"] --> B["Tokenizer produces IDs"]
    B --> C["Embeddings encode vectors"]
    C --> D["Context drives generation"]

Core concepts

A tokenizer converts text into model-specific integer IDs. Tokens may be words, word pieces, punctuation, whitespace, or bytes.
An embedding maps an item to a vector whose geometry can encode useful relationships.
The context window is the bounded sequence available to a model invocation: system instructions, conversation, retrieved material, tool descriptions, and generated tokens.
Positional information lets the model distinguish ordering.

Token count is not character count. Two models may tokenize the same text differently, affecting cost and available context.

Three embedding uses

Input token embeddings inside the language model.
Retrieval embeddings used to compare documents and queries.
Multimodal representations connecting text with images, audio, or other data.

They are related ideas, not necessarily interchangeable models or vector spaces.

Exercise

Use an official tokenizer tool for one model. Compare a paragraph of prose, JSON, code, and a non-English sentence. Record token counts and explain the differences without assuming tokens equal words.

Checklist

Budget context by purpose, not by filling the window.
Keep trusted instructions distinguishable from untrusted data.
Measure retrieval usefulness and position effects.
Redact secrets before context assembly.