Inference, Quantization, and Serving

Key jargon

Term	Plain-language meaning
Inference engine	Runtime software that executes a trained model.
Quantization	Representing weights or activations with lower precision to reduce memory or compute.
Throughput	Work completed per unit time, often tokens or requests per second.
Time to first token (TTFT)	Delay between request submission and the first generated token.

Key concepts

Serving is a queueing and resource-management problem as well as a model problem.
Quantization trades resource use against quality and hardware compatibility; validate on the target task.

Concept map

flowchart LR
    A["Load model representation"] --> B["Schedule and batch requests"]
    B --> C["Generate tokens"]
    C --> D["Measure quality latency throughput"]

Serving concepts

Prefill processes input context; decoding generates subsequent tokens.
The KV cache accelerates autoregressive decoding but consumes memory.
Batching improves throughput while affecting queueing latency.
Quantization reduces numerical precision and memory/compute requirements; quality impact is model/task/hardware specific.
Speculative decoding and optimized kernels can improve speed without changing the application contract.

Metrics

Time to first token, inter-token latency, total latency, input/output tokens per second, concurrency, queue time, memory, cache utilization, error rate, cost per successful task, and task-quality regression.

Exercise

Benchmark a small local model on ten non-sensitive prompts at two quantizations. Record hardware, runtime, model hash, quality rubric, latency, memory, and power if available.

Misconception

Requests per second alone hides prompt length, generation length, batching, and success quality.