Model Families and Multimodality

Key jargon

Term	Plain-language meaning
Modality	A form of information such as text, image, audio, video, or sensor data.
Encoder	A model component that converts input into useful representations.
Decoder	A component that generates an output sequence.
Multimodal model	A model designed to consume or produce more than one modality.

Key concepts

Architecture should follow the task: classification, representation, generation, or transformation.
Multimodal pipelines must preserve provenance and alignment between text, image, audio, and other inputs.

Concept map

flowchart LR
    A["Task and modalities"] --> B["Choose model family"]
    B --> C["Encode and reason"]
    C --> D["Generate or classify output"]

Family	Typical output	Example tasks
Classifier/regressor	Label or number	Fraud score, demand forecast, routing
Encoder/embedding model	Vector	Search, clustering, deduplication
Autoregressive language model	Token sequence	Writing, code, extraction, dialogue
Vision model	Label, boxes, mask, vector	Recognition, detection, segmentation
Diffusion/generative media model	Image/audio/video	Synthesis and editing
Speech model	Text or audio	Transcription and voice generation
Multimodal foundation model	Mixed modalities	Document understanding, visual question answering
Control/reinforcement model	Action or policy	Robotics, games, adaptive control

Selection principle

Choose the smallest system that meets the task’s quality, latency, cost, privacy, and control requirements. A deterministic parser or classifier can be better than a general-purpose generative model.

Exercise

For five tasks—invoice extraction, semantic search, logo generation, demand forecasting, and customer support—select a model family and identify the evaluation metric. Do not name a vendor until the requirements are clear.

Staleness warning

Model catalogs and benchmark leaders change quickly. Treat provider comparisons as dated experiments; keep the taxonomy and evaluation method durable.