Model Families and Multimodality

Fact-check status

Reviewed claim by claim on 2026-09-02 against the primary or official sources identified inline as S1 and S2. Recommendations are design guidance unless a cited standard makes them normative.

Key jargon

Term	Plain-language meaning
Modality	A form of information such as text, image, audio, video, or sensor data. S1
Encoder	A model component that converts input into useful representations. S1
Decoder	A component that generates an output sequence. S1
Multimodal model	A model designed to consume or produce more than one modality. S1

Key concepts

Architecture should follow the task: classification, representation, generation, or transformation. S1
Multimodal pipelines must preserve provenance and alignment between text, image, audio, and other inputs. S1

Concept map

flowchart LR
    A["Task and modalities"] --> B["Choose model family"]
    B --> C["Encode and reason"]
    C --> D["Generate or classify output"]

Diagram semantics verified against S1 and S2 on 2026-09-02.

Family	Typical output	Example tasks S1
Classifier/regressor	Label or number	Fraud score, demand forecast, routing S1
Encoder/embedding model	Vector	Search, clustering, deduplication S1
Autoregressive language model	Token sequence	Writing, code, extraction, dialogue S1
Vision model	Label, boxes, mask, vector	Recognition, detection, segmentation S1
Diffusion/generative media model	Image/audio/video	Synthesis and editing S1
Speech model	Text or audio	Transcription and voice generation S1
Multimodal foundation model	Mixed modalities	Document understanding, visual question answering S1
Control/reinforcement model	Action or policy	Robotics, games, adaptive control S1

Selection principle

Choose the smallest system that meets the task’s quality, latency, cost, privacy, and control requirements. A deterministic parser or classifier can be better than a general-purpose generative model. S1

Exercise

For five tasks—invoice extraction, semantic search, logo generation, demand forecasting, and customer support—select a model family and identify the evaluation metric. Do not name a vendor until the requirements are clear.

Staleness warning

Model catalogs and benchmark leaders change quickly. Treat provider comparisons as dated experiments; keep the taxonomy and evaluation method durable. S1