Threat-Model the Complete AI System

Key jargon

Term	Plain-language meaning
Threat model	A structured analysis of assets, trust boundaries, adversaries, failure modes, and controls.
Asset	Something valuable that requires protection, such as data, credentials, decisions, or availability.
Trust boundary	A point where data or control crosses between different trust levels.
Attack surface	All reachable interfaces and behaviors an adversary might influence.

Key concepts

Threat-model the full system—users, data, model, retrieval, tools, infrastructure, and operators—not just the model endpoint.
Trace data and authority across boundaries to identify where untrusted text can become consequential action.

Concept map

flowchart LR
    A["Identify assets and actors"] --> B["Map data flows and trust boundaries"]
    B --> C["Enumerate threats and impact"]
    C --> D["Choose testable controls"]

Assets and boundaries

Include user identity, prompts, retrieved data, model/provider, embeddings, vector/graph stores, tools, credentials, code execution, memory, logs, evaluation data, supply chain, administrators, and downstream consumers.

Questions

Who can supply instructions or data?
Which content is trusted, untrusted, or mixed?
What sensitive information enters or leaves?
What external effects can tools cause?
Which identity and authorization govern each effect?
How can the system fail open, repeat, or conceal failure?
What evidence and rollback exist?

Exercise

Threat-model a document assistant with one vector database and email tool. Draw boundaries among user, host, model provider, document store, tool gateway, and mail system. Rank risks by consequence and precondition.