AI Safety & Security
All 12 cards in this deck
The new attack surface and the defenses — prompt injection, jailbreaks, guardrails, and evaluating for harm. Answer each question before you open it — then run the review above, which schedules every card with spaced repetition so it comes back right before you would forget it.
Prompt injection vs jailbreak — the difference?
Jailbreak: the *user* coaxes the model past its own rules. Injection: a *third party* hides instructions in data the model reads (a page, a doc, a tool result) to hijack it.
Why is indirect prompt injection so dangerous for agents?
An agent reads untrusted content (web, email, files) into its context, where instructions and data blur — malicious text can redirect its tools and exfiltrate data with no user action.
The core defense against injection?
Never trust retrieved/tool content as instructions: separate data from commands, constrain tool permissions, require confirmation for high-impact actions, and sanitize/side-channel untrusted input.
What is the confused-deputy problem in AI systems?
The agent has legitimate privileges; injected instructions trick it into misusing them on the attacker's behalf. Fix: least privilege and per-action authorization, not blanket trust.
What are guardrails, concretely?
Input/output filters and policies around the model — blocking disallowed requests, redacting PII, checking outputs for unsafe content — a layer independent of the model's own alignment.
Why keep humans in the loop for high-stakes actions?
Models are confidently wrong and manipulable; a confirmation step on irreversible or costly actions (payments, deletes, sends) bounds the blast radius of a bad or injected decision.
What is data poisoning?
Corrupting training or retrieval data so the model learns a backdoor or wrong behavior. Defenses: provenance, curation, and evaluating on trusted held-out sets.
What is model extraction / stealing?
Reconstructing a model's behavior or weights via many queries. Mitigations: rate limits, output constraints, watermarking, and monitoring for scraping patterns.
PII in LLM pipelines — the risk and fix?
Sensitive data can leak into prompts, logs, traces, or training. Fix: redact/tokenize before the model, scrub logs, and set retention/no-train policies with the provider.
What does red-teaming an AI system mean?
Adversarially probing it for failures — jailbreaks, injections, harmful outputs, data leaks — before attackers do. Offense that informs defense; distinct from safety alignment work.
Why evaluate for safety with a held-out adversarial set?
Because "looks safe in demos" ≠ safe under attack. A maintained suite of adversarial and edge cases measures real robustness and catches regressions as prompts/models change.
Alignment vs security — how do they differ?
Alignment: making the model *want* to do the right thing (training, RLHF). Security: assuming it can be attacked or wrong and building defenses around it. You need both.