Attack and defend real LLM applications. Prompt injection, jailbreaks, RAG and agent attacks, model-level attacks, guardrails and secure architecture, red-teaming, and AI governance — anchored in the OWASP LLM Top 10, MITRE ATLAS, and NIST AI RMF.
Ten modules that build the full attack-and-defend picture for LLM and agentic AI systems, in the order a working AI security practitioner actually needs it. You start with why LLM applications break differently than classic web apps and the OWASP LLM Top 10, then go deep on the two attacks that dominate real incidents — prompt injection and jailbreaks — before moving into the newer, higher-stakes surfaces: RAG and data-layer attacks, agent and tool abuse, and attacks aimed at the model itself (extraction, inversion, membership inference, denial-of-wallet). The back half of the course is defensive: guardrails and structured-output patterns, secure architecture (least privilege, secrets handling, tenant isolation, supply chain, egress control), building and running an AI red-team program, and the governance layer — audit logging, incident response, EU AI Act and NIST AI RMF basics — that turns individual controls into a program an organization can sustain and be audited against. Every attack is shown defensively and educationally: real, concrete payloads paired with the filters and architectural controls that catch them, never turnkey exploitation tooling. Python-first throughout, with every technique shown as runnable code you can adapt into your own system.
Built by Lakshya Kumar
We grant free access case-by-case — students, career-switchers, builders on a tight budget. Sign in to send us a note.
Sign in to applyComplete all modules, then submit the required number of capstone projects. Each must earn a passing rating from an admin reviewer.
Red-team a provided (or your own) AI application against the OWASP LLM Top 10. Produce a findings report mapping every discovered vulnerability to its OWASP category, with severity, reproduction steps, and impact. Then implement the fixes for at least your three highest-severity findings, and ship a regression suite (reusing this course's harness patterns) that proves each hole is closed and will stay closed on future changes.
Paste this into any AI chat. Fill in the bracketed parts with your context — you'll get back a straight answer on whether this belongs on your plate.
I'm taking an "LLM & AI Security" course. It covers, in order: the LLM threat model and the OWASP LLM Top 10; prompt injection (direct, indirect, exfiltration, encodings, tool-output injection); jailbreaks and guardrail bypass (persona/roleplay, obfuscation, crescendo, translation gaps, payload splitting, many-shot); RAG and data-layer attacks (poisoning, access control, tenant isolation, embedding attacks, citation exploitation); agent and tool abuse (excessive agency, confused-deputy, tool-call hijacking, sandbox escapes, SSRF, multi-agent propagation, approval gating); model attacks (prompt extraction, inversion, membership inference, memorization, denial-of-wallet, distillation); defenses and guardrails (input/output filtering, structured output, spotlighting, least-functionality, dual-LLM isolation); secure AI architecture (least privilege, secrets, PII redaction, tenant isolation, supply chain, egress control, logging); AI red-teaming (adversarial suites, automated attacker loops, grading, coverage, CI gating, bounty triage, findings reports, tabletops); and governance/compliance (audit logging, incident response, EU AI Act, NIST AI RMF, model cards, vendor risk, disclosure). My context: 1. The AI system I want to secure is: [describe — chatbot / RAG assistant / tool-using agent / something else] 2. Its current security posture: [none / basic input filtering / a real guardrail pipeline / I genuinely don't know] 3. The attack I'm most worried about right now: [prompt injection / jailbreaks / data leakage via RAG / agent taking a bad action / I'm not sure] 4. My role and constraints: [solo builder / small team, no dedicated security hire / have a security team but they don't know AI-specific risks / other] Given that, answer: - Which module should I start with, and why, given what I'm most worried about? - Name the first three test cases I should write against my system today, specific to my described attack surface. - Name the single highest-leverage defense (from this course) I should add this week given my current posture. - If I can only build one artifact from this course in the next month, should it be a test harness, a guardrail pipeline, or a governance document — and why, given my role and constraints? - Name one thing this course will NOT fully solve for me (this is a genuinely unsolved area of security), so I have realistic expectations.
Build a reusable guardrail/defense middleware library implementing at least five distinct controls from this course (input risk classification, output schema validation, spotlighting, least-functionality tool scoping, dual-LLM isolation, or an SSRF/egress-safe fetch tool). Ship it with a documented API, a regression test suite covering both attack-blocking and false-positive cases, and a README showing how to wire it into an existing LLM application in under an hour.
Build an automated red-teaming harness: a taxonomy-mapped seed corpus (at least 6 OWASP LLM Top 10 categories), a mutate-test-select attacker loop that generates novel variants, a layered grader distinguishing attack success from failure, a coverage report, and a CI-style gate that blocks a regression in caught rate. Run it against a real or realistic target system and submit the harness, its coverage report, and the findings it surfaced.
The most-cited ongoing writing on prompt injection and its defenses. Modules 1, 2, and 7 lean on it directly.