Pick a depth. Each prompt opens in your AI pre-loaded with the lesson. Click a row to preview the prompt.
It is dangerously easy to forget that model output is still untrusted, attacker-influenceable content — and every downstream system that consumes it without validation inherits whatever an attacker managed to steer the model into producing. This is a direct rerun of classic injection, one layer removed: instead of user input flowing unsanitized into a SQL query or a shell command, now it's model output doing it, and the model output was itself shaped by a prompt-injected input upstream. Teams that carefully sanitize user input but then pipe the model's response straight into eval, a database query, or unescaped HTML have built a fully working exploit chain — they just moved the untrusted step one hop later, which makes it easier to miss in review.
The demo shows a support-bot 'action' feature that lets the model output a shell command to run diagnostics, first the vulnerable way (blind exec), then hardened with a strict allowlist — output handling has to validate before use, exactly like input handling does.
run_diagnostic_safe before it ever reaches a subprocess call.traceroute to ALLOWED_DIAGNOSTICS and confirm an unknown command like 'reboot' is still rejected.shell=True in run_diagnostic_unsafe is itself a code smell independent of AI, and why the AI angle makes it worse, not different in kind.import subprocess, shlex
# VULNERABLE: whatever the model outputs is executed directly.
def run_diagnostic_unsafe(model_output: str):
# If a prompt injection upstream got the model to output
# "ping 1.1.1.1; rm -rf /tmp/data", this runs it verbatim.
return subprocess.run(model_output, shell=True, capture_output=True, text=True)
# HARDENED: the model may only select from a fixed allowlist of commands
# and pass validated arguments -- its raw text is never executed.
ALLOWED_DIAGNOSTICS = {
"ping": lambda host: ["ping", "-c", "1", host],
"dns_lookup": lambda host: ["nslookup", host],
}
def run_diagnostic_safe(command: str, target: str):
if command not in ALLOWED_DIAGNOSTICS:
raise ValueError(f"command '{command}' is not in the allowlist")
if not target.replace(".", "").isalnum():
raise ValueError("target must be a simple hostname or IP")
argv = ALLOWED_DIAGNOSTICS[command](target)
return subprocess.run(argv, capture_output=True, text=True)
# The model NEVER gets to supply a raw shell string, only a
# (command, target) pair the caller validates against a fixed menu.
try:
result = run_diagnostic_safe("ping", "1.1.1.1; rm -rf /tmp/data")
except ValueError as e:
print("blocked:", e)
print(run_diagnostic_safe("ping", "1.1.1.1").stdout[:60])python3 main.py