Jailbreak
A prompt designed to talk a model past its learned safety rules and make it produce something it would normally refuse.
See it
What it is
A jailbreak is a prompt crafted to talk a model past the safety behavior learned during training. It may use role-play, encoded instructions, long fictional setups, or pressure to follow a conflicting goal. The early ChatGPT-era DAN, short for 'Do Anything Now', is the cultural anchor: pages of text that asked the model to pretend its rules no longer applied. The 2023 grandma prompts worked the same way from the other direction, wrapping a refused request inside a bedtime story told by a late relative.
Reach for the term when red-teaming a base model or testing whether a safety refusal survives adversarial wording. Keep it distinct from prompt injection: a jailbreak targets the model's own safety behavior, while prompt injection targets an application's instructions, often through a document or webpage the model reads.
Gotcha: blocking a list of famous prompts is not a defense. Variants, translations, and new framing appear faster than a denylist can grow. Test whole categories of attacks, constrain tools and permissions in code, and judge the actual harmful output, not whether the input happened to contain a suspicious phrase.
Ask AI for it
Build a jailbreak regression suite around the open-source garak scanner. Run its jailbreak probes plus our own role-play, encoding, translation, and multi-turn cases against the staging model endpoint, store the full prompt and response for each case, and score the response against a written harm rubric rather than keyword matching. Add benign refusal tests to catch overblocking, report attack success rate by probe family, and fail CI when either harmful compliance rises or benign answer rate falls beyond its checked-in baseline.