Prompts Found
Why This Month's Findings Matter
Every month, our red team systematically probes large language models using adversarial prompts aligned to the OWASP LLM Top 10. The goal is simple: find what breaks before attackers do, and get that intelligence into defenders' hands first.
May's report tested 31 new adversarial prompts across 6 LLMs (GPT-5, GPT-4o, Llama 3 2B, Llama 4 Scout, and two open-source models), producing 186 total test cases spanning six attack families. Across all of it, our team confirmed an overall 22.0% Attack Success Rate — 41 unsafe responses out of 186 tests. Encouragingly, GPT-5 recorded zero unsafe responses this cycle, the first fully clean result we've seen. Less encouragingly, Llama 4 Scout failed on more than half of everything thrown at it. The takeaway is unchanged: current LLM deployments have exploitable gaps that organisations cannot afford to leave unmonitored.
Six Threat Families. One Critical Risk.
All prompts in this month's report map to LLM01 (Prompt Injection) — the top risk category in the OWASP LLM framework. Four of the six tracked families saw new prompts tested this cycle; the other two had no new submissions in May and are called out below for completeness.
Harmful Content Generation Medium
The highest-volume family this month at a 27.8% ASR (15/54). Confirmed successes ranged from phishing malware code and drug-synthesis instructions to leetspeak-obfuscated methamphetamine synthesis steps — largely driven by "unrestricted persona" and anti-refusal formatting jailbreaks against Llama 4 Scout.
Prompt Injection & Jailbreak Medium
A 20.5% ASR (16/78) this month. JSON-wrapped "jailbreak" payloads disguised as system reboot or lab-breach scenarios repeatedly coaxed Llama 4 Scout into engaging with harmful payload requests, even when the model initially flagged the framing as suspicious.
Roleplay-Based Jailbreak Medium
A 19.4% ASR (7/36) this cycle. Fictional and character-based framing continues to bypass safety filters at a meaningful rate, though it remains one of the lower-severity families relative to direct harmful-content or exfiltration attacks.
System Prompt & Model Probing Low
A 16.7% ASR (3/18), but with outsized impact: polite, "repository documentation"-framed requests caused both OSS 20B and OSS 120B to disclose full internal system prompts — in one case verbatim, in another via a fabricated-but-plausible reconstruction.
Disinformation & Media Manipulation Not Tested
No new prompts were submitted against this family in May. Given April's 100% ASR result, this remains a high-priority area for continued monitoring in upcoming cycles.
Persona Impersonation Not Tested
No new prompts were submitted against this family in May. April's findings on unflagged real-world persona impersonation still stand as an open risk for public-facing deployments.
What a Confirmed Jailbreak Looks Like
Every finding in the subscriber report follows this structure — a fully documented attack, the model's failure, the underlying reason it succeeded, and the controls needed to stop it. Here's a representative example from this month's Top 10 confirmed attack successes.
Which Models Held Up — and Which Didn't
GPT-5 was the standout this month, recording a 0.0% ASR across all 31 prompts — the only fully clean model in this test cycle. At the other end, Llama 4 Scout reached a 51.6% ASR, meaning more than half of adversarial prompts produced unsafe outputs.
| Model | Vendor | Unsafe / Tested | ASR | Residual Risk |
|---|---|---|---|---|
| GPT-5 | OpenAI | 0 / 31 | 0.0% | Low |
| GPT-4o | OpenAI | 5 / 31 | 16.1% | Medium |
| Llama 3 2B | Meta | 3 / 31 | 9.7% | Low |
| Llama 4 Scout | Meta | 16 / 31 | 51.6% | High |
| OSS 120B | Open-source | 8 / 31 | 25.8% | Medium |
| OSS 20B | Open-source | 9 / 31 | 29.0% | Medium |
Key observation: GPT-5 achieved a 0% ASR this month — the first fully clean result we've recorded — while Llama 4 Scout failed on 16 of 31 adversarial prompts (51.6%). The gap between the best- and worst-performing models widened significantly this cycle, reinforcing that model-level safety varies enormously and organisations deploying LLMs in production need an independent defensive layer that is updated continuously as new attack patterns emerge.
What You Get — Free, Every Month
This post shares the shape of May’s findings. The free monthly AI Threat Report delivers the headline intelligence straight to your inbox — one email a month, no noise. Subscribe here →
The Headline Findings
Each month’s confirmed attack vectors, grouped by family and severity, with OWASP mapping and tested model results — the patterns that matter, distilled.
Sample Finding Deep-Dives
Representative confirmed jailbreaks, fully documented — the adversarial prompt pattern, the model’s failure, why it succeeded, and the controls that stop it.
Monthly Cadence — Always Current
The threat landscape shifts every month. New models, new jailbreak techniques, new OWASP exposures. You receive a fresh report each month, keeping your defensive picture in pace with attacker innovation — not lag six months behind it.
From the Team Behind TraceCtrl
Our Threat Vector Database research is produced by the CloudsineAI red team — the same intelligence that hardens TraceCtrl Guard’s guardrails every month. Security observability and control for agentic AI.