Security
More Attack Data Doesn't Mean Stronger AI Agent Defenses
Researchers found that retraining models on larger collections of prompt injections actually weakened security in most tests.
Key takeaways
- AI agents retrained on 7,132 prompt injection attacks were compromised more often than those trained on 4,282 attacks in three of four test scenarios.
- Automated red-team tools generate thousands of reworded variations of the same core attacks, creating the illusion of comprehensive coverage.
- PIPETTE, a technique-focused training system, reduced successful attacks to under 0.5% while maintaining or improving legitimate task completion rates.
- Detection-based defenses like PromptGuard can block attacks but leave agents unable to complete legitimate user tasks more than 80% of the time.
- Organizations should evaluate AI agent security based on technique diversity, adaptive attack resistance, and task completion rates—not total attack counts.
Quantity over quality backfires in AI security training
AI agents trained to resist prompt injection attacks performed worse when developers used a larger dataset of successful exploits, according to research from the University of Waterloo, Zhejiang University, and KTH Royal Institute of Technology.
The counterintuitive finding challenges a common assumption in AI security: that exposing models to more attack examples during training produces more robust defenses. Instead, the research suggests that attack diversity matters more than attack volume.
The team tested two automated red-teaming tools against Qwen3-4B, a small language model, across 560 practice scenarios. PISmith generated 7,132 successful attacks with a 68.75% success rate, while RL-Hammer produced 4,282 attacks at 33.39% success. After retraining the model on each collection and testing against novel attacks, the version trained on PISmith's larger dataset was compromised 47.5% of the time at worst. The RL-Hammer version peaked at 27.5% compromise rate.
The pattern held across three of four test configurations, according to the study first reported by Help Net Security.
Why attack tools generate redundant data
The core problem lies in how automated attack tools operate. Once a tool discovers an effective technique—such as convincing an agent that a user's task is already complete—it tends to recycle that approach across multiple scenarios. Both tools include diversity checks, but those checks measure surface-level differences in wording rather than underlying attack methodology.
RL-Hammer's own developers acknowledged this limitation, noting their tool satisfied diversity requirements by simply changing capitalization or inserting irrelevant text. The result: thousands of variations on a handful of core attacks, which trains agents to defend against specific phrasings while leaving fundamental vulnerabilities unaddressed.
A technique-focused alternative
The researchers developed PIPETTE, a system that categorizes attacks by technique rather than wording and rewards discovery of genuinely novel approaches. PIPETTE also trains agents to verify that each action serves the user's original request before executing it—a critical step that prevents agents from simply refusing any suspicious input along with legitimate user tasks.
In testing on AgentDojo, a standard benchmark for agent tasks, off-the-shelf injection attacks succeeded against PIPETTE-trained agents less than 0.5% of the time on average. Adaptive attackers given roughly 250 attempts per task achieved up to 10% success on known weak points. Importantly, the hardened Qwen3-4B completed 64.95% of legitimate tasks compared to 61.86% for the baseline model, demonstrating that security didn't come at the cost of functionality.
By contrast, PromptGuard—a detection-based defense that blocks flagged content—stopped slightly more attacks but left the agent unable to complete the user's task 83.81% of the time in one test scenario. PIPETTE's agent finished the same tasks 58.10% of the time.
Why it matters
As enterprises deploy AI agents with access to email, databases, and payment systems, prompt injection represents a serious attack vector. When vendors claim their agents were tested against thousands of successful attacks, this research suggests the right question isn't "how many attacks" but "how many distinct attack techniques." Organizations evaluating agent security should ask for technique coverage metrics, performance against adaptive attackers allowed hundreds of attempts, and task completion rates under attack conditions—not just raw attack counts.
The research tested models ranging from 4 billion to 27 billion parameters. The team noted that larger computational budgets for adaptive attacks may yield higher success rates, and they have not validated their technique-coverage scoring against human judgment.
The findings were reported by Help Net Security based on the research team's published paper.