Post

How to Build Red Teaming Datasets That Stay Relevant as AI Attack Surfaces Evolve

Table of Contents
    Add a header to begin generating the table of contents

    Red teaming datasets must evolve alongside AI agents as new risks emerge across tools, memory, and connected systems. Learn how updated risk taxonomies, adversarial scenarios, regression testing, and expert reviews help keep datasets relevant as AI capabilities and attack methods evolve.

     

    Ask an AI model the wrong way, and it may do something it shouldn’t. Attackers phrase prompts to slip past safety rules or bury instructions in documents and web pages the model will read. 

    AI red teaming gets ahead of them by attacking models on purpose, using adversarial prompts and scenarios to surface those weak spots first. These prompts and scenarios are stored in red teaming datasets. The challenge is that the AI attack surface, meaning every point where an AI system can be attacked, keeps growing as agents gain tools, memory and connections to other agents.

    AI agent attack surface: prompts, tools, memory and connected agents

    In May 2025, researchers showed that a malicious public GitHub issue could trick a connected AI agent into leaking private repository data. The OWASP Top 10 for Agentic Applications, a list of the top security risks for AI agents, cites this GitHub MCP exploit as evidence of these new risks.

    These risks keep evolving, so red teaming datasets cannot be built once and reused. They have to be updated as the attack surface changes. This article explains why these datasets go stale, what chatbot-era datasets miss, and the four practices that keep them current.

    Why Red Teaming Datasets Go Stale

    A red teaming dataset is designed for a specific AI system and the threats known when it was built. When the system or its AI attack surface changes, the dataset no longer tests everything that matters. Four changes cause this gap.

    1. AI Agents Can Be Attacked in More Places

    A chatbot takes a prompt and returns a response. An AI agent can also call tools, store memory, and pass tasks to other agents. Each of these abilities gives attackers a new way in, and attackers can chain several steps together into a single attack.

    In November 2025, researchers demonstrated a chained attack on ServiceNow’s Now Assist platform. A user with limited access hid instructions in a support ticket. When an admin later used an AI agent to summarize tickets, the agent read those instructions. It then recruited a second, more capable agent to copy restricted records into the attacker’s ticket.

    The agents worked with the admin’s access, so the attacker could see data they were not allowed to view. ServiceNow’s built-in prompt injection protection was turned on, but it did not stop the attack.

    2. New Attack Techniques Keep Appearing

    Security frameworks that track attack techniques are still catching up with attacks on AI agents. In October 2025, MITRE ATLAS, a public knowledge base of attacks on AI systems, added 14 techniques and sub-techniques focused on AI agents. A March 2026 analysis by the Cloud Security Alliance reports that MITRE has since identified seven more. They came from investigating two vulnerabilities in OpenClaw, an open-source AI agent.

    These seven techniques were seen in real-world attacks on agent platforms. The analysis also found gaps that ATLAS does not yet cover, such as attacks that spread from one agent to another. A dataset built before these updates would not test any of them.

    3. Models Stop Failing Known Attacks

    Once attack prompts are published, they can end up in the data used to train new models. Models then learn to resist them.

    Benchmark researchers have seen this play out. Fixed safety benchmarks tend to become saturated or contaminated as they age. Either models pass almost everything, or the test prompts have quietly made their way into training data. Updating them by hand is slow work, too. So a team can end up with a dataset that says a model is safe, even though it was never tested against anything new. 

    4. The AI System Itself Keeps Changing

    Models, prompts, and policies change often, and each update can make some tests irrelevant. Say a team connects its agent to email. Now an attacker can try to talk it into sending data outside the company, and an older dataset won’t have a single test for that. 

    Policy updates are harder to spot. If the rules about what an agent may do get rewritten, tests designed around the old rules may keep checking for behavior that no longer matters. 

    Put these four shifts side by side, and it’s easy to see why chatbot-era datasets miss so many of the attacks AI agents face today.

    What Chatbot-Era Red Teaming Datasets Miss

    Red teaming datasets from the chatbot era were mostly built around a single question: will the model say something harmful if prompted the right way? The real incidents cited in the OWASP Agentic Top 10 point to a different risk, where hidden instructions push an agent to do something harmful 

    More than 100 industry experts, researchers, and practitioners helped OWASP put the list together as a guide to agentic AI security. Each risk category is backed by publicly reported incidents, collected in an exploit tracker. The risks are numbered ASI01 to ASI10. The five incidents in the table below were chosen to show attacks on tools, memory, and connected agents, that prompt-only dataset would not test and OWASP cites them under three of these risks:

    • ASI04: Agentic supply chain vulnerabilities, meaning risks from third-party tools and components an agent relies on.
    • ASI06: Memory and context poisoning.
    • ASI07: Insecure inter-agent communication.
    Attack Surface What Happened Where OWASP Cites it
    Tools May 2025: researchers showed that a malicious public GitHub issue could make an agent leak private repository data into a public pull request. ASI04 references and exploit tracker
    Tools July 2025: an attacker added a harmful prompt to version 1.84 of Amazon’s Q coding assistant extension for VS Code. It told the AI agent to delete the user’s local files and AWS cloud resources. ASI04 example and exploit tracker
    Tools September 2025: a fake MCP server posing as the one for the Postmark email service was published on npm, a public software registry. It secretly sent a copy of every email to an attacker. ASI04 example and exploit tracker
    Memory February 2025: a researcher used prompt injection, hiding instructions in content Gemini processed, to plant false information in its long-term memory. The false memories stayed active in later conversations. ASI06 references
    Connected Agents April 2025: in the Agent2Agent (A2A) protocol, each agent describes its skills in a public profile called an agent card, and a coordinating agent uses these cards to pick which agent gets a task. A researcher showed that a rogue agent could overstate its skills in its card, so the coordinating agent choses it and sends it users’ tasks and data. ASI04 example; ASI07 in exploit tracker

     

    None of these attacks can be tested with a single prompt and response. Each one needs tools, data, permissions or other agents to reproduce. Keeping red teaming datasets current therefore means changing how each test is built, not just adding new prompts.

    Four Practices for Keeping Red Teaming Datasets Current

    A red teaming dataset stays useful only if someone keeps working on it. Think of it as an ongoing program rather than a file built once and left alone. The four practices below cover organizing attacks, adding new ones, retiring old ones, and refreshing the whole set over time.

     

    Four-step cycle for keeping red teaming datasets current

     

    1. Organize Attacks by an Updated Risk Taxonomy

    An AI risk taxonomy is the set of categories used to group attacks by the type of risk they test, such as data leakage or tool misuse. Without consistent categories, it is hard to see which risks have too few tests. Researchers behind the RedBench dataset found that existing red teaming datasets often define risk categories inconsistently, which makes them hard to compare.

    To fix this, RedBench labels every attack with one of 22 risk categories and one of 19 domains. For AI agents, teams can group attacks using the risk categories in frameworks such as the OWASP Agentic Top 10 and MITRE ATLAS. Each attack should also be labeled with the part of the system it targets: prompts, tools, memory or connected agents.

    When OWASP or MITRE add a new category, teams should check whether it applies to their system’s tools, memory, and connected agents, and add it only if it does. Any relevant category with few or no attacks then shows where the dataset is falling behind. This makes the taxonomy a map of both what the dataset covers and what it still needs.

    2. Add New Attack Categories as They Emerge

    New attack techniques appear regularly, so teams need a steady process for adding them. Useful sources include incident reports, security research and framework updates. OWASP, for example, maintains a public tracker of real exploits and incidents involving AI agents, and its Agentic Top 10 document says this list is updated every week.

    Each new attack should become a full test scenario, meaning a test that recreates the conditions the attack needs. For the GitHub MCP exploit, that means the agent’s tool access, a test repository with private data and the public issue that carries the hidden instruction. A single prompt cannot recreate these conditions.

    Teams should also create several versions of each scenario with different wording, tools, or data. This tests the attack method rather than one specific example. Researchers also recommend rotating hidden scenario variants, meaning regularly swapping in test versions the model has not seen, so defenses are not tuned only to tests the team already knows.

    3. Retire Prompts the Model Has Already Learned to Resist

    A prompt that every current model already resists cannot show whether a new model version is safer. A dataset full of such prompts also makes pass rates look high without testing unseen risks. Teams can retire a prompt once none of several current models fall for it, then replace it with attacks that still work.

    Research on renewable safety benchmarks follows a related idea: it generates many attack prompts and keeps only those that still work across several models. Retired prompts should still be kept in a regression test set that is rerun on every release. This checks that old weaknesses have not returned after a model update. Research on agentic AI security recommends this kind of regression gate, which blocks a release if results on known attacks get worse.

    4. Run a Regular Refresh Cycle With Expert Review

    The OWASP GenAI Red Teaming Guide calls for continual oversight because no AI model is ever truly “done” or “secure.” A refresh cycle puts this into practice through a routine review and update of the dataset. It should run on a fixed schedule and whenever something important changes, such as a new model version, a new tool or a newly reported attack.

    Each cycle reviews coverage against the taxonomy, adds new scenarios, retires resisted prompts and saves a new dataset version. Saved versions let teams compare results over time. Experts who follow new attack techniques should guide each cycle.

    These experts decide which new attacks matter, which prompts to retire and whether existing scenarios still match the system’s current tools and permissions. To learn why the attacks themselves need expert vetting, read From Edge Cases to Exploits: Why Red-Teaming Needs Expert-Vetted Datasets.

    How iMerit Helps Keep Red Teaming Datasets Current

    Running this refresh cycle at scale requires two things: experts who follow new attack techniques, and workflows that turn their findings into reviewed test data. iMerit’s red teaming and adversarial data construction capability, used in frontier model safety programs, builds and refreshes adversarial datasets in this way. It supports each of the four practices:

    • Building attack datasets by risk category: iMerit’s red teaming services cover scenario testing, bias detection and safety audits, combining workflow automation with domain experts.
    • Tracking new attacks: iMerit Scholars are graduate-level domain experts, selected for expertise rather than crowd-sourced labeling. They design multi-step adversarial scenarios against agent workflows and track new attack techniques as they emerge.
    • Testing agent-specific attacks: iMerit’s AI agent evaluation reviews every step an agent takes, including how it handles prompt injection, manipulated tool outputs and data exfiltration attempts. It can also serve as a regression gate before each release.
    • Running refresh and review cycles: Ango Hub supports custom review workflows with multiple reviewers for new and retired attacks.

    For a conversational AI company building a multilingual large language model (LLM), iMerit subject matter experts wrote and reviewed 60,000 prompts in 10 languages. Some prompts mixed Hindi and English in the same sentence, the way many users actually type. 

    Conclusion

    Red teaming datasets go stale when the AI attack surface grows faster than the datasets are updated. Real incidents cited in the OWASP Agentic Top 10 show attacks that chatbot-era datasets were never designed to test. Keeping AI red teaming datasets relevant means managing them as an ongoing program.

    An updated risk taxonomy shows which risks have too few tests, and new attacks become full test scenarios. Resisted prompts move to a regression set. And with experts reviewing each refresh, the dataset keeps pace with how AI systems are actually being attacked.

    Key takeaways

    • Once AI agents can use tools, store memory, and talk to other agents, older red teaming datasets start to go stale.
    • Incidents cited in the OWASP Agentic Top 10 show attacks that single-prompt datasets cannot test.
    • An updated AI risk taxonomy makes coverage gaps visible.
    • New attack techniques should become full test scenarios with the tools, data and permissions they need.
    • Prompts the model already resists belong in a regression set, not the active dataset.
    • A regular refresh cycle with expert review keeps adversarial datasets aligned with current risk.

    Talk to an iMerit expert about building and refreshing AI red teaming datasets that keep pace with new attacks.