Guide
AI Red Teaming: What It Is, How It Works and When You Need It
AI red teaming is a structured, adversarial test of an AI system: a team takes the attacker's point of view and deliberately tries to push a language model, a chat application or an AI agent into unwanted behavior, such as leaking confidential data, bypassing guardrails or misusing connected tools. Unlike an automated scan, it starts from a threat model and concrete attack goals. Unlike a classic pentest, it also targets the behavior of the model itself. The result is a report showing which scenarios succeeded, how reliably, and what stops them.
- NIST describes AI red teaming as a structured testing exercise that probes an AI system for flaws and vulnerabilities, often in a controlled environment and together with the developers.
- A pentest asks: where is the application technically vulnerable? An automated LLM scan asks: does the model react to known attack patterns? AI red teaming asks: can an attacker with a specific goal cause real damage in this system?
- An engagement follows a fixed sequence: goals and scope in writing, threat model, attack scenarios, execution with repeated attempts, assessment, report and retest.
- MITRE ATLAS, the OWASP GenAI Red Teaming Guide and the NIST AI RMF with its generative AI profile (NIST AI 600-1) provide orientation.
What it is
Definition and core idea.
Go to sectionComparison
Pentest, scan and red teaming side by side.
Go to sectionProcess
From scope to retest.
Go to sectionAttack goals
Data leakage, jailbreaks, tool misuse.
Go to sectionFrameworks
MITRE ATLAS, OWASP, NIST, AI Act.
Go to sectionWhen it pays off
Triggers and alternatives.
Go to sectionWhat is AI red teaming?
In its Generative AI Profile (NIST AI 600-1), the US National Institute of Standards and Technology (NIST) defines AI red teaming as a structured testing exercise used to probe an AI system for flaws and vulnerabilities, such as inaccurate, harmful or discriminatory outputs. It often takes place in a controlled environment and in collaboration with the system's developers. Elsewhere in the same document, NIST spells out the purpose: identify what adverse behavior a model or system can show, how it could occur, and stress-test the safeguards in place. NIST also calls AI red teaming an evolving practice.
For a company that builds AI into its own products or workflows, this has a practical consequence. The focus is not the model in isolation but the system you operate: the system prompt, the connected data sources, the tools an agent may call, the permissions it runs with, and the people who interact with it. A red team asks the attacker's question: what can I achieve in exactly this system if I set out to?
Three traits set AI red teaming apart from other kinds of testing:
- Goal-driven. Every engagement starts with attack goals that matter to your business, such as "read another tenant's customer data" or "get the agent to trigger a payment". What gets measured is whether the goal is reachable.
- Creative and adaptive. A red team combines techniques, reacts to the system's answers and follows promising leads. That cannot be fully captured in a fixed test list.
- Statistical, not one-off. Language models can answer the same input differently. An attack that fails once may succeed on the next attempt. Results are therefore documented with repeated runs and a success rate.
AI red teaming vs. AI pentesting vs. automated LLM scans
These terms are not used consistently in the market. Some providers call any AI security test "red teaming", others mean only content-safety testing of a model. The comparison below describes what each type of testing typically delivers. What actually counts is what the proposal and the scope say.
| Aspect | Automated LLM scan | AI pentest | AI red teaming |
|---|---|---|---|
| Core question | Does the model react to known attack patterns? | Which vulnerabilities does the application have, as completely as possible? | Can an attacker reach a specific goal in this system? |
| Starting point | Prebuilt prompt collections | Test catalog covering every area of the application | Threat model and concrete attack goals |
| Approach | Broad, fast, repeatable | Systematic, area by area | Scenario-based, chains weaknesses, adapts |
| Strength | Regression testing after every change | Broad coverage, including classic web and API flaws | Realistic attack paths and their actual impact |
| Limitation | Knows nothing about your data, permissions or business logic | Chained, multi-step scenarios often only at the margins | Not a complete list of all vulnerabilities |
| Output | List of hits, often with many false positives to triage | Findings per test area, with evidence | Attack paths with success rate, impact and countermeasures |
The three are not mutually exclusive. A common setup is automated scans as a continuous task during development, an AI pentest before launch, and AI red teaming where a successful attack would be especially costly. Our LLM pentesting checklist shows what an AI pentest covers in detail. How a classic penetration test runs is described in Penetration Testing: Process, Duration and What It Costs.
A note on automated tools: they are useful in red teaming too, for example to generate many variants of an attack. NIST AI 600-1 explicitly lists red teaming in which generative AI supports human teams and notes that it can be more cost-effective than human red teamers alone. Deciding whether a hit actually means harm in your system, however, remains the job of people who understand that system.
How an AI red teaming engagement works
An engagement is a project with clear boundaries, not open-ended poking around. The six steps below have proven themselves in practice and are consistent with the holistic approach described in the OWASP GenAI Red Teaming Guide.
1. Put goals, scope and authorization in writing
Three questions come first. What is being tested (a chat application, an internal assistant, an agent with tools, a specific workflow)? In which environment (a test system with realistic but non-critical data, or production)? And what is explicitly out of scope? Add test accounts with different roles, an emergency contact and rules for what happens immediately after a critical finding. No testing starts without written authorization from the system owner. This also applies to AI systems that run on third-party platforms, whose terms of use set their own limits.
2. Build the threat model
The threat model describes who might attack your system and what they want to achieve. Typical attacker roles are an anonymous visitor in the chat, a logged-in customer after someone else's data, an employee with overly broad permissions, and a third party who plants text in documents, emails or web pages that your system will later read. The attack goals to be tested follow from these roles, the data and the tools. Where the trust boundaries of an agent lie is covered in our overview AI Agent Security.
3. Plan attack scenarios
For each goal, the team plans concrete scenarios and maps them to a shared vocabulary so findings stay comparable. MITRE ATLAS, with its identifiers for tactics and techniques, works well for this (more below). A scenario records the attacker role, entry point, techniques used, expected impact and how success will be recognized.
4. Execute and repeat
Execution combines manual attacks with tool-assisted variants. Every scenario runs several times, including slightly modified versions, across multiple conversation turns and through every channel by which text reaches the model. When an attack succeeds, it is documented so your team can reproduce it: exact inputs, response, triggered action, timestamp and model version.
5. Assess
A finding is not an odd answer, it is the impact. A model that writes a rude sentence is a different matter from an agent that sends data to an outside address. Findings are therefore rated by impact (which data, which action, what damage), preconditions (anonymous or account required) and reliability (how many out of ten attempts succeed).
6. Report and retest
For each attack path, the report describes the goal, the steps, the evidence, the success rate, the ATLAS mapping and a concrete proposal for closing the path. A management summary puts the results in context. After the fixes, a retest checks whether the paths are closed. Because a model swap or a new system prompt alone can shift behavior, the report should also set a rule for when to test again.
Typical attack goals
Which goals get tested follows from the threat model. Four groups show up in almost every engagement. The ATLAS identifiers in the table come from the ATLAS data published by MITRE (September 2026 release).
| Attack goal | Core question | Examples from MITRE ATLAS |
|---|---|---|
| Data leakage | Do confidential data, other users' records or the system prompt reach the attacker? | AML.T0057 LLM Data Leakage, AML.T0056 Extract LLM System Prompt, AML.T0024 Exfiltration via AI Inference API |
| Jailbreak | Can the guardrails of the model or the application be bypassed? | AML.T0054 LLM Jailbreak |
| Hijacking via injected text | Does the system follow instructions hidden in inputs, documents or web pages? | AML.T0051 LLM Prompt Injection |
| Tool misuse by agents | Can the attacker get an agent to use one of its tools, with its permissions, on the attacker's behalf? | AML.T0053 AI Agent Tool Invocation |
Data leakage
This is the most common business risk. An assistant with access to customer data, contracts or internal wikis can hand content to people who should not see it. The red team checks whether permission filters in the data layer hold, whether other users' records can be reached through clever questions, and whether data can leave through side channels such as links, images or tool calls.
Jailbreaks
A jailbreak gets the model to ignore its own safety rules or the operator's instructions. For a business, it matters most when those rules are meant to protect commercial boundaries: no binding commitments, no discounts, no answers on certain topics. The red team checks whether these boundaries live only in the prompt or are also enforced technically.
Prompt injection as the lever
Prompt injection is rarely the goal itself, but often the way there. The indirect form in particular, where instructions hide in documents, emails or web pages, opens a door for attackers who do not even need an account. How to test for it systematically is covered in our test plan Prompt Injection Testing.
Tool misuse by agents
As soon as an agent calls tools, a wrong answer becomes a wrong action: an email to the wrong address, a database write, a ticket, a payment. The red team checks which tools are reachable with which permissions, whether risky actions require confirmation, and whether several harmless steps can be chained into a harmful one. The related risk classes are covered by the OWASP Top 10 for Agentic Applications.
Frameworks for orientation
None of the following frameworks is a test standard that a red teaming exercise can "pass". They provide a shared vocabulary, a structure for the engagement and a link to risk management.
| Framework | Publisher | How it helps in an engagement |
|---|---|---|
| MITRE ATLAS | MITRE | Adversary tactics and techniques against AI systems, with identifiers for classifying scenarios and findings |
| GenAI Red Teaming Guide | OWASP GenAI Security Project | Structure and scope of an engagement for LLM and GenAI applications |
| AI RMF 1.0 and NIST AI 600-1 | NIST | Feeding results into AI risk management, types of red teaming |
MITRE ATLAS
ATLAS stands for Adversarial Threat Landscape for AI Systems. MITRE describes it as a public knowledge base of adversary tactics, techniques and procedures targeting AI systems. Besides tactics and techniques, the published data includes mitigations and case studies, and refers to the well-known MITRE ATT&CK framework in several places. In an engagement, ATLAS is most useful as a classification system: every finding gets an identifier that your security team, your vendors and future tests all understand the same way.
OWASP GenAI Red Teaming Guide
The OWASP GenAI Security Project published the GenAI Red Teaming Guide on 22 January 2025. It describes a holistic approach across four areas: model evaluation, implementation testing, infrastructure assessment and runtime behavior analysis. According to the announcement, it also addresses the challenges of agentic AI and is written for security professionals and AI/ML engineers alike. For your planning, it is a useful yardstick for whether a proposal covers only the model or the whole system. The risks themselves are described in the OWASP Top 10 for LLM Applications.
NIST AI RMF and NIST AI 600-1
NIST released the AI Risk Management Framework (AI RMF 1.0) on 26 January 2023. It is intended for voluntary use and organizes AI risk management into four functions: Govern, Map, Measure and Manage. On 26 July 2024, NIST followed with NIST AI 600-1, a cross-sectoral profile of the AI RMF for generative AI. It recommends red teaming in several places and distinguishes four types, depending on the use case:
- General public: users without particular AI or technical expertise who use the system the way it will be used in practice.
- Expert: specialists with knowledge of the domain, for example medicine, biotech or cybersecurity.
- Combination: experts who, for example, verify or modify prompts written by non-experts.
- Human / AI: red teaming in which generative AI works alongside human teams.
NIST also stresses that the quality of the results depends on the background and expertise of the red team, and that results should receive additional analysis before they feed into decisions. For a company, that means the results of an engagement belong in the Measure part of your risk management, not in a drawer.
What the EU AI Act says about adversarial testing
Article 55(1)(a) of the AI Act (Regulation (EU) 2024/1689) requires providers of general-purpose AI models with systemic risk to perform model evaluation that includes conducting and documenting adversarial testing of the model. Whether and which AI Act obligations apply to your organization is a legal question. This article is not legal advice. For an overview, see AI Act and AI Governance.
When does a company need AI red teaming?
Not every AI application needs a red teaming engagement. An internal writing assistant with no access to confidential data and no tools is usually well served by a solid pentest and regular automated tests. AI red teaming pays off when at least one of the following applies:
- The system accesses personal, confidential or tenant-specific data.
- An agent can act: send emails, change records, execute code, place orders or trigger payments.
- The system reads content that third parties can influence, such as incoming emails, uploaded documents or web pages.
- The application is publicly reachable, and misbehavior would be visible to your customers or damage your brand.
- An enterprise customer, an investor or an auditor asks for solid evidence on the security of your AI features.
- A major change is coming: a new model, new tools, new data sources, or the step from assistant to agent.
For smaller teams, a staged approach is realistic: first cover the basics with the checklist and automated tests, then an AI pentest before launch, and a red teaming engagement for the two or three scenarios that would genuinely keep you up at night.
What AI red teaming does not do
A red teaming engagement shows which of the tested attack paths succeeded at the time of testing. It is not a guarantee that no others exist, and it is not a certification. Because models, prompts, tools and data sources change, the result is a snapshot. It delivers the most value when the scenarios found become permanent regression tests in your development process and you retest after major changes.
Can your AI application withstand a real attack?
We run AI red teaming for chat applications, internal assistants and AI agents, within a scope agreed in writing and built around your attack goals: data leakage, jailbreaks, injected instructions and tool misuse, across multiple roles and with repeated attempts. Every successful attack path comes with reproducible evidence, a success rate, an ATLAS mapping and a concrete fix. Learn more about AI pentesting & MCP security.
Request AI red teamingCyberSec42 is an independent technical security assessment, not an accredited certification body, and does not provide legal advice. CyberSec42 is not affiliated with MITRE, OWASP or NIST.
Frequently asked questions
What is the difference between AI red teaming and an AI pentest?
An AI pentest examines an application as completely as possible across all test areas and produces a list of vulnerabilities. AI red teaming starts from concrete attack goals and tests whether an attacker can reach them in your system, including through chained steps. The two complement each other.
Is an automated LLM scan enough?
Automated scans make sense as regression tests after every change. But they know nothing about your data, permissions or business logic and cannot judge whether a hit causes harm in your system. For systems with sensitive data or agents that take actions, they do not replace targeted testing by people.
Does AI red teaming attack the language model itself?
For companies using a third-party model, the focus is your own system: system prompt, data connections, tools, permissions and outputs. The model's behavior is tested to the extent it affects your application. Testing the model itself is the model provider's job and is limited by its terms of use.
What is MITRE ATLAS?
ATLAS (Adversarial Threat Landscape for AI Systems) is a public MITRE knowledge base of adversary tactics, techniques and procedures against AI systems, complemented by mitigations and case studies. In red teaming, it serves as a shared vocabulary for classifying scenarios and findings.
Does the EU AI Act require AI red teaming?
Article 55(1)(a) of the AI Act requires providers of general-purpose AI models with systemic risk to conduct and document adversarial testing. Whether and which obligations apply to your organization is a legal question. This article is not legal advice.
How often should AI red teaming be repeated?
There is no fixed interval. A new test makes sense after major changes, such as a model swap, new tools, new data sources or new permissions for an agent. The scenarios found should also run as automated regression tests.
Further reading
- LLM Pentesting Checklist: Testing an AI Application Before Launch
- Prompt Injection Testing: Test Plan, Test Cases and Scoring
- The OWASP Top 10 for LLM Applications 2026, Explained in Practice
- AI Agent Security: A Practical Overview
- The OWASP Top 10 for Agentic Applications, Explained in Practice
- Service: AI pentesting & MCP security
Sources
- NIST AI 600-1: Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (July 2024)
- NIST AI 100-1: Artificial Intelligence Risk Management Framework (AI RMF 1.0)
- NIST: AI Risk Management Framework
- MITRE ATLAS
- MITRE ATLAS Data (GitHub, release 2026.09)
- OWASP GenAI Security Project: GenAI Red Teaming Guide
- OWASP GenAI Security Project: Announcing the OWASP Gen AI Red Teaming Guide
- EUR-Lex: Regulation (EU) 2024/1689 (Artificial Intelligence Act)