Get Started With AI Red-Teaming

The double-edged AI revolution has brought tangible benefits for IT but also opportunities for threat actors to deploy more sophisticated and varied cyberattacks. Our research offers a structured starting point for IT and security leaders looking to employ AI red-teaming exercises to identify and mitigate vulnerabilities in their AI models, securing their organization鈥檚 future growth and innovation.

AI red-teaming can be an effective stress test, but it is relatively new 鈥� to maximize its potential for mitigating risk, organizations must approach it differently than traditional red-teaming exercises. IT and security leaders must be clear about their AI red-teaming goals and involve the right people, processes, and technology to ensure their effectiveness.

1. Define your goals early.

Hunting for security risks doesn't need to be a fishing expedition. Setting out a specific scope for your red-teaming exercise and aligning it with your organization鈥檚 security frameworks will ensure your efforts are effective at uncovering AI-based vulnerabilities.

2. Harness the power of collaboration.

Adversarial testing of your AI systems is more complex than in traditional red-teaming exercises and, as such, requires a larger and more diverse group. A multidisciplinary approach, involving experts in AI, compliance, cybersecurity, data, and ethics, will ensure you get the most out of your red-teaming exercises.

3. AI red-teaming shouldn鈥檛 be your only tool.

AI red-teaming can be a tremendously valuable risk detection and mitigation tool, but it is only one aspect of nurturing a safe and secure AI environment. Organizations must develop strong governance practices and enhanced security measures to effectively secure their AI technologies in the long term.

Use this research as a starting point for your AI red-teaming strategy

Our research offers guidance on understanding the benefits of AI red-teaming and taking a methodical approach to planning a red-teaming framework. Ensure you have the right goals, tools, and team to ensure an optimal approach that detects risks to your AI technology, shields it from threats, and allows it to operate securely in your organization.

Define the scope of your AI red-teaming exercise, including the systems being tested and the type of testing conducted.
Develop your framework by identifying the people and processes to involve while ensuring alignment with best practices.
Assemble what you need by selecting the tools, technologies, and vendors that will be most valuable in developing an effective AI red-teaming exercise.
Establish metrics and KPIs to assess the effectiveness of your AI red-teaming practice.

Get Started With AI Red-Teaming 91制片厂 & Tools

1. Get Started With AI Red-Teaming Deck 鈥� A valuable guide for designing a red-teaming framework optimized for your unique AI landscape.

Use this deck to plan your approach to AI red-teaming, execute your AI red-teaming exercise in a way that makes sense for your organization, and build the right guardrails to protect your AI models from threat actors.

Gain insight into how bad actors target AI systems and models and how AI red-teaming offers safeguards traditional red-teaming does not.
Review AI security regulations emerging in different jurisdictions.
Consider in-house vs. outsourced solutions, with a high-level overview of tools, technologies, and metrics to consider for your organization鈥檚 use.
Introduce yourself to commonly used red-teaming frameworks and guidelines such as MITRE ATLAS, Microsoft AI Red Teaming, and the NIST AI RMF Playbook.

Get Started With AI Red-Teaming

Develop your AI red-teaming practices to effectively mitigate AI and security risks.

Analyst perspective

Ensure effective AI red-teaming with a strategic plan.

The advent of AI technology has provided profound benefits for organizations to enhance productivity, enable growth and innovation, and strengthen their security posture. But AI has also brought its challenges, with threat actors leveraging the technology to improve the sophistication of their attacks and increase their attack vectors. Hence, organizations need to use security best practices to protect themselves against the evolving threats of AI. This includes conducting AI red-teaming exercises to identify vulnerabilities within their AI models and systems and effectively mitigating those threats through appropriate guardrails. However, implementing effective AI red-teaming exercises requires a strategic plan that highlights the main goal and identifies the right people, process, and technology to achieve an organization's desired outcome. Your plan will provide assurance in your AI systems securely furthering the growth and innovation of your organization.

A picture of Ahmad Jowhar
91制片厂 Analyst, Security & Privacy
91制片厂

Ahmad Jowhar
91制片厂 Analyst, Security & Privacy
Info-Tech 91制片厂 Group

Executive summary

Your Challenge

The advent of AI technology introduced new security and privacy risks, which exacerbated the security threat landscape.
Many organizations don't have a plan or framework to combat AI threats through red-teaming strategies.
Without proper guardrails to test and mitigate risks in AI systems, the organization's risk exposure will continue to proliferate.

Common Obstacles

The organization needs guidance to develop a plan to red-team its AI systems.
Difficulty in prioritizing AI systems to red-team results in technologies being deployed with vulnerabilities that are exploitable by threat actors.
A lack of resources hinders an organization's ability to conduct a comprehensive AI red-teaming exercise.

Info-Tech's Approach

Gain a better understanding of the current AI red-teaming landscape, including associated AI-based threats.
Obtain insights into best practices for developing a red-teaming framework that meets your organization's needs.
Understand the critical elements of AI red-teaming, including AI threat modeling and testing, as well as associated tools and techniques.

Info-Tech Insight

Proactively prepare for potential AI vulnerabilities and risks by establishing a strategic framework to identify and mitigate AI-based threats and build resilient AI systems.

Your challenge

The exponential growth of AI technology poses various challenges to defending against its threats.

The emergence of AI technology has resulted in challenges to defending against AI-based threats. As the technology continues to evolve, threats will continue to exacerbate an organization's attack vectors.
Many organizations already struggle to respond to security threats due to resourcing constraints and constant shifting in the threat landscape. The sophistication of AI-based threats will further the security challenges that organizations face.
With more than 80% of security leaders not confident in traditional solutions to defend against AI-based threats (Darktrace, 2024), a new approach is required to respond to the evolving threats of AI.
Red-teaming for AI systems and models is an emerging technique that will enhance security posture, but organizations struggle with a lack of guidance on developing their red-teaming framework.
Implementing a plan that will support the development of a framework to red-team AI technologies will ensure appropriate measures are in place to deploy an effective AI red-teaming practice.

71%	71% of organizations are experiencing severe impact from AI-powered cyberthreats.
60%	60% of organizations are not adequately prepared to defend against AI-powered threats and attacks.
85%	85% of security leaders don't believe traditional solutions can defend against AI-based threats.

Source: Darktrace, 2024

Get started with AI red-teaming

Proactively prepare for potential AI vulnerabilities and risks by establishing a strategic framework to identify and mitigate AI-based threats and build resilient AI systems.

Developing an effective AI red-teaming framework through a strategic plan will ensure you create a comprehensive strategy to identify and mitigate threats and vulnerabilities of AI systems.

Develop a framework that will enable the continuous monitoring and improvement of your AI technologies

Strengthen your security posture by protecting your AI technologies

Proactively prepare for potential AI vulnerabilities and risks by establishing a strategic framework to identify and mitigate AI-based threats and build resilient AI systems.

Define your goals early

Set the scope for your red-teaming exercise and align it with frameworks to ensure the effective testing of your AI models and systems.

Different goals require different methodologies

Red-teaming for AI is not the same as red-teaming for typical security systems 鈥� it requires a multidisciplinary approach.

Leverage industry standards

Discover and manage your AI risks by following red-teaming best practices when testing your AI technologies.

Harness the power of collaboration

Developing an effective AI red-teaming practice requires a multidisciplinary approach that includes a diverse set of stakeholders, a comprehensive process, and tools and technologies to assist in adversarial testing.

Securing AI technologies goes beyond AI red-teaming

AI red-teaming is only one part of developing a safe and secure AI practice. Strong governance and enhanced security measures are also essential to effectively secure AI technologies.

Blueprint benefits

IT/InfoSec Benefits	Business Benefits
Improved understanding of AI red-teaming, its benefits, and the main components it includes Risks associated with AI addressed and mitigated through the development of an AI red-teaming strategy Improved visibility into developing an AI red-teaming framework that aligns with industry standards	Adherence to regulatory standards on leveraging AI red-teaming to address and mitigate AI risks Increased productivity of AI technologies through the remediation of vulnerabilities Reduction of financial and reputational risks through implemented safeguards and measures to mitigate the likelihood and impact of AI-based threats

AI red-teaming vs. traditional red-teaming

What is AI red-teaming and how does it differ from traditional red-teaming?

AI red-teaming is the process of challenging AI systems to identify vulnerabilities, risks, and potential biases that are not apparent during regular development.
AI red-teaming stems from traditional cybersecurity red-teaming practices that saw professionals attempt to attack a system or network with the aim of identifying and exploiting its vulnerabilities.
This practice has been adapted into the context of AI, and hence AI red-teaming is similar to traditional red-teaming, but there are major differences between both practices.

Scope/Objective Focus: Although traditional red-teaming focuses on identifying security vulnerabilities through real-world simulated attacks, AI red-teaming focuses on targeting vulnerabilities within AI systems and machine learning (ML) models.

Methodology: Traditional red-teaming follows attack frameworks that employ techniques that mimic human attacker methods such as penetration testing and social engineering (e.g. MITRE ATT&CK). AI red-teaming uses ML techniques to uncover problematic model behavior such as data poisoning and prompt injection.

People: Traditional red-teaming includes mainly security personnel and subject matter experts of specific systems. AI red-teaming requires a multidisciplinary team with experts in AI, compliance, cybersecurity, data, and ethics.

Technology: Traditional red-teaming leverages different technologies depending on the scope of the exercise, such as Metasploit for pen testing and Wireshark for network scanners. AI red-teaming could leverage ML tools such as CleverHans and PyRIT to help generate adversarial attacks.

AI security regulations

Many nations are implementing AI red-teaming measures to ensure the safe use of AI technologies and mitigate security risks for their citizens.


USA	Canada	EU	UK	Australia
Establishing AI governance laws has been of utmost importance for the US government.	Canada is in the process of developing a comprehensive regulatory framework for AI, with a code of conduct currently in effect.	The EU AI Act contains components on red-teaming as part of its strategic regulatory approach.	The UK's National Cyber Security Centre (NCSC) has emphasized the importance of red-teaming as part of the broader scope of AI security.	The Australian Cyber Security Centre (ACSC) provides guidance for the secure development and use of AI systems.
The AI Bill of Rights emphasizes the importance of external red-team testing for key AI risks. NIST formed the TRAINS taskforce to govern the safety and security of AI technologies.	Canada recommends adversarial testing such as red-teaming to identify vulnerabilities in AI systems. It recommends developers engage with third parties to conduct red-teaming exercises.	The act will require general-purpose AI models to undergo red-team testing throughout their product lifecycles. Developers of AI technologies must also disclose measures used for red-team testing.	The NCSC promotes the assessment of current safeguards to ensure effectiveness through red-teaming efforts. It highlights principles of mitigating ML threats through red-teaming efforts.	The ASCS emphasizes continuously monitoring AI systems through adversarial testing to assess model resilience against compromises. It provides best practices on securing AI through frequent audits and testing.

LLM threat model

Leverage this threat model to better understand how adversaries are targeting AI systems and models and their impacts on AI technologies.

Examples of threats against Gen AI

Types of attacks using the LLM threat model

Prompt Injection	Evasion	Data Exfiltration	Inversion Attacks	AI Model Theft
Using cleverly written prompts to make an AI system comply with prohibited requests.	Using an input that the AI system misinterprets, causing it to malfunction (i.e. go against training).	Stealing data to better understand how an AI system works (e.g. training data).	Using inputs and measuring outputs to determine whether they contain sensitive information about the model or training data.	Stealing the file containing the AI model.
This is a common attack against Gen AI using an LLM. An acceptable use policy and user monitoring are recommended defense measures.	Evasion is often used against AI systems designed for image or pattern recognition. It is rarely a significant risk for Gen AI, unless the system combines language and image processing.	This often precedes an input attack so that it can be better executed. It is a risk for all types of AI but can be mitigated using standard data protection techniques.	The goal is to rebuild the AI model or the data it contains via outputs. These attacks can affect any AI system that includes sensitive data, especially if it is included in outputs.	Theft may accompany an input attack or be motivated by other factors, as in other forms of data exfiltration Strong data protection controls can create a perimeter around the AI system.

Examples of threats against Gen AI

Types of attacks using the LLM threat model

Data Poisoning	Weaponized AI model	Sponge Attacks
Gaining access to AI model training data and altering it with the intention of corrupting it.	Compromising an AI model with malicious code (e.g. ransomware).	Entering a series of difficult-to-process inputs into an AI system to slow down its processing speed and increase energy consumption.
This attack is enabled by a lack of good training data and trust mechanisms for public data sets. Defend with a strong perimeter around the AI model, including complete encryption and intrusion detection.	Often used against AI systems designed for image or pattern recognition. Rarely a significant risk for Gen AI, unless the system combines language and image processing.	This technique is similar to a DDoS attack. Defend against sponge attacks by setting an energy consumption threshold that will reset the system if met.

Download the blueprint Address Security and Privacy Risks for Generative AI.

Develop: AI kill chain

Understand the types of tactics attackers use throughout different phases of an AI attack.

The MITRE ATLAS Matrix provides detailed insights into the tactics leveraged by attackers so organizations can respond to and mitigate threats.

Examples include:

Tactic	Technique	Description	Mitigations
LLM Prompt Injection	Initial Access	Using cleverly written prompts to make an AI system comply with prohibited requests.	Implementing generative AI guardrails and guidelines (e.g. input validation, model constraints).
User Execution	Execution	Leveraging social engineering techniques to trick users into executing malicious code.	Advanced user training and implementing behavior prevention on endpoints and network intrusion systems.
Data Poisoning	Persistence	AI code generation involves fine-tuning a pretrained LLM using special data sets.	Implementing data validation and anomaly detection to identify and remove suspicious data points.
LLM Jailbreak	Privilege Escalation	Leveraging prompt injection to evade security safeguards and mechanisms, enabling the generation of harmful content.	Implementing Gen AI guardrails such as monitoring the input and output of a model and ensuring safety filters are embedded on both the input and output of the prompt.
Evade ML Model	Defense Evasion	Adversaries evade detection by AI model modifying inputs or exploiting vulnerabilities in the model's design.	Conducting adversarial training that will expose evasion techniques, implementing robust mechanisms to validate inputs, conducting regulatory review, and updating models to defend against new techniques.

Source: MITRE ATLAS, 2024

AI red-teaming framework

Organizations have developed various AI red-teaming frameworks and guidelines to help others strategically red-team their AI technologies.

Microsoft AI Red-Teaming	MITRE ATLAS	NIST AI RMF Playbook	OWASP Gen AI Red-Teaming Guide
Provides insights into safeguarding AI systems by identifying and mitigating vulnerabilities.	Presents detailed insights into adversary tactics and techniques for AI-enabled systems.	Identifies key risk management components for organizations implementing AI technologies.	Provides practical strategies for organizations to enhance the security and safety of their AI systems.
Takes a multidisciplinary approach by providing frameworks that integrate threat modeling with advanced testing techniques.	This knowledge base is mapped to the MITRE ATT&CK framework for threat assessment and mitigation.	Provides a framework for managing AI risks grouped into four functions, along with best practices and guidance to ensure the security of AI applications.	Offers guidelines for developing an AI red-teaming strategy that highlights risks, threat modeling, and techniques to effectively strengthen an organization's AI practices.

Info-Tech Insight

Each AI red-teaming framework provides unique benefits to help build your AI red-teaming practice. Having a defined scope will help you select the right framework to meet your security goals.

Select: AI red-teaming framework

Determine which framework will help build your AI red-teaming practice.

Capabilities	Microsoft Red-Team	MITRE ATLAS	NIST AI RMF Playbook	OWASP Gen AI Red-Teaming Guide
Provides security best practices for AI development	鉁�		鉁�	鉁�
Provides insights into adversarial attacks and training	鉁�	鉁�		鉁�
Provides insights into evaluating and testing AI models	鉁�		鉁�	鉁�
Provides insights and guidance on addressing AI risks	鉁�	鉁�	鉁�	鉁�
Provides insights into tools and techniques to defend against adversarial attacks	鉁�			鉁�

Info-Tech Insight

Whether you outsource or use in-house AI red-teaming practices, align your framework and approach to these industry standards to ensure you are following best practices to effectively mitigate AI risks.

In-house development vs. outsourcing

Decide which approach will meet your organizational and AI red-teaming needs.

Pros/Cons	In-House	Outsourcing
鉁�	Enables the development of tailored exercises to meet specific organizational needs and alignment with regulatory requirements.	This cost-effective approach would be appropriate for short-term projects.
鉁�	Mitigates risk of data leakage and promotes the confidentiality of data.	Access to experienced professionals enables easy scaling based on scope and goals.
鉁�	Allows full control throughout the lifecycle, enabling easier integration with existing process and quick adaptation to findings and remediation.	External experts handle red-teaming efforts, allowing internal teams to focus on core activities.
X	Requires significant investments in people and technology.	Less customization and control over process results in alignment challenges.
X	Difficult to scale practices with limited resources and challenging to find and retain talent.	Outsourced red-teaming increases the risk of data breaches or leaks.

Info-Tech Insight

The innovation of AI tools and technologies will result in vendors continuously emerging with new and improved solutions. ensure you are up to date with the latest technologies and associated metrics to assess their efficacy in defending against adversarial threats.

Tools, technologies, and metrics

Leverage the right tools, technologies, and metrics to ensure the effective development of your AI red-teaming framework.

Identifying the right people and process is only part of the framework for defending against AI-based threats. Selecting the right tools and technologies will ensure adversarial testing is conducted in an efficient and scalable approach that enables continuous monitoring and improvements to the security of AI technologies.
Likewise, establishing applicable metrics will enable the evaluation of AI red-teaming outcomes through a quantifiable approach to assess the effectiveness of the testing. Furthermore, metrics will allow for continuous process improvement and data-driven decision-making regarding enhancing the security of AI systems.

Below are examples of tools, technologies, and metrics that can be assessed and leveraged when developing your AI red-teaming framework.

Tool	Description	Technology	Description	Metric	Outcome
Microsoft PyRIT	This open-source framework is used to identify and mitigate Gen AI risks.	AWS	Red-team testing that focuses on defending against various AI-based threats is conducted on Amazon's Nova AI models.	Frequency and severity of vulnerabilities discovered	Decrease in events after mitigating the vulnerabilities
AI Fairness 360	This open-source toolkit was developed by IBM to detect and mitigate biases in ML models.	IBM	IBM's X-Force Red team provides testing services for AI systems and models including Gen AI application and model safety and security testing.	Number of successful adversarial attacks	Decrease in attacks after appropriate safeguards are implemented
CleverHans	This Python library benchmarks ML system vulnerabilities against examples of attacks by adversaries.	HiddenLayer	Automated AI red-teaming allows for vulnerability testing on AI solutions through simulated attacks.	Compliance with regulatory standards	Increase in adhering to compliance requirements due to guardrails and controls put in place

Establish: Checklist for developing your red-teaming plan

Leverage this checklist when developing your AI red-teaming framework.

Define Scope
Identify the main goal of your AI red-teaming exercise.
Develop Framework
Identify the right people and process and ensure alignment with existing AI frameworks.
Select Technology/Vendor
Identify the right tools, technologies, and vendors that would assist in developing an effective AI red-teaming exercise.
Establish Metrics
Define key indicators for a successful red-teaming exercise that highlights an organization's current approach for mitigating AI-based threats and areas for continuous improvement.

Related Info-Tech research

Address Security and Privacy Risks for Generative AI

Leverage this Info-Tech blueprint for guidelines on determining risks that apply to your generative AI use cases and develop a plan to implement necessary improvements to your data security posture.

research

A picture of Altaz Valani

Altaz Valani
Principal Advisory Director
91制片厂

A picture of Jon Nelson

Jon Nelson
Principal Advisory Director
Info-Tech 91制片厂 Group

Get Started With AI Red-Teaming visualization

Develop your AI red-teaming practices to effectively mitigate AI and security risks.

About Info-Tech

91制片厂 is the world鈥檚 fastest-growing information technology research and advisory company, proudly serving over 30,000 IT professionals.

We produce unbiased and highly relevant research to help CIOs and IT leaders make strategic, timely, and well-informed decisions. We partner closely with IT teams to provide everything they need, from actionable tools to analyst guidance, ensuring they deliver measurable results for their organizations.

What Is a Blueprint?

A blueprint is designed to be a roadmap, containing a methodology and the tools and templates you need to solve your IT problems.

Each blueprint can be accompanied by a Guided Implementation that provides you access to our world-class analysts to help you get through the project.

Table of Contents

You Get:

Get Started With AI Red-Teaming Deck

Talk to an Analyst

Our analyst calls are focused on helping our members use the research we produce, and our experts will guide you to successful project completion.

Book an Analyst Call on This Topic

You can start as early as tomorrow morning. Our analysts will explain the process during your first call.

Get Advice From a Subject Matter Expert

Each call will focus on explaining the material and helping you to plan your project, interpret and analyze the results of each project step, and set the direction for your next project step.