AI Security Testing Guide: What It Is and Why It Matters for AI Risk Posture
AI security testing focuses on the unique vulnerabilities of AI systems. Here’s what it means, how testers carry it out, and why it’s an essential addition to your cybersecurity testing.
Conventional security testing remains an important part of an organisation’s cybersecurity strategy. But when it comes to securing AI systems, traditional testing no longer covers the full scope.
Because AI systems generate responses based on context and probability rather than following only fixed rules, security testing needs to account for these additional attack paths.
AI security testing covers these routes by examining the complex and dynamic vulnerabilities that can arise across AI models, training data, and prompts.
As businesses integrate AI models into their systems and give AI agents more autonomy, this type of testing becomes an important part of responsible AI use.
What is AI security testing?
AI security testing is the process of assessing an AI system for vulnerabilities, unsafe behaviour, and ways an attacker could manipulate it. Testers simulate potential attacks to uncover vulnerabilities and recommend remediation.
Effective AI security testing covers the full system:
Models and prompts
Assessing different AI models, their prompts, inputs, and system instructions to determine whether an attacker can manipulate them.
Retrieval-augmented generation (RAG) and data sources
Examining what data the AI can access and whether an attacker could trick it into exposing information that it shouldn’t.
APIs and integrations
Testing the different APIs, integrations, and tools that connect to the AI to discover vulnerabilities across platforms.
Application and infrastructure
Assessing the AI’s surrounding application and infrastructure to identify vulnerabilities in the wider system.
AI security testing doesn’t replace traditional security testing; it adds an extra layer focused on AI-specific risks.
How is AI security testing different from traditional security testing?
AI security testing examines how AI systems behave and how attackers can manipulate them, while traditional security testing targets defined vulnerabilities in application code and infrastructure.
Organisations may carry out conventional testing, discover there is no SQL injection, XSS, or serious infrastructure vulnerabilities, and think their application is secure. But the assessment doesn’t expose whether an attacker could manipulate the AI to reveal sensitive information, carry out unauthorised actions, or deliberately influence its outputs in ways that could harm business decisions.
There’s a second difference, and it changes what a test result means. Conventional software produces the same output for the same input, so a vulnerability that’s fixed stays fixed. LLM outputs are probabilistic. The same prompt can produce different responses, and an attack that fails ninety-nine times can succeed on the hundredth. No test can prove an AI system can’t be manipulated. Prompt injection in particular has no complete fix today, because models can’t reliably separate the instructions they’re given from the content they process.
That’s why effective AI security testing asks a different question. Not “can the model be manipulated?” but “when it is, what can an attacker reach?” The answer depends on architecture: what data the model can see, which tools it can call, what permissions those tools carry, and where a human has to approve an action. Those controls can be tested, measured, and fixed.
| Conventional security testing | AI security testing | |
| Primary focus | Applications, code, infrastructure, and known security weaknesses | AI models, behaviour, data, prompts, integrations, and AI-specific weaknesses |
| What testers look for | Vulnerabilities such as SQL injection, XSS, broken authentication, and access control issues | Prompt injection, sensitive data exposure, model manipulation, data poisoning, and excessive agency |
| How systems behave | Generally follows defined rules and logic | Can produce different outputs depending on context, inputs, and model behaviour |
| Testing approach | Testers identify and exploit technical vulnerabilities in the application and infrastructure | Testers probe the AI with adversarial inputs and simulate attacks against the model and its surrounding system |
| Main concern | Can an attacker exploit a weakness in the software or infrastructure? | Can an attacker manipulate the AI, access information, or cause it to take unintended actions? |
| Relationship | Provides the foundation for application and infrastructure security | Adds AI-specific testing to the existing security testing approach |
| What a clean result means | The tested vulnerabilities aren’t present | The tested attacks didn’t succeed, and controls limit the damage if a future attack does |
What are the main AI security vulnerabilities?
The OWASP Top 10 for LLM Applications (2025 edition) identifies key security risks for applications using large language models and generative AI. The following examples show the types of vulnerabilities AI security testing assesses.
Prompt injection
Prompt injection happens when an attacker uses a malicious prompt to manipulate an LLM in harmful ways.
There are two types of prompt injection: Direct prompt injection, where an attacker directly alters the model’s behaviour through a prompt, and indirect prompt injection, which uses external sources such as websites or files to influence the model.
Prompt injection can cause the LLM to disclose sensitive information, produce incorrect or biased outputs, access functions or commands without authorisation, and influence important business decisions.
Sensitive information disclosure
Sensitive information disclosure occurs when an LLM reveals confidential data, such as personally identifiable information (PII), financial details, health records, security credentials, and legal documents.
This can happen if an attacker uses prompt injection to extract information the AI can access through its connected data sources, RAG systems, tools, or other integrations. It can also happen if the system lacks strict output filters or boundary controls between users.
Supply chain vulnerabilities
Supply chain vulnerabilities come from the third-party components an AI system depends on: pre-trained models, fine-tuning adapters, datasets, and software libraries. A model downloaded from a public repository can contain hidden backdoors or malicious code, and a compromised dataset can affect every model trained on it.
Most enterprise AI systems are built on models the organisation didn’t train and can’t fully inspect. Verifying where models and data come from is part of the security assessment.
Improper output handling
Improper output handling is when an application trusts and processes content generated by an LLM without properly validating, encoding, or sanitising it first.
An attacker can manipulate the model into generating malicious content, which the application may then process as a command or executable input.
This can result in vulnerabilities such as SQL injection, remote code execution, or cross-site scripting (XSS).
Data and model poisoning
Data and model poisoning occurs when an attacker introduces malicious or misleading data into the datasets used to train, fine-tune, or embed an AI model, or tampers with the model itself. This can affect the model’s accuracy, behaviour, and outputs.
Attackers can also manipulate data sources that an AI system relies on, such as information retrieved through RAG. In some cases, attackers can introduce hidden backdoors that only activate when the system encounters a specific trigger, making these vulnerabilities difficult to detect.
Excessive agency
Excessive agency refers to when an AI model has more functionality, permissions, or autonomy than it should, leading to harmful actions.
Attackers can exploit excessive agency through techniques such as prompt injection, causing the AI to misuse connected tools or perform actions outside its intended scope. This can lead to unauthorised transactions, data deletion, and access to sensitive information.
AI agents are more at risk of excessive agency attacks because they have autonomous capabilities and can use tools or take actions with less direct human intervention.
Vector and embedding weaknesses
RAG systems store documents as embeddings in a vector database and retrieve the most relevant ones to answer a query. If the vector database doesn’t enforce the same access controls as the original documents, users can retrieve content they shouldn’t see. For example, a board paper or HR file could surface in a general employee assistant.
Attackers can also plant documents containing hidden instructions in a knowledge base, which the AI later retrieves and follows. In some cases, they can partially reconstruct sensitive source text from the stored embeddings themselves.
For enterprises deploying RAG over internal documents, this is often the most relevant category on the list.
AI security testing guide: How does it work?
AI testing helps organisations identify vulnerabilities affecting their specific systems. It follows a similar methodology to traditional testing: Identification, simulation, assessment, and remediation.
Step one: Map the AI attack surface
First, testers establish how the AI system works, what data it can access, what instructions it follows, and which external systems it connects to.
This could include the LLM, prompts and system instructions, RAG and data sources, APIs, tools, authentication, and the surrounding application and infrastructure.
Step two: Identify threats and abuse cases
Next, testers identify ways an attacker could manipulate the AI or abuse the systems it connects to.
They can use OWASP guidance and testing methodologies as a reference to identify potential vulnerabilities such as prompt injection, sensitive information disclosure, improper output handling, data poisoning, and excessive agency.
Step three: Simulate attacks
Testers simulate real-world attacks to see whether they can manipulate the model, extract information, bypass restrictions, or cause unintended actions.
They test input and output scanning, least-privilege controls, tool permissions, sensitive data in the model’s context, and human approval for consequential agent actions.
Step four: Assess the impact
Finding a vulnerability isn’t enough. Testers need to establish what an attacker could actually achieve if they exploited it.
For an AI system, that might mean accessing another user’s data, influencing an important decision, triggering an unauthorised tool call, or using the AI to reach another system.
Because no test can rule out manipulation entirely, this is the step that matters most. The goal is to show how far a successful attack can travel, and where the architecture stops it.
Step five: Report findings and recommendations
After testers complete the assessment, they document the vulnerabilities and their business impact, and provide practical steps to remediate them.
Step six: Remediate and retest
The organisation applies the recommended remediation steps, and testers then retest the system to confirm that the vulnerabilities have been addressed. They also verify that the changes haven’t introduced new security issues.
Because AI behaviour is probabilistic and changes with every model update, a single assessment is a snapshot, not a guarantee. Security testing should continue after the initial assessment.
AI security audits should be part of the organisation’s software development lifecycle (SDLC) and machine learning operations (MLOps), covering AI development, deployment, and ongoing maintenance. Integrating security testing throughout these stages helps organisations identify and address vulnerabilities as their AI systems evolve.
Teams should also retest after significant changes, such as updating the model, changing system prompts, adding new data sources, connecting new tools, or increasing an AI agent’s permissions. This makes AI security assessments part of a continuous testing approach rather than a one-time evaluation.
How does prompt injection testing work?
Prompt injection testing is a form of adversarial input testing. It assesses whether an AI system can resist malicious instructions that attempt to override its intended behaviour. Testers use crafted inputs to see whether they can bypass restrictions, extract sensitive information, manipulate outputs, or cause the AI to take unauthorised actions.
Testing covers both direct and indirect prompt injection. Direct testing uses malicious instructions entered through the system’s normal input, while indirect testing places those instructions in external content, such as a webpage, document, or tool output, that the AI may later process.
Testers then assess what happens when an attack succeeds and how far it can go within the wider system. For an AI agent, this can include checking whether a successful prompt injection can access sensitive data, call connected tools, or trigger actions beyond the user’s intended permissions.
What is AI red teaming?
AI and LLM red teaming take an attacker-focused approach to security testing, combining different attack techniques to achieve a specific objective. It is particularly relevant for agentic AI security because agents can access data and external tools to take autonomous actions.
For example, if a red teamer wanted to discover if an attacker could cause a data breach, they could start with prompt injection testing to manipulate an AI agent into ignoring its intended instructions. They could then use the agent’s access to an internal database to retrieve sensitive customer information and its connected email tool to send that information to an external address.
AI red teaming services can form part of a broader AI security strategy, alongside vulnerability assessments and other security testing.
Which frameworks and standards can guide AI security testing?
Which frameworks and standards guide AI security testing?
Infinum’s AI security testing follows four OWASP sources:
- OWASP Top 10 for LLM Applications: the core list of security risks for applications built on large language models
- OWASP Machine Learning Security Top 10: risks specific to machine learning systems, including attacks on training data and models
- OWASP AI Testing Guide: a methodology for testing the security of AI systems
- OWASP AI Maturity Assessment: a structure for evaluating how mature an organisation’s AI security and governance practices are
Broader frameworks set the wider context:
- NIST AI Risk Management Framework (AI RMF): a structure for managing AI risk across the system lifecycle
- MITRE ATLAS: a knowledge base of real-world adversary tactics and techniques used against AI systems
- ISO/IEC 42001: the international standard for AI management systems, relevant where testing feeds into AI governance
Frameworks provide a foundation, but effective testing also needs people who combine traditional cybersecurity expertise with an understanding of how models behave, how AI architectures are built, and how the systems around them connect.
Test your AI systems with a team that builds them
AI security testing usually comes from one of two places: penetration testing firms adding AI to an existing methodology, or AI consultancies adding security to an existing offering. Infinum does both as core work. Our AI and data engineering teams build LLM applications, RAG systems, and AI agents for enterprises. Our cybersecurity team attacks them.
That combination shapes how we test. We know where AI systems tend to be weak because we design around the same constraints: how retrieval permissions get set, where an agent’s tool access grows beyond what it needs, and which shortcuts teams take on the way to production. When testing finds a problem, the remediation advice comes from people who know what the fix takes to build.
Infinum has spent 20 years shipping more than 1,000 digital products. We hold NCSC CHECK, CREST, and CREST STAR accreditations, alongside ISO 27001 and ISO 9001 certifications and SOC 2 attestation. Our director of cybersecurity, Martin Walsham, has helped shape security standards through his role on the UK CREST Executive Board.
Book an AI security scoping call with Infinum.
We’ll map your AI attack surface and tell you which systems need testing first.