Guide

LLM Pentesting Checklist: Testing an AI Application Before Launch

By Lucas Weber, CyberSec42 · Updated 28 Oct 2026

An LLM pentest, often called an AI security assessment, tests an AI application the way an attacker would approach it: not just the language model, but above all everything built around it, meaning prompts, connected data, tools and agents, connectors, identities and logging. A practical LLM pentesting checklist covers seven areas: inputs and prompt injection, data leakage and permissions, tool and agent privileges, output handling, supply chain, abuse and cost, and logging. If you have tested those areas before launch and can map the findings to the OWASP lists for LLM and agentic applications, you know where your AI application actually stands.

What an AI pentest covers: the model or the application?

Most companies do not train their own model. They call a provider's model through an API, give it a system prompt, connect it to their own documents and let it call tools. You usually cannot change the model itself. What you control is everything you built around it, and that is where an AI pentest focuses.

In its Top 10 for LLM Applications, the OWASP GenAI Security Project works on the assumption that a model can be fooled. The real question is therefore not whether someone can get the model to say something it should not, but what happens next: which data can it reveal, which actions can it trigger, where does its output end up? A useful scope lists every component separately:

One boundary matters: the test covers your application in your environment, under written authorisation. The model provider's infrastructure is out of scope. Check the provider's terms of use beforehand for anything that applies to testing through their API.

The checklist: seven test areas

The items below are grouped by attack surface rather than by OWASP number. The mapping to OWASP identifiers follows in the next section.

1. Inputs and prompt injection

Prompt injection remains the core risk: text that looks like an instruction redirects the model. Direct injection comes through the chat box; indirect injection comes from content the application reads on its own.

• Every input channel mapped: chat, uploaded files, fetched web pages, emails, tickets, images, audio, tool responses.
• Direct attempts to bypass the system prompt's rules tested, in several languages and encodings.
• Indirect injection tested through every source a third party can write to, such as a document in the knowledge base or an inbound email.
• Tested what a successful injection can achieve: disclosing data, calling tools, manipulating output.

For a full test plan with test cases, see our guide on how to test for prompt injection. For the MCP variant, where poisoned tool descriptions carry the injection, see prompt injection and tool poisoning in MCP.

2. Data leakage and permissions

A model has no notion of access rights. It outputs whatever is in its context. Whether user A can see user B's documents is therefore decided by the application alone, before anything reaches the context.

• Knowledge base retrieval filters by the requesting user's permissions, not by those of a service account.
• Tenant isolation tested with two accounts in different tenants, including in the vector database.
• No credentials, internal addresses or confidential rules in the system prompt, because hidden context can often be coaxed out.
• Memory and conversation history: nothing carries over between users or sessions unless intended.
• Exfiltration paths through output tested, such as links or images whose URL sends data to an outside server as soon as the interface loads them.

3. Tool and agent privileges

Once the model can act, every successful manipulation becomes an action. For agents, this is usually where the greatest damage lies. Our overview of AI agent security covers the fundamentals.

• Each tool has only the privileges its task needs. Read-only agents have no write access.
• Destructive, outbound or paid actions require human confirmation, and the confirmation dialog shows what will actually run.
• Tool call parameters are validated server-side, not in the prompt.
• The agent acts with the user's identity or narrowly scoped tokens, not a broad service account that lets a user reach more than they are allowed to.
• With multiple agents: messages between them are authenticated, and no agent blindly follows another agent's instructions.

4. Output handling

Model output is input from an untrusted source. Passed on unchecked, it recreates classic vulnerabilities through a new route.

• Output that reaches the browser is encoded or sanitised so no script runs.
• Output that flows into database queries, shell commands, file paths or templates is treated like any user input.
• Model-generated code only runs in an isolated environment with no access to secrets or the internal network.
• Markdown rendering restricted so that no external resources load unprompted.

5. Supply chain, models and plugins

• Origin and version of every model documented; provider-side model changes are noticed and retested.
• Self-hosted models and adapters only from traceable sources, in formats that do not execute code when loaded.
• Plugins, extensions and MCP servers reviewed like any third-party software: who maintains them, what privileges they request, are versions pinned? See securing MCP servers.
• Clear rules on who may add content to the knowledge base or training data, because every open write path is a route for poisoned content.

6. Abuse and cost

OWASP lists unbounded resource use as a risk of its own (Unbounded Consumption). With pay-per-use models, abuse does not just create load, it creates a bill.

• Rate limits per user, per key and per IP address, including for unauthenticated access.
• Caps on input length, output length and context size.
• Agent loops bounded: a maximum number of steps and tool calls per task.
• Budget limits and alerts configured with the model provider.
• The application cannot be used as free access to the model for unrelated purposes.

7. Logging and traceability

• Inputs, retrieved sources, tool calls and results are logged with user, timestamp and model version.
• Anomalies such as clusters of refusals, unusual tool calls or cost spikes trigger an alert.
• Logs often contain personal and confidential data. Access, retention and deletion are defined.
• After a test attack, the logs let you reconstruct what happened. The pentest should check exactly that.

Mapping to the OWASP LLM Top 10 and Agentic Top 10

Two lists from the OWASP GenAI Security Project form the usual framework: the OWASP Top 10 for LLM Applications, 2026 edition, and the OWASP Top 10 for Agentic Applications for 2026. The first looks at the model as a component of the application, the second at the agent that plans, acts and remembers. The table shows which entries each of the seven test areas mainly covers. Each entry is explained in the linked guides.

One entry is deliberately left out of the checklist: LLM07:2026 Misinformation. Whether a model gives factually wrong answers is more a quality assurance question than an attack. It becomes relevant to a pentest where users or downstream systems rely on output without checking it.

If you also need a risk management framework: in July 2024 the US National Institute of Standards and Technology (NIST) published the Generative AI Profile (NIST AI 600-1), a companion resource to the AI Risk Management Framework (AI RMF 1.0). It lists information security as a distinct risk of generative AI and names prompt injection and data poisoning explicitly. Among its suggested actions are red-teaming and independent external evaluations. The profile is voluntary and does not replace testing, but it helps place the results within a risk management process.

What a classic web pentest covers, and what it misses

An AI application is first of all a web application with an API. Login, session management, object-level authorisation and interfaces need testing just like any other application, for example along our API pentest checklist. What a classic test does not capture is how the model behaves as part of the application.

There is a methodological difference too: models do not always answer the same way. An attack that fails on the first attempt may succeed on the fifth. An AI pentest therefore repeats attacks and documents how reliably a finding occurs. For how a pentest runs in general, see penetration testing: process, duration and cost.

What the report should contain

An AI pentest report has to be readable for management and actionable for engineering. Look for these elements:

• Tested state. Model and version, system prompt revision, connected data sources and tools, test window and environment. If any of these change, the finding needs reassessing.
• The full attack path per finding. Where the input came from, what the model made of it, which tool or output was affected, and what happened in the end.
• Reproduction. The exact inputs used and how often the attack succeeded across repeated attempts.
• Impact and severity. Tied to your data and workflows, not in the abstract.
• Mapping. The matching OWASP identifiers, so findings stay comparable and trackable.
• System-level recommendations. Restrict privileges, add confirmation, encode output, filter before the context. "Improve the prompt" on its own is rarely a robust fix.
• What was not tested. What was explicitly out of scope or not covered for lack of time.
• Retest. Confirmation that the fixes work.

To see what such a report looks like in practice, take a look at our sample report on a tested MCP integration.

How to prepare your team

The better the preparation, the more testing time goes into real attack paths instead of access issues.

• Written scope. Which application, which environment, which components from the table above, and which tools may actually be executed. Write and outbound actions are best tested in a staging environment.
• Test accounts. At least two accounts per role and, for multi-tenant applications, accounts in two different tenants. That is the only way to check whether data crosses boundaries.
• Test data. Synthetic data instead of real customer data, plus a few documents carrying unique marker strings. If a marker later shows up in the wrong place, the leak is proven.
• Insight into the design. System prompt, list of tools with their privileges, an architecture sketch. A test with this knowledge finds more in the same time than a pure black-box test.
• Limits and contacts. A cost budget for model calls during the test, stop criteria and a reachable contact person.
• Logging switched on. So that afterwards you can check whether your monitoring would have seen the attacks.

Frequently asked questions

Further reading

• AI red teaming explained
• The OWASP Top 10 for LLM Applications 2026, Explained in Practice
• The OWASP Top 10 for Agentic Applications, Explained in Practice
• Prompt Injection Testing: Test Plan, Test Cases and Scoring
• Service: AI pentest and MCP security

Sources

• OWASP GenAI Security Project: Top 10 for LLM Applications 2026
• OWASP GenAI Security Project: 2026 Top 10 for LLM Applications announcement
• OWASP GenAI Security Project: Top 10 for Agentic Applications for 2026
• OWASP MCP Top 10
• NIST AI 600-1: Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile
• NIST AI 100-1: Artificial Intelligence Risk Management Framework (AI RMF 1.0)