Prompt injection in practice: attack patterns we test for

How we use these prompt injection examples

Prompt injection means tricking an AI with hidden instructions: text the model follows as a command even though it came from a user, a document or another system. Our prompt injection guide explains how the attacks work and which defenses reduce them. This post covers the practical side: the patterns we try during AI red teaming, our structured attack testing of AI systems.

The list follows the attack types in the OWASP Top 10 for LLM Applications (opens in a new tab) and the NIST taxonomy of adversarial machine learning (opens in a new tab). We describe each pattern in general terms and do not publish working attack text. These are patterns we test for, not incidents from client projects. In a real test, we adapt each pattern to the agent's tools, data and users.

Direct attacks through the chat window

1. Instruction override in user input

• What it looks like: A user writes a message that tells the agent to disregard its rules, claims to be an administrator or asks it to repeat its hidden instructions.
• What could go wrong: The agent reveals its system prompt, shows data meant for other users or calls a tool outside its task.
• What we check: Whether the agent's permissions, not only its prompt, block the action, and whether secrets or other users' data can be reached from the chat at all.

2. Role-play and persona attacks

• What it looks like: The user asks the agent to play a character, a game or a test mode that supposedly has no rules. NIST lists role-play among known jailbreak techniques.
• What could go wrong: The agent steps out of its role, says things your business would not say or agrees to actions it would normally refuse.
• What we check: Whether the agent keeps its scope under many framings, and whether actions requested inside a role-play still pass the same permission and approval checks.

Attacks that hide their intent

3. Multi-turn gradual manipulation

• What it looks like: Each message is harmless on its own. Over many turns, the conversation moves step by step toward a request the agent would refuse if asked directly. NIST describes multi-turn attacks of this kind.
• What could go wrong: Filters that judge one message at a time see nothing wrong, and the agent reaches the harmful result anyway.
• What we check: Long conversations with slow drift, and whether policy checks run at each action instead of only on the first message.

4. Encoding and obfuscation

• What it looks like: The instruction is encoded, written in a less common language, hidden in invisible characters or split across several fields that the model puts back together.
• What could go wrong: Input filters that look for known phrases miss the attack, while the model still understands it.
• What we check: The same test cases in several encodings and languages, with invisible characters and split across form fields, to see which layer catches them.

Hidden instructions in content the agent reads

These are indirect attacks. The attacker does not interact with your AI. They place text where the AI will read it later, and the harm usually falls on your own user.

5. Hidden instructions in a web page

• What it looks like: An agent that browses or summarizes web pages reads text a person does not see on screen, such as text hidden by formatting or placed in page code.
• What could go wrong: The agent writes a false summary, recommends a fraudulent site or sends conversation data to an address the page names.
• What we check: Test pages with planted instructions, and whether the agent holds any sensitive tools or data while it browses.

6. Poisoned document in the knowledge base

• What it looks like: Someone adds or edits a document in a source the AI retrieves answers from, such as a shared drive, a help center or a wiki. The document carries instructions or false facts aimed at certain questions.
• What could go wrong: The agent gives wrong answers with confidence, points customers to the wrong contact or payment details or follows the planted instructions. NIST cites research in which a single crafted document was enough.
• What we check: Who can add content to each source, whether changes are logged and how the agent behaves when it retrieves a test document with planted instructions. Our post on RAG security covers knowledge base risks in depth.

7. Email-borne instructions to an inbox agent

• What it looks like: An agent that reads, sorts or answers email receives a message from an outside sender with instructions addressed to the AI.
• What could go wrong: The agent forwards private emails, replies with customer data or sends similar messages to your contacts. NIST describes this self-spreading pattern as a possible AI worm.
• What we check: Whether sending and forwarding need approval, whether recipients are limited to allowed domains and whether rate limits stop mass sending.

8. Tool-output injection

• What it looks like: The attack text arrives through a tool: a search result, a ticket field, a CRM note, a database record or the response from a Model Context Protocol (MCP) server.
• What could go wrong: The agent treats the output as a trusted instruction and takes its next action on the attacker's behalf, with its own permissions.
• What we check: Whether tool outputs are handled as untrusted data, whether they pass through strict schemas and whether the next tool call is checked against policy before it runs.

Attacks on what happens after the model answers

9. Data exfiltration through links or images

• What it looks like: Planted instructions tell the model to put private data, such as a customer's details, into a link or an image address in its reply.
• What could go wrong: When the chat interface loads the image or the user clicks the link, the data goes to a server the attacker controls. NIST notes that image rendering in chat can be used this way. The agent needs no email or web tool for this to work.
• What we check: Whether replies can contain links and images from any domain, whether images load automatically and whether an allow-list blocks unknown domains.

10. Attacks on the hand-off to a human

• What it looks like: The agent asks a person to approve an action or passes a case to staff with a summary. The attacker shapes the content so the summary looks harmless while the real action is not.
• What could go wrong: The reviewer approves a payment, an email or a data export based on a misleading summary. OWASP notes that the displayed action can differ from the executed one, and that reviewers lose focus when approvals are frequent.
• What we check: Whether the approval screen shows the exact action (recipient, amount and full text), whether hidden characters are removed before display and how many approvals each reviewer handles. Our post on human-in-the-loop approvals covers how to design this step.

What we do with the results

Each finding in our report includes the pattern, the input that triggered it, what the agent did, a risk rating and a fix your developers can apply. When a pattern succeeds, the fix is usually a limit on permissions, an approval step or an output check. More wording in the prompt rarely helps on its own, because prompt rules can be bypassed. We retest after the fixes and again when you change the model, prompts, tools or data sources.

No test proves an agent is immune. The UK National Cyber Security Centre (opens in a new tab) warns that prompt injection may never be fully mitigated, so the aim is to keep each successful attack small and visible. Our AI security consulting service runs these tests on agents and chatbots, whoever built them.