Prompt injection testing: building an adversarial suite for CI

Decide what a failure actually is

Before writing payloads, define observable failure conditions. "The model said something bad" is not testable. "The model called the refund tool without a confirmation step" is.

• ▸Unauthorised tool invocation, or a tool called with out-of-scope arguments.
• ▸Disclosure of system prompt, retrieved documents outside the user’s permissions, or secrets.
• ▸Output that fails schema validation, or contains executable markup that reaches a renderer.
• ▸Silent ignoring of a required confirmation or refusal path.

Build the payload corpus

Keep payloads as data files, versioned next to the prompts they attack, so a reviewer can see both change in the same diff. Group them by delivery channel, because indirect injection is the class that surprises teams.

• ▸Direct: instruction override, role confusion, encoded and multilingual variants.
• ▸Indirect: payloads embedded in retrieved documents, web pages, PDFs, filenames and tool responses.
• ▸Exfiltration: prompts that ask the model to place secrets into a URL, image tag or tool argument.
• ▸Chained: a payload that instructs the model to write a poisoned note into persistent memory for a later session.

Assert on behaviour, not on wording

Asserting exact response text produces a flaky suite that everyone disables. Assert on structure and side effects instead: which tools were called, with what arguments, whether output parsed against the schema, and whether the confirmation gate fired.

Use a stub tool layer in tests so a successful injection is recorded rather than executed. A classifier can grade free-text refusals, but the deterministic assertions are what you gate the pipeline on.

Wire it into CI

Prompts, tool definitions and retrieval configuration are code. Any change to them should trigger the suite.

• ▸Run the deterministic subset on every pull request that touches prompts, tools or retrieval.
• ▸Pin the model version in CI so a provider-side change is a visible, separate diff.
• ▸Fail the build on any unauthorised tool call or schema violation; report soft refusal regressions as warnings.
• ▸Run the full corpus nightly, and re-run before any model or provider upgrade.

Testing is not the mitigation

A suite tells you whether your authorisation design holds. It does not create one. The durable controls remain least-privilege tools scoped per request and per user, human confirmation for irreversible actions, retrieval filtered with the requesting user’s permissions, and output escaped at the render layer. Testing keeps those honest as the product changes.

Frequently asked