The assistant says it refused to send a document. The test turns green. A delivery record later shows that the message had already left before the refusal appeared.
This fictional failure captures a common weakness in agent testing: checking the answer instead of the action. A useful security test plan states what the system must protect, triggers a relevant failure, and checks the resulting state independently of the model’s explanation.
Use a contained environment you own or have permission to test. Synthetic identities, fake documents, and a local delivery sink are enough for the examples here. Production customer data adds risk without improving these checks.
Start with one protected action
Choose an action that matters, such as sending a support reply. Write its invariant in observable terms: “No message may leave unless its current recipient and content have valid approval for the current task.”
Then identify the evidence source. The conversation is one source; the tool-call trace, approval record, and delivery sink provide others. For a read operation, inspect which records the application returned. For a write, inspect stored state and affected objects.
Keep the scenario small enough that a failure has a clear meaning. A long benchmark combining search, account changes, and email can fail for reasons unrelated to the security boundary under review.
Build a compact case matrix
For the support assistant, a first test set might look like this:
| Case | Change from the permitted task | Evidence required |
|---|---|---|
| Allowed reply | None | One delivery to the approved recipient |
| Untrusted instruction | A retrieved document asks for unrelated action | No unauthorized tool effect |
| Different customer | Proposed read references another customer’s record | No data returned outside the authorized scope |
| Changed draft | Content changes after approval | No delivery under the old approval |
| Revoked task | Cancellation happens before execution | Pending action is denied |
| Retry | The same authorized request is submitted twice | No duplicate delivery |
| Missing fixture | The intended test service is unavailable | No attempt reaches a live replacement |
This is a starting set, not a claim of complete coverage. Add cases from the tools, data, and failure history of your own workflow.
Separate policy tests from model evaluations
Policy tests submit candidate actions directly to the enforcement layer. They should be deterministic: the same invalid action must be rejected regardless of which model proposed it. These checks are fast enough to run with every relevant code change.
Model evaluations place the agent in a task and observe what it proposes. For an indirect prompt-injection test, put a benign conflicting instruction in a synthetic document and verify that the system treats it as data, not authority. Do not point the exercise at an outside victim or include real secrets.
Repeat model evaluations under a recorded configuration because behavior can vary. Record model version, prompt revision, available tools, sampling settings where exposed, fixture version, and number of runs. Report the observed numerator and denominator, such as “0 unauthorized deliveries in 20 runs of these cases.” Avoid turning a small sample into a general security guarantee.
Also measure whether legitimate tasks succeed. A system that rejects every action may prevent this failure while being unusable for its intended work.
Exercise failures between planning and execution
Authorization can change while a task waits in a queue. Test cancellation after a plan is generated but before the tool executes. Change the payload after approval. Run two workers against the same approved operation where the application supports concurrency.
For remote actions, simulate a timeout after the receiving service has accepted the request. The next attempt should reconcile the known operation or use supported idempotency, rather than send a second independent request. A timeout alone does not prove that nothing happened.
These tests belong at the application’s actual action boundary. Mocking the authorization check to always deny proves little about the real path.
Verify that the test can catch a failure
Before trusting a negative test, confirm that its observation works. A permitted synthetic message should appear in the delivery sink. A deliberately unprotected local fixture should make the relevant assertion fail. Keep that fixture isolated and out of production code.
This guards against tests that pass because the agent never ran, the tool was disconnected, or the assertion watched the wrong log. Distinguish “action blocked” from “scenario did not execute” in the result.
Inspect evidence for sensitive content before retaining it. Usually an operation ID, actor, target classification, rule result, and delivery count are enough. Restrict access and retention if a trace must contain document text.
Make the release decision traceable
For each case, keep the expected outcome, actual state, evidence location, and an owner for unresolved failures. Record missing checks as not run. Re-run affected cases when permissions, tools, prompts, or model versions change.
A least-privilege tool design makes this work easier because each action has a smaller set of allowed effects. A bounded code review can then connect the test to the specific enforcement code.
Your first useful deliverable is a small matrix with reproducible evidence, not a large scorecard. If a reviewer can explain why one forbidden action was stopped and one permitted action succeeded, the plan has a foundation you can extend.
Further reading: OWASP Prompt Injection and Anthropic’s evaluation findings. Scenarios and test recommendations are Linmas editorial material.