AI Agent Security Testing

Use these techniques only on systems you own or are explicitly authorized to test. Run high-impact checks in an isolated environment with synthetic data, test credentials, controlled callback services, and rollback procedures.

Map agent authority

• Enumerate every tool, credential, filesystem path, network destination, data source, memory store, and delegated agent available in each role.
• Compare advertised tool purpose with actual backend capability. A read-named tool that accepts arbitrary URLs or shell fragments may be a write or execution primitive.
• Identify actions that are destructive, external, expensive, privacy-sensitive, or difficult to reverse.

Test injection-to-action paths

• Place a canary instruction in user input, a retrieved document, a website, tool output, and memory. Observe whether it can trigger a privileged tool.
• Mutate tool arguments after the model selects a tool: object IDs, tenant IDs, file paths, URLs, commands, recipients, quantities, and cloud regions.
• Test whether the backend re-authorizes the exact action or trusts the model and client.
• Verify approval dialogs show the final resolved target and cannot be bypassed by retries, alternate tools, batching, or background execution.

Runtime and autonomy tests

• Trigger repeated planning, delegation, retry, and tool-call loops. Confirm step, time, token, spend, and fan-out budgets stop the run.
• Test the kill switch while tools are in flight and verify queued or child-agent actions are cancelled.
• For code agents, attempt read-only sandbox checks for user ID, mounted secrets, filesystem scope, network egress, and access to other sessions.
• Review traces for enough detail to reconstruct why every consequential action happened.

Practical test examples

Tool authorization test matrix

For every mutating tool test:
- valid own object
- another user's object
- another tenant's object
- missing object ID
- traversal or alternate identifier
- replayed request
- concurrent duplicate request
- action after approval expires
- action after kill switch

Useful tools

• garak – automated LLM vulnerability scanner with probes for leakage, jailbreaks, encoding, and unsafe behavior.
• PyRIT – Microsoft framework for orchestrating repeatable AI red-team attacks and scoring responses.
• promptfoo – declarative adversarial evaluations and CI regression testing for prompts, models, and RAG systems.
• Giskard – model and LLM application testing for hallucination, prompt injection, data leakage, and bias.
• Inspect AI – evaluation framework for reproducible model capability and safety tests.
• Burp Suite or Caido – intercept chat requests, alter hidden parameters, replay conversations, and compare authorization behavior.
• curl, HTTPie, or Postman – replay API calls with different users, model IDs, tool settings, and conversation identifiers.
• mitmproxy – inspect streaming APIs, WebSocket traffic, agent callbacks, and model-provider requests in a controlled test.
• Semgrep or CodeQL – trace untrusted prompts and model output into shell, SQL, HTML, file, URL, and deserialization sinks.
• Gitleaks or detect-secrets – find model-provider keys, MCP tokens, and credentials in repositories and logs.
• Syft plus Grype, Trivy, or another SCA scanner – inventory and scan agent frameworks, MCP servers, plugins, containers, and transitive dependencies.
• OpenTelemetry traces – correlate prompts, plans, tool arguments, results, approvals, and side effects.
• OPA or Cedar – express deterministic authorization and action policies outside the model.

How to fix and retest

1. Broker every tool call through server-side schema validation, authorization, quotas, and policy checks.
2. Use separate least-privilege credentials per agent role and task. Do not expose a broad human or production token to the model runtime.
3. Require injection-resistant human approval for high-impact actions, plus idempotency, replay protection, timeouts, and rollback where possible.

Retest checklist

• Repeat the original proof with the same identity, state, model version, and input source.
• Try equivalent variants through every other input channel and fallback model.
• Confirm the control is enforced by the backend or runtime, not only by prompt wording.
• Check logs and alerts, then add the case to a versioned regression suite.

Conclusion

AI security issues are easiest to manage when every piece of content is treated according to its real trust level and every consequential action is controlled outside the model. Keep the test evidence reproducible, limit blast radius, and rerun the suite whenever the model, prompt, retrieval pipeline, tool, server, or policy changes.