AI Security Testing Tools

Use these techniques only on systems you own or are explicitly authorized to test. Run high-impact checks in an isolated environment with synthetic data, test credentials, controlled callback services, and rollback procedures.

LLM and prompt security evaluation

• garak – automated LLM vulnerability scanner with probes for leakage, jailbreaks, encoding, and unsafe behavior.
• PyRIT – Microsoft framework for orchestrating repeatable AI red-team attacks and scoring responses.
• promptfoo – declarative adversarial evaluations and CI regression testing for prompts, models, and RAG systems.
• Giskard – model and LLM application testing for hallucination, prompt injection, data leakage, and bias.
• Inspect AI – evaluation framework for reproducible model capability and safety tests.

Application and API testing

• Burp Suite or Caido – intercept chat requests, alter hidden parameters, replay conversations, and compare authorization behavior.
• curl, HTTPie, or Postman – replay API calls with different users, model IDs, tool settings, and conversation identifiers.
• mitmproxy – inspect streaming APIs, WebSocket traffic, agent callbacks, and model-provider requests in a controlled test.
• Playwright – automate multi-user chat, approval, streaming, file-upload, and browser-rendering security tests.
• Canarytokens or a private callback service – detect controlled exfiltration and unexpected outbound requests.

MCP inspection and testing

• MCP Inspector – enumerate resources, prompts, tools, schemas, and server responses during authorized testing.
• mcp-scan – inspect MCP configurations and servers for tool poisoning and related risks.
• A JSON schema validator and an intercepting proxy – mutate tool arguments, reject unknown fields, and verify server-side authorization.

Code and supply-chain review

• Semgrep or CodeQL – trace untrusted prompts and model output into shell, SQL, HTML, file, URL, and deserialization sinks.
• Gitleaks or detect-secrets – find model-provider keys, MCP tokens, and credentials in repositories and logs.
• Syft plus Grype, Trivy, or another SCA scanner – inventory and scan agent frameworks, MCP servers, plugins, containers, and transitive dependencies.
• Cosign – verify signatures and provenance for containers and release artifacts.
• OSV-Scanner – scan lockfiles and source trees for known vulnerable dependencies.

Runtime controls and detection

• OpenTelemetry – correlate model requests, retrieval, tool calls, approvals, side effects, latency, and spend.
• OPA or Cedar – deterministic policy and authorization checks around model-selected actions.
• Falco, EDR, and cloud runtime monitoring – detect unexpected processes, file access, and network connections.
• Presidio or another DLP engine – find and redact personal data and secrets in model traffic and logs.

How to choose tools

• Start from a security property and attack path. A large scanner list does not replace manual authorization and business-logic testing.
• Use synthetic data and isolated targets. Many tools send prompts to models and can create cost, retention, or privacy concerns.
• Pin tool versions, review plugins and attack datasets, record configuration, and save raw evidence so results are reproducible.
• Combine black-box evaluations, source review, runtime traces, and downstream side-effect checks.

Practical test examples

Minimal repeatable tool record

tool: promptfoo
version: <pinned version>
target: staging chat API
model: <exact snapshot>
dataset_commit: <git SHA>
configuration_commit: <git SHA>
attempts_per_case: 5
synthetic_data_only: true
result_artifact: artifacts/ai-security/<run-id>/
owner: product-security

Useful tools

• Use the categories above as a starting point. Verify project health, licensing, data handling, and compatibility before adoption.

How to fix and retest

1. Keep an approved, version-pinned tool inventory with owners, review dates, and the security property each tool measures.
2. Run stable regression suites in CI and deeper red-team campaigns before high-risk releases.
3. Do not upload production prompts, customer data, credentials, or proprietary documents to an unapproved testing service.

Retest checklist

• Repeat the original proof with the same identity, state, model version, and input source.
• Try equivalent variants through every other input channel and fallback model.
• Confirm the control is enforced by the backend or runtime, not only by prompt wording.
• Check logs and alerts, then add the case to a versioned regression suite.

Conclusion

AI security issues are easiest to manage when every piece of content is treated according to its real trust level and every consequential action is controlled outside the model. Keep the test evidence reproducible, limit blast radius, and rerun the suite whenever the model, prompt, retrieval pipeline, tool, server, or policy changes.