Insecure LLM Output Handling

Use these techniques only on systems you own or are explicitly authorized to test. Run high-impact checks in an isolated environment with synthetic data, test credentials, controlled callback services, and rollback procedures.

Find dangerous sinks

• Search for model output flowing into innerHTML, Markdown renderers with raw HTML, template engines, eval, exec, subprocess, shell commands, SQL, file paths, HTTP clients, and deserializers.
• Intercept a response and replace model text with a harmless sink-specific canary. This separates output-handling flaws from the difficulty of persuading a model.
• Test streaming paths and partial rendering. A sanitizer applied after the final chunk does not protect content already inserted into the page.
• Check CSV, PDF, office document, email, and ticket exports for formula injection, active links, or unsafe markup.

Practical payload classes

• HTML and Markdown: event handlers, javascript URLs, SVG, iframe, and remote image callbacks.
• Command execution: separators, command substitution, option injection, and path traversal.
• Database: model-generated filters, SQL fragments, NoSQL operators, and schema-changing statements.
• Backend URL fetching: loopback, private ranges, cloud metadata, redirects, DNS rebinding, and alternate IP encodings.

Confirm the application flaw

• Prove the sink is reached using a benign canary such as a local test file, calculator command in a sandbox, or controlled callback.
• Document the parser and encoding context. The correct fix for HTML differs from shell, SQL, URL, or filename handling.

Practical test examples

Repository triage

rg -n 'innerHTML|dangerouslySetInnerHTML|eval\(|exec\(|subprocess|os\.system' .
rg -n 'SELECT|INSERT|UPDATE|requests\.get|fetch\(|open\(' src/

# Review only paths where untrusted model output can reach the sink.

Harmless renderer canary

<img src="https://canary.invalid/llm-output-test" alt="output test">

Expected: displayed as text or removed by an allowlist sanitizer.
Unexpected: browser or backend requests the URL.

Useful tools

• Burp Suite or Caido – intercept chat requests, alter hidden parameters, replay conversations, and compare authorization behavior.
• curl, HTTPie, or Postman – replay API calls with different users, model IDs, tool settings, and conversation identifiers.
• mitmproxy – inspect streaming APIs, WebSocket traffic, agent callbacks, and model-provider requests in a controlled test.
• Semgrep or CodeQL – trace untrusted prompts and model output into shell, SQL, HTML, file, URL, and deserialization sinks.
• Gitleaks or detect-secrets – find model-provider keys, MCP tokens, and credentials in repositories and logs.
• Syft plus Grype, Trivy, or another SCA scanner – inventory and scan agent frameworks, MCP servers, plugins, containers, and transitive dependencies.

How to fix and retest

1. Treat all model output as untrusted and apply context-specific encoding or strict allowlist validation at the final sink.
2. Use typed, allowlisted actions instead of executing free-form model-generated commands, SQL, paths, or URLs.
3. Sandbox unavoidable execution, remove network and host access, use short timeouts, and run under a disposable low-privilege identity.

Retest checklist

• Repeat the original proof with the same identity, state, model version, and input source.
• Try equivalent variants through every other input channel and fallback model.
• Confirm the control is enforced by the backend or runtime, not only by prompt wording.
• Check logs and alerts, then add the case to a versioned regression suite.

Conclusion

AI security issues are easiest to manage when every piece of content is treated according to its real trust level and every consequential action is controlled outside the model. Keep the test evidence reproducible, limit blast radius, and rerun the suite whenever the model, prompt, retrieval pipeline, tool, server, or policy changes.