LLM Threat Modeling and Attack Surface Mapping

Use these techniques only on systems you own or are explicitly authorized to test. Run high-impact checks in an isolated environment with synthetic data, test credentials, controlled callback services, and rollback procedures.

What to inventory

• Models and providers: hosted APIs, local models, fine-tunes, fallback models, moderation models, and embedding models.
• Context sources: system and developer prompts, chat history, RAG documents, web results, files, image OCR, audio transcripts, memory, and tool output.
• Action surfaces: function calls, MCP tools, code execution, browser automation, database queries, cloud APIs, and outbound messages.
• Output sinks: HTML and Markdown renderers, SQL, shell commands, templates, URLs, logs, exports, and downstream agents.

Trust-boundary questions

• Can one tenant influence another tenant’s retrieval, cache, memory, conversation, or tool credentials?
• Can a document author, web page, email sender, or tool server insert instructions that the model treats as trusted?
• Does the backend independently authorize each tool call and object ID, or trust the model to choose safely?
• Can the model generate a URL that the backend fetches, a query the database runs, or markup the browser renders?
• What changes when the context window fills and older policy instructions are truncated or deprioritized?

Abuse-case workshop

• For each component, ask what an attacker can read, write, trigger, persist, impersonate, or make more expensive.
• Model attacker roles separately: anonymous user, authenticated user, malicious tenant, poisoned content author, compromised MCP server, insider, and supply-chain attacker.
• Add a concrete security test and owner to every high-impact abuse case.

Practical test examples

Minimal data-flow template

Untrusted source -> parser -> retrieval/context -> model
model -> policy decision? -> tool broker -> privileged API
model output -> renderer/sink
memory write -> future session

For each arrow record:
- identity and tenant
- validation and authorization
- data sensitivity
- logs and alerting
- failure mode

Useful tools

• OWASP Threat Dragon, draw.io, or Mermaid – maintain a versioned data-flow diagram.
• STRIDE and attack trees – useful prompts for finding spoofing, tampering, disclosure, denial, and privilege escalation.
• Repository search and runtime tracing – verify the diagram matches actual model, retrieval, and tool calls.

How to fix and retest

1. Review the threat model whenever a model, data source, tool, MCP server, memory feature, or autonomous action is added.
2. Give every trust boundary a deterministic control and a testable security property.
3. Track assumptions explicitly, such as whether provider data is retained or whether retrieved documents are tenant-scoped.

Retest checklist

• Repeat the original proof with the same identity, state, model version, and input source.
• Try equivalent variants through every other input channel and fallback model.
• Confirm the control is enforced by the backend or runtime, not only by prompt wording.
• Check logs and alerts, then add the case to a versioned regression suite.

Conclusion

AI security issues are easiest to manage when every piece of content is treated according to its real trust level and every consequential action is controlled outside the model. Keep the test evidence reproducible, limit blast radius, and rerun the suite whenever the model, prompt, retrieval pipeline, tool, server, or policy changes.