LLM02: Sensitive Information Disclosure

Here is an example of Python code that is vulnerable to sensitive information disclosure in an LLM application:

🥺 Vulnerable Code

def build_prompt(user_question):
    # Vulnerable: the whole customer row, secrets included, goes into the prompt
    customer = db.query("SELECT * FROM customers WHERE id = %s", session["customer_id"])
    return f"Customer record: {json.dumps(customer)}\n\nQuestion: {user_question}"


prompt = build_prompt(question)
answer = llm.complete(prompt)

log.info("prompt=%s answer=%s", prompt, answer)   # full record written to logs

SELECT * drags card tokens, national ID numbers, internal risk notes, and sometimes an API key into the context window. The model will happily repeat any of it when asked the right way, and the logging line copies the whole record into your observability platform, where a much wider group of people can read it. If the provider retains prompts for training, the data leaves your control entirely.

😎 Secure Code

Here is a version of the same code that is secured against sensitive information disclosure in an LLM application:

ALLOWED_FIELDS = ("first_name", "plan", "renewal_date", "open_tickets")


def build_context(user_question):
    customer = db.query(
        "SELECT first_name, plan, renewal_date, open_tickets FROM customers WHERE id = %s",
        session["customer_id"],
    )
    return {
        "profile": {field: customer[field] for field in ALLOWED_FIELDS},
        "question": user_question,
    }


response = llm.chat(messages=to_messages(build_context(question)),
                    metadata={"training_opt_out": True})

answer = response.output_text
if pii_detector.scan(answer).has_findings:
    answer = redact(answer)

log.info(
    "llm_call customer=%s tokens=%s answer_sha256=%s",
    hash_id(session["customer_id"]), response.usage.total_tokens, sha256(answer),
)

Only the four fields the assistant actually needs cross the boundary, so there is nothing sensitive for the model to leak in the first place. Secrets never belong in a prompt - inject them server-side inside a tool. The logs record identifiers and metrics instead of content, the output is scanned for personal data before it reaches the user, and the provider is told not to retain the request for training.

Security Resources

🥺 Vulnerable Code

😎 Secure Code