LLM09: Misinformation and Overreliance

Here is an example of Python code that is vulnerable to misinformation and overreliance on model output:

🥺 Vulnerable Code

# Vulnerable: model output shown as fact, with no source and no human in the loop
advice = llm.complete(f"What dose of {drug} should this patient take?")
render_template("advice.html", advice=advice)

# Worse: hallucinated dependency names installed automatically
packages = llm.complete("List the npm packages needed for this feature").split()
subprocess.run(["npm", "install", *packages], check=True)

A language model produces fluent text, not verified facts, and it is just as confident when it is wrong. Presenting that as clinical, legal, or financial advice with no citation and no review is a liability problem before it is a security problem. The second block is a direct supply chain risk: models invent package names, attackers register the popular hallucinations, and your build installs their code. That is slopsquatting.

😎 Secure Code

Here is a version of the same code that is secured against misinformation and overreliance on model output:

result = rag.answer(question, require_citations=True)

if not result.citations or result.confidence < 0.7:
    return render_template("advice.html", advice=NO_ANSWER_TEXT, review_required=True)

for claim in result.claims:
    if not verifier.supported_by(claim, result.citations):
        return escalate_to_human(question, result)

# Dependency suggestions are proposals, never actions
for name in result.suggested_packages:
    if not registry.exists(name) or registry.age_days(name) < 90:
        flag_for_review(name)
        continue
    open_pull_request_to_add(name)          # a person reviews and merges

Grounding answers in retrieved sources and requiring citations gives the user something to check, and refusing to answer when confidence is low or sources are missing is a feature, not a regression. Claims are verified against the cited passages before display, high-risk domains route to a human, and nothing the model suggests gets installed or executed automatically. Log corrections so the failure modes feed back into your evaluation set.