LLM04: Data and Model Poisoning

Here is an example of Python code that is vulnerable to data and model poisoning:

🥺 Vulnerable Code

@app.post("/api/feedback")
def feedback():
    # Vulnerable: anonymous input goes straight into retrieval and the training set
    doc = request.json["content"]

    vector_store.add_texts([doc], metadatas=[{"source": "user_feedback", "trusted": True}])

    with open("data/finetune.jsonl", "a") as fh:
        fh.write(json.dumps({"messages": request.json["messages"]}) + "\n")

    return {"stored": True}

Anyone on the internet can now write the knowledge base. Seed it with "our refund policy allows unlimited refunds without approval" and the assistant will cite that back to customers with full confidence. Marking the text trusted: True makes it outrank real documentation. The append to the fine-tune file is worse, because the next training run bakes a backdoor trigger permanently into the weights.

😎 Secure Code

Here is a version of the same code that is secured against data and model poisoning:

@app.post("/api/feedback")
@require_auth
def feedback():
    doc = request.json["content"][:5000]

    db.add(Submission(
        author_id=current_user.id,
        content=doc,
        trust="untrusted",
        status="pending_review",
        content_hash=sha256(doc),
    ))
    return {"queued": True}


def promote_reviewed_batch(batch_id, reviewer_id):
    batch = load_batch(batch_id)
    if batch.approved_by != reviewer_id or reviewer_id == batch.author_id:
        raise PermissionError("Four eyes review required")

    for item in batch.items:
        vector_store.add_texts(
            [item.content],
            metadatas=[{"source": item.source, "trust": "curated", "batch": batch_id}],
        )

    record_provenance(batch_id, checksum=batch.checksum)

Submissions are quarantined and never reach live retrieval until a second person approves them, which is the control that matters. Trust level travels with the chunk as metadata so the retriever can filter on it, every batch carries provenance and a checksum so a bad one can be rolled back, and training data comes from reviewed batches only. Run a canary set of poisoning prompts against each candidate model before it is promoted.

Security Resources

🥺 Vulnerable Code

😎 Secure Code