LLM10: Unbounded Consumption

Here is an example of Python code that is vulnerable to unbounded consumption:

🥺 Vulnerable Code

@app.post("/api/chat")
def chat():
    # Vulnerable: no authentication, no input cap, no output cap, no budget
    messages = request.json["messages"]

    response = llm.chat(model="expensive-model", messages=messages)
    return {"answer": response.output_text}

An unauthenticated endpoint that bills per token is a denial of wallet waiting to happen. A script can push a megabyte of context on every request, ask for the longest possible completion, and run it in parallel until the invoice or the rate limit at the provider stops it. Long contexts also tie up workers, and unlimited querying is exactly how model extraction and distillation attacks are carried out.

😎 Secure Code

Here is a version of the same code that is secured against unbounded consumption:

MAX_PROMPT_CHARS = 12_000
MAX_OUTPUT_TOKENS = 700


@app.post("/api/chat")
@require_auth
@rate_limit(key=lambda: current_user.id, limit=30, window_s=60)
def chat():
    messages = request.json["messages"]

    if total_chars(messages) > MAX_PROMPT_CHARS or len(messages) > 40:
        return {"error": "Conversation too long"}, 413

    if billing.tokens_used_today(current_user.id) > current_user.daily_token_quota:
        return {"error": "Daily quota reached"}, 429

    response = llm.chat(
        model=pick_model(current_user.plan),     # cheaper model for the free tier
        messages=messages,
        max_output_tokens=MAX_OUTPUT_TOKENS,
        timeout_s=20,
    )

    billing.record(current_user.id, response.usage.total_tokens)
    return {"answer": response.output_text}

Authentication comes first, because you cannot enforce a quota against an anonymous caller. Input size, conversation length, output tokens, and request timeout are all bounded, and a per-user daily token quota puts a hard ceiling on cost. Record usage as you go so spend is visible in real time, set budget alerts at the provider as a backstop, and watch for accounts whose query pattern looks like someone systematically harvesting the model.

Security Resources

🥺 Vulnerable Code

😎 Secure Code