Rate Limiting & Cost
Rate limiting and cost management for LLM APIs: token budgets, per-user quotas, and cost optimization strategies.
Last updated
Rate Limiting & Cost
Requests, tokens, and dollars are three separate currencies — any one of them can veto a call. Four algorithms, a four-layer enforcement stack, and the five levers that shrink what you need to limit.
A classic API gateway counts requests per second and calls it done. An LLM gateway cannot: one request can carry 50 tokens or 500,000, cost $0.0003 or $3, and the provider meters you on all three axes at once, per model. So rate limiting for LLM systems is really budget enforcement in three currencies, with a fallback plan for every “no.” This section builds the real thing: the four classic algorithms with running code, a four-layer stack that enforces per-tenant quotas, and the cost math that decides how much limiting you even need.