LLM Gateway Design: Model Routing, Fallbacks, Rate Limits and Budgets

Once several teams call several model providers, every service ends up reimplementing retries, keys, rate limits, logging and cost tracking. An LLM gateway (or AI gateway) is a single internal entry point for all model calls. It owns provider credentials, routing between models, fallbacks when a provider fails, token-based rate limits and budgets per team or tenant, caching, and observability. It's the API gateway pattern applied to LLM traffic, and a common interview question in its own right.

Architecture

What the gateway owns

Routing strategies

• Static rules: the caller's alias maps to a model; per-tenant overrides for compliance (EU tenants to an EU-region endpoint).
• Cost-based cascade: try a small model first; escalate to a larger one if a confidence check or validator fails.
• Classifier router: a fast classifier (or embedding similarity to labelled examples) predicts whether a request needs the strong model.
• Latency-aware: route away from a provider whose p95 is spiking.
• Canary routing: send a percentage of traffic to a new model or prompt version for evaluation; see LLMOps.

Fallbacks and retries

route "support-chat":
  primary : provider_a / medium-model       timeout 20 s (streaming: 3 s to first token)
  retry   : same model, 2 attempts, exponential backoff + jitter
            only on 429, 5xx, timeouts (never on 400 validation errors)
  fallback: provider_b / comparable-model   prompt variant tuned for it
  last    : small self-hosted model with a "limited mode" notice
  breaker : open provider_a for 30 s after 50% errors over 20 s window

• Fallback models behave differently; keep prompt variants per model and evaluate the fallback path like a release.
• Streaming complicates retries: once tokens have reached the user you can't silently switch models mid-answer; fall back only before the first token, or restart visibly.
• Alert when both primary and fallback fail; that's usually a systemic problem (bad request shape, expired key), not a provider outage. See retries and timeouts and circuit breakers.

Token-based rate limiting and budgets

Request counts are a poor limit for LLMs: one request can be 100 tokens or 100,000. Limit tokens per minute and spend per period, at several levels (organisation, team, tenant, end user).

• Estimate input tokens before the call and reserve them from a token bucket; reconcile with actual usage (including output) afterwards.
• Keep counters in a fast shared store (for example Redis) with atomic updates; tolerate small overshoot for latency.
• Return clear errors with retry-after hints; let critical traffic draw from a reserved pool.
• Respect provider-side limits too: the gateway smooths bursts across your fleet so one team can't exhaust the shared provider quota.

Scale and latency

Traffic     : 3,000 LLM requests/s at peak across 40 services
Gateway hop : auth + quota + cache lookup ≈ 5-15 ms, small next to 1-10 s model calls
Connections : long-lived streaming responses → many concurrent open connections
              (3,000 req/s × ~8 s average = ~24K concurrent streams); use async I/O
Metering    : write usage events to a queue, aggregate asynchronously for billing
              and dashboards, so metering never blocks the request path

The gateway is now a critical dependency: run it stateless and horizontally scaled across zones, keep its own dependencies (quota store, cache) fast and fail-open where safe, and give it a bypass plan for emergencies.