Sign inStart practicing
Start practicing Sign in
System design interview question · AI engineering

Design an LLM gateway

Give every team one safe way to call many model providers, with routing, fallbacks and budgets.

Intermediate 45 minutesLLM infrastructure Plus
More AI engineering questions

Concepts this round covers

  • Provider fallback
  • Token rate limits and budgets
  • Routing and caching
  • LLM infrastructure

This is a Plus round

The brief, the scale to design for, the questions interviewers ask and the common mistakes for this round are available to Plus members. Plus is coming soon.

Try “ChatGPT” free Already on Plus? Sign in

More AI engineering system design interview questions

See all
Free
AI engineering·Assistants

Design ChatGPT

Serve a chat assistant to millions of people with streamed answers, saved history and fair usage limits.

Advanced 60 min
Free
AI engineering·AI agents

Design a customer support AI agent

Resolve support chats with tools and refunds within policy, and hand off to a human when needed.

Advanced 60 min
Plus
AI engineering·Assistants

Design an AI code assistant

Suggest code as developers type, using context from their repository, in a few hundred milliseconds.

Advanced 60 min
Plus
AI engineering·AI agents

Design a real-time voice AI agent

Answer phone calls with an AI that listens, books appointments and replies without awkward pauses.

Advanced 60 min