Sign inStart practicing
Start practicing Sign in
System design interview question · AI engineering

Design an LLM inference platform

Serve open-weight LLMs on a GPU fleet with high throughput and predictable latency.

Advanced 60 minutesLLM infrastructure Plus
More AI engineering questions

Concepts this round covers

  • Continuous batching and the KV cache
  • Autoscaling GPUs
  • Many models and adapters
  • LLM infrastructure

This is a Plus round

The brief, the scale to design for, the questions interviewers ask and the common mistakes for this round are available to Plus members. Plus is coming soon.

Try “ChatGPT” free Already on Plus? Sign in

More AI engineering system design interview questions

See all
Free
AI engineering·Assistants

Design ChatGPT

Serve a chat assistant to millions of people with streamed answers, saved history and fair usage limits.

Advanced 60 min
Free
AI engineering·AI agents

Design a customer support AI agent

Resolve support chats with tools and refunds within policy, and hand off to a human when needed.

Advanced 60 min
Plus
AI engineering·Assistants

Design an AI code assistant

Suggest code as developers type, using context from their repository, in a few hundred milliseconds.

Advanced 60 min
Plus
AI engineering·AI agents

Design a real-time voice AI agent

Answer phone calls with an AI that listens, books appointments and replies without awkward pauses.

Advanced 60 min