Design a Text-to-Image Generation Service: GPUs, Queues and Safety

The question: design a text-to-image service like Midjourney: users type a prompt and receive several generated images in seconds to a minute; they can upscale, vary or edit them, and browse a gallery. Unlike text LLMs, each request is a heavy, fixed-cost GPU job rather than a stream of tokens, so the design centres on job queues and scheduling, GPU efficiency, safety on both prompts and images, and storage and delivery of the results.

1. Clarify requirements

2. Estimates

Jobs        : 20M/day ≈ 230/s avg, ~600/s peak
GPU time    : say ~4 GPU-seconds per 4-image grid on a modern GPU with an
              optimised diffusion model (illustrative)
Load        : 600 jobs/s × 4 s = 2,400 GPUs busy at peak; plan ~3,000 with headroom
              → GPU cost is the business; batching and step reduction matter
Storage     : 80M images/day × ~1.5 MB (PNG) ≈ 120 TB/day before thumbnails/WebP
Delivery    : images served via CDN; gallery thumbnails are most of the reads

3. How generation works (briefly)

Diffusion models start from random noise and denoise it over a number of steps, guided by an embedding of the text prompt. Latent diffusion runs those steps in a compressed latent space and decodes to pixels at the end, which makes it far cheaper than working on full-resolution pixels. Cost scales with resolution and number of steps; distilled models that need fewer steps cut GPU time substantially.

4. Architecture

5. Queues and scheduling

• Generation is asynchronous: the API returns a job id immediately; progress and results arrive over a WebSocket or polling. See message queues.
• Separate queues per tier; the scheduler gives fast-tier jobs priority and lets relaxed jobs use spare capacity, a form of load shedding.
• Batching compatible jobs (same model, resolution and step count) on one GPU improves throughput.
• Quotas in GPU-seconds per user per period, deducted when a job starts and refunded on failure.
• Idempotent job ids so client retries don't create duplicate jobs; see idempotency.

6. GPU fleet

• Workers keep models loaded in GPU memory; loading a multi-gigabyte model takes far longer than a generation, so route jobs to workers that already hold the right model (and LoRA styles) warm.
• Autoscale on queue depth and wait time; GPU capacity takes minutes to add, so forecast daily peaks and keep a buffer.
• Use cheaper interruptible GPUs for the relaxed queue, with retries on preemption.
• Speed-ups: fewer sampling steps with distilled models, lower-precision inference, compiled kernels; see model compression.

7. Safety

These are the same per-harm classifier and review-queue ideas as in harmful content detection and LLM guardrails.

8. Storage and gallery

• Store originals in object storage and generate WebP thumbnails; serve through a CDN.
• Metadata (prompt, parameters, seed, model version, user, visibility) in a database; the seed and parameters make variations reproducible.
• Lifecycle policies move old, rarely viewed originals to cheaper storage tiers.
• The public feed is a ranking problem (recency, engagement, quality) built on the metadata; see recommendation systems.

9. Metrics

• Queue wait and generation time per tier (p50, p95), GPU utilisation, cost per image.
• Safety block rates and false positives (appeals), reports per thousand images.
• Quality: upscale and download rates as implicit signals; periodic human preference studies when changing models.