Sign inStart practicing
Start practicing Sign in
System design interview question · Backend

Design a distributed web crawler

Discover and fetch public pages continuously while respecting site policies.

Advanced 60 minutesSearch Plus
More Backend questions

Concepts this round covers

  • Frontier partitioning
  • Politeness and robots rules
  • Deduplication
  • Search

This is a Plus round

The brief, the scale to design for, the questions interviewers ask and the common mistakes for this round are available to Plus members. Plus is coming soon.

Try “URL shortener” free Already on Plus? Sign in

More Backend system design interview questions

See all
Free
Backend·Storage

Design a URL shortener

Build a service that turns long links into short, shareable URLs.

Beginner 35 min
Plus
Backend·Messaging

Design a chat system

Create a real-time messaging experience for millions of people.

Intermediate 45 min
Plus
Backend·Data systems

Design a news feed

Deliver a relevant, fresh feed as people and publishers post.

Advanced 60 min
Plus
Backend·Storage

Design cloud file storage

Sync large files reliably across devices and collaborators.

Advanced 60 min
Plus
Backend·Reliability

Design an API rate limiter

Protect a public API without slowing down well-behaved clients.

Intermediate 40 min
Plus
Backend·Media

Design video streaming

Stream adaptive-quality video to a global audience.

Advanced 60 min