Design an AI Code Assistant Like GitHub Copilot: System Design

The question: design an AI code assistant like GitHub Copilot: as a developer types in their editor, it suggests the next lines of code; it also offers a chat about the codebase and, increasingly, an agent mode that edits files and runs tests. The hardest constraint is latency: inline suggestions are useless if they arrive after the developer has typed on. The other key challenges are picking the right context from the repository, privacy of customers' code, and measuring usefulness.

1. Clarify requirements

2. Estimates

Completion requests: 2M devs × ~5 active hours × ~120 requests/hour ≈ 1.2B/day
                    ≈ 14K/s avg, ~40K/s peak (work hours across time zones)
Per request       : ~1,500-4,000 prompt tokens of context, ~20-60 output tokens
Implication       : prompt-heavy, output-light → prefill dominates; a small, fast
                    model is mandatory for inline; caching and cancellation are crucial
Cancellation      : many requests are obsolete before they finish (the user kept
                    typing) → cancel in-flight requests to save GPU time

3. Inline completion architecture

4. Context: the core quality lever

The model only knows what's in its prompt. Within a few thousand tokens, include what most helps predict the next lines:

• Prefix and suffix: code before and after the cursor. Models trained for fill-in-the-middle (FIM) use both, so a completion fits between existing lines.
• Open tabs and recently edited files: strong signals of what the developer is working on.
• Relevant snippets from the repository: definitions of symbols used near the cursor (from a language server or symbol index), and similar code found by embedding search.
• File path, language and imports.
• Context is ranked and trimmed to a token budget; the plugin does much of the gathering locally to avoid shipping whole repositories.

For chat and agent mode, a richer repository index (chunked by function and class, embedded, plus a symbol graph) supports retrieval over the whole codebase; see chunking strategies.

5. Latency engineering

• Debounce keystrokes (send after a short pause) and cancel stale requests server-side.
• A small model specialised for code, possibly distilled from a larger one; see model compression.
• Stop generation early: at a line or block boundary, or when confidence drops.
• Serve from regions near developers; keep the prompt's stable parts first for prefix caching; see LLM caching.
• Client-side cache: if the user types exactly what was suggested, keep showing the rest without a new request.

6. Post-processing and safety

• Trim to syntactically sensible boundaries; drop suggestions that duplicate the following lines.
• Filter secrets and obviously unsafe patterns; optionally flag suggestions that closely match public code (licence concerns).
• Don't suggest in files marked sensitive or excluded by policy.

7. Chat and agent mode

• Chat uses a stronger model with retrieved repository context and the current selection; answers stream.
• Agent mode runs a tool loop: search code, read files, edit, run tests and linters in a sandbox, iterate. Tests act as the success check.
• Changes are presented as diffs for review; the agent never pushes or merges on its own.
• Repository content (including comments and READMEs) can carry injected instructions: sandbox without secrets or network by default. See prompt injection.

8. Privacy

• Enterprise mode: prompts processed in memory, not retained, not used for training; contractually and technically enforced.
• Region pinning for data residency; per-organisation policy for which repositories and files may be sent.
• Telemetry (acceptance events) stored without code content.

9. Measuring usefulness

Offline, evaluate on held-out repositories with tests: does the completion make the tests pass? Online, A/B test model and context changes on acceptance and retention. See LLM evaluation.