AI Application Architecture: Retrieval, Tools and Guardrails

An AI application is a product system wrapped around a model, not just a prompt sent to an API. A reliable design authenticates the user, enforces permissions, assembles bounded context, chooses a model or tool, validates the result, handles timeouts and cost, and measures whether the response actually helped.

A request path around the model

The orchestrator is the application boundary: it decides what context and tools are permitted, limits model work, and turns provider-specific responses into a stable product contract. Keep the synchronous path short. Long-running generation, document ingestion, and bulk evaluation usually belong in a queue with explicit progress and cancellation behavior.

Retrieval-augmented generation

1. Ingest source documents, remove or mark sensitive content, and preserve source IDs and access-control metadata.
2. Split documents into useful chunks, compute embeddings, and store them with searchable text and metadata.
3. At request time, apply the user's authorization filters before returning candidates; combine lexical and vector search when exact terms matter.
4. Rerank and trim the retrieved context to the model's token budget, then ask for an answer grounded in the supplied sources.
5. Return citations or source links and define what the product does when retrieval is empty or low confidence.

A vector index improves semantic candidate retrieval, but it is not the source of truth for permissions or current documents. Keep an authoritative document store and propagate deletes and access changes to the index. Retrieval quality depends on chunking, metadata, query rewriting, and evaluation—not only the embedding model.

Tools, safety and trust boundaries

• Treat model output and retrieved text as untrusted input. A document can contain prompt injection instructions; never let retrieved content grant new privileges.
• Give each tool a narrow schema and least-privilege identity. Validate arguments in application code and request user confirmation before consequential actions.
• Enforce user and tenant authorization before retrieval, tool execution, and result delivery; filtering only at the final UI is too late.
• Use moderation and policy checks appropriate to the product, but keep deterministic business rules outside the model.
• Redact secrets and personal data from logs. Decide whether prompts and outputs are retained, who can inspect them, and how users can request deletion.

A prompt is not an authorization mechanism. The server must enforce every permission and validate every action even when the model appears to follow instructions.

Latency, cost and graceful failure

Evaluate the application, not just the model

Build a representative test set from real product tasks and measure answer usefulness, groundedness, citation correctness, refusal behavior, and tool selection. Run it when prompts, retrieval settings, models, or data change. Use shadow traffic or a small rollout for changes that affect users, and retain a safe rollback. Production telemetry should include latency, token cost, retrieval hit quality, tool errors, and user feedback without recording secrets.

A practical interview sequence

1. Clarify the user task, data sensitivity, response latency, quality bar, and what actions the assistant may take.
2. Draw authentication, orchestration, retrieval, model inference, validation, and response as explicit boundaries.
3. Explain document ingestion, authorization-aware retrieval, citations, and deletion propagation.
4. Bound tools, context, retries, tokens, and provider concurrency; describe failure and fallback behavior.
5. Finish with offline evaluations, staged rollout, quality monitoring, and cost controls.