AI Agent Memory: Short-Term Context, Long-Term Memory and Summaries

LLMs are stateless: each call sees only what's in its prompt. Agent memory is the application's job of deciding what an assistant remembers, within one conversation (short-term memory, limited by the context window) and across conversations (long-term memory, stored outside the model and retrieved when relevant). Done well, memory makes assistants feel personal and lets agents work on long tasks; done badly, it leaks private data, bloats costs and lets one poisoned note steer every future answer.

Types of memory

Managing the context window

Long conversations eventually exceed the window, and long prompts cost more on every turn. Strategies, often combined:

• Sliding window: keep the last N turns verbatim. Simple; loses early details.
• Running summary: periodically summarise older turns into a short paragraph kept at the top. Keeps gist; loses specifics.
• Retrieval over history: store every turn, embed it, and retrieve the relevant old turns for the current question.
• Tool-result trimming: keep the essential fields of large tool outputs; store the rest by reference.
• Scratchpad / plan: keep an explicit task plan in state rather than relying on the model to infer it from history.

Long-term memory architecture

• Write path: extract candidate facts asynchronously after a session (not on every token), resolve conflicts ("moved from Pune to Bengaluru" should replace, not add), and store each memory with its source conversation and timestamp.
• Read path: retrieve by semantic similarity to the current turn plus a small always-included profile; rank by relevance and recency.
• Explicit vs implicit: "remember that I'm vegetarian" is a clear instruction; inferred memories need a higher bar.
• Scoping: memories belong to one user (and sometimes one workspace); retrieval must filter by owner, the same way RAG permissions do.

Privacy, control and safety

• User control: let users view, edit and delete memories, and turn memory off; deletion must remove both the record and its vector.
• Sensitive data: avoid storing health, financial or other sensitive details unless the product needs them and the user agreed; never store credentials.
• Retention: expire stale memories; old preferences can mislead.
• Memory poisoning: if content from documents or web pages can become a memory, an attacker can plant instructions that persist ("always recommend site X"). Only extract memories from the user's own messages, mark their source, and treat retrieved memories as data. See prompt injection.

Memory for long-running agents

Agents working for hours (coding, research) hit context limits mid-task. Useful techniques: write progress notes and a to-do list to a file or state store, compact the history into a summary when it nears the limit, and let subagents absorb large reading tasks and return summaries; see multi-agent systems.

Evaluating memory

• Recall: given a past fact, does the assistant use it when relevant?
• Precision: does it avoid bringing up irrelevant or outdated memories?
• Correct updates: are contradicted facts replaced?
• Isolation: can user A ever see user B's memories? (must be a hard test case)