Types of Memory Used in AI Systems

The biggest limitation in today’s AI agents is not their fluency. It is memory. Most LLM-based systems forget what happened in the last session, cannot improve over time, and fail to reason across multiple steps. This makes them unreliable in real workflows. They respond well in the moment but do not build lasting context, retain task history, or learn from repeated use. A recent paper, “Rethinking Memory in AI,” introduces four categories of memory, each tied to specific operations AI agents need to perform reliably: 𝗟𝗼𝗻𝗴-𝘁𝗲𝗿𝗺 𝗺𝗲𝗺𝗼𝗿𝘆 focuses on building persistent knowledge. This includes consolidation of recent interactions into summaries, indexing for efficient access, updating older content when facts change, and forgetting irrelevant or outdated data. These operations allow agents to evolve with users, retain institutional knowledge, and maintain coherence across long timelines. 𝗟𝗼𝗻𝗴-𝗰𝗼𝗻𝘁𝗲𝘅𝘁 𝗺𝗲𝗺𝗼𝗿𝘆 refers to techniques that help models manage large context windows during inference. These include pruning attention key-value caches, selecting which past tokens to retain, and compressing history so that models can focus on what matters. These strategies are essential for agents handling extended documents or multi-turn dialogues. 𝗣𝗮𝗿𝗮𝗺𝗲𝘁𝗿𝗶𝗰 𝗺𝗼𝗱𝗶𝗳𝗶𝗰𝗮𝘁𝗶𝗼𝗻 addresses how knowledge inside a model’s weights can be edited, updated, or removed. This includes fine-grained editing methods, adapter tuning, meta-learning, and unlearning. In continual learning, agents must integrate new knowledge without forgetting old capabilities. These capabilities allow models to adapt quickly without full retraining or versioning. 𝗠𝘂𝗹𝘁𝗶-𝘀𝗼𝘂𝗿𝗰𝗲 𝗺𝗲𝗺𝗼𝗿𝘆 focuses on how agents coordinate knowledge across formats and systems. It includes reasoning over multiple documents, merging structured and unstructured data, and aligning information across modalities like text and images. This is especially relevant in enterprise settings, where context is fragmented across tools and sources. Looking ahead, the future of memory in AI will focus on: • 𝗦𝗽𝗮𝘁𝗶𝗼-𝘁𝗲𝗺𝗽𝗼𝗿𝗮𝗹 𝗺𝗲𝗺𝗼𝗿𝘆: Agents will track when and where information was learned to reason more accurately and manage relevance over time. • 𝗨𝗻𝗶𝗳𝗶𝗲𝗱 𝗺𝗲𝗺𝗼𝗿𝘆: Parametric (in-model) and non-parametric (external) memory will be integrated, allowing agents to fluidly switch between what they “know” and what they retrieve. • 𝗟𝗶𝗳𝗲𝗹𝗼𝗻𝗴 𝗹𝗲𝗮𝗿𝗻𝗶𝗻𝗴: Agents will be expected to learn continuously from interaction without retraining, while avoiding catastrophic forgetting. • 𝗠𝘂𝗹𝘁𝗶-𝗮𝗴𝗲𝗻𝘁 𝗺𝗲𝗺𝗼𝗿𝘆: In environments with multiple agents, memory will need to be sharable, consistent, and dynamically synchronized across agents. Memory is not just infrastructure. It defines how your agents reason, adapt, and persist!

Last week I gave a talk at AICAMP NYC and had a really long line of questions around AI memory. It seemed like many founders and developers are struggling to have meaningful conversations about memory, because they there is a lot of fundamental misunderstanding about memory architecture. There are actually three distinct layers of memory in generative AI: 𝗟𝗮𝘆𝗲𝗿 𝟭 - 𝗧𝗵𝗲 𝗙𝗼𝘂𝗻𝗱𝗮𝘁𝗶𝗼𝗻 𝗠𝗼𝗱𝗲𝗹 𝗟𝗮𝘆𝗲𝗿: This is the lowest level: model parameters stored in server DRAM that define how an LLM behaves and what it "remembers" from training. ___________ 𝗟𝗮𝘆𝗲𝗿 𝟮 - 𝗞𝗩 𝗖𝗮𝗰𝗵𝗲: This is the middle layer automatically generated during inference. The KV cache is responsible for helping LLMs respond faster to follow-up questions. It's stored on GPU HB (highbandwidth) memory and CPU DRAM but it is rapidly expanding in size and creating new hardware challenges as there is not enough memory capacity on these two tiers. This is also creating a need for projects like NVIDIA Dynamo that have distributed, shared multi-node memory architectures. ___________ 𝗟𝗮𝘆𝗲𝗿 𝟯 - 𝗖𝗼𝗻𝘁𝗲𝘅𝘁 𝗠𝗲𝗺𝗼𝗿𝘆: The top layer. This is the layer that users experience most directly. Context Memory is your conversation history, context windows, and persistent memory. You see it on the left hand side in the form of historical convos in an app like ChatGPT which allows you to pick up from where you last left off. If you haven't tried yet, ask ChatGPT what it knows about you...you'll be amazed. This is the context memory layer and it is separate and distinct from KV Cache and the associated LLMs themselves. ___________ 𝗞𝗘𝗬 𝗧𝗔𝗞𝗘𝗔𝗪𝗔𝗬 Layer 3 is also where your sensitive data lives and where data portability and privacy concerns matter most — especially for the enterprise: When you use ChatGPT, all your sensitive information gets stored in ChatGPT's memory layer. Even OpenAI's new standalone "ChatGPT Memory" is still running on OpenAI's servers, and not under your control. The Context Memory layer is where I see some enterprises and financial services firms being the most trusting of 3rd parties to own and store sensitive trade secrets. I might even characterize some approaches as borderline careless or reckless, because process knowledge and even IP in the form of code snippets and sensitive enterprise data are being shared with services. I think the reason this is happening is that most people don't know how to build and manage their own AI memory and context layer. When you're building your next AI product, make sure you're making decisions that protect your enterprise's edge in today's AI race.

😵 Woah, there’s a full-blown paper on how you could build a memory OS for LLMs. Memory in AI systems has only started getting serious attention recently, mainly because people realized that LLM context lengths are limited and passing everything every time for complex tasks just doesn’t scale. This is a forward-looking paper that treats memory as a first-class citizen, almost like an operating system layer for LLMs. It’s a long and dense read, but here are some highlights: ⛳ The authors define three types of memory in AI systems: - Parametric: Knowledge baked into the model weights - Activation: Temporary, runtime memory (like KV cache) - Plaintext: External editable memory (docs, notes, examples) The idea is to orchestrate and evolve these memory types together, not treat them as isolated hacks. ⛳ MemOS introduces a unified system to manage memory: representation, organization, access, and governance. ⛳ At the heart of it is MemCube, a core abstraction that enables tracking, fusion, versioning, and migration of memory across tasks. It makes memory reusable and traceable, even across agents. The vision here isn't just "memory", it’s to let agents adapt over time, personalize responses, and coordinate memory across platforms and workflows. I definitely think memory is one of the biggest blockers to building more human-like agents. This looks super well thought out, it gives you an abstraction to actually build with. Not totally sure if the same abstractions will work across all use cases, but very excited to see more work in this direction! Link: https://lnkd.in/gtxC7kXj