Melaya Blog

AI Agent Memory: Types, Architecture and How to Build It

How AI agent memory works: context window vs. long-term memory, the four memory types, write and retrieval policies, retention controls, and model portability.

AI agent memory is application-level infrastructure that stores, retrieves and updates information outside the model, so an agent can use it across sessions. The model itself keeps nothing. You give an agent memory by building (or choosing) a layer that decides what gets written, what gets recalled, and what gets forgotten. This guide covers the memory types, the write and read policies, the controls that keep memory safe, and how to keep memory portable when you swap models.

TL;DR

• The context window is the model's workspace for one call. Memory is a separate store your application writes to and reads from.
• Use four memory types as a checklist: working, episodic, semantic and procedural. Not every agent needs all four on day one.
• The write policy is the hard part. Score, classify and deduplicate before anything becomes long-term.
• Memory needs scope, retention and a deletion path, and consequential actions should stay behind human approval.
• Keep memory in structured, model-independent form so a model swap does not break it.

What AI agent memory is, and what it is not

A language model is stateless. On every request it sees the tokens you send and nothing else. As the Walrus docs put it, the model keeps none of that context once the request ends. Anything that looks like remembering is your code putting the right text back into the prompt.

That is the practical definition: memory is a property of the harness around the model, not of the model. Neurals states this directly and draws the consequence: forgetting behavior is an engineering question about your retrieval and write-back code.

Three things people confuse with memory:

1. The context window. It is a fixed token budget, transient by design. A bigger window does not decide what should persist or how it should reach future calls. Your application still has to make that decision.
2. RAG over documents. Retrieval over a static knowledge base answers questions about content. Memory changes as the agent works: it records preferences, outcomes and state.
3. Chat history. A transcript is raw material. Memory is what you extract from it, keep, and retrieve later.

Why bother? Long-term memory pulls only the relevant pieces into context on demand, which avoids replaying entire histories and prevents token-budget crowding. That helps cost, latency and correctness at once.

The memory types an agent can use

The vocabulary varies by vendor, but the split is consistent. IBM describes short-term memory for recent inputs and immediate decisions, long-term memory for persistence across sessions, episodic memory for specific past experiences, and semantic memory for generalized factual knowledge. Add procedural memory (successful action patterns) and you have the working set most builders design around.

The ACL 2026 survey on LLM agent memory adds a lower-level view: token-level memory (natural language, good for information that changes often), latent memory (efficient session-level continuity, but transient and capacity-limited), and parametric memory (stored in weights, strongest persistence, costly to write). Its recommendation is to pick mechanisms by how volatile the knowledge is and how widely it is used, then combine layers into a hierarchy.

For most teams building on hosted or local models, token-level memory is the practical choice: it is readable, editable and deletable. Parametric memory is a research-grade option, and the portability section below shows why.

A two-layer architecture that works

Start with two layers and add more only when a failure forces you to.

Layer 1: working memory. The context window holds the current task state, fresh tool results and hard constraints. Re-inject the constraints at the top of every system prompt so they cannot scroll out of attention. A two-layer pattern described by Actian's engineers keeps working memory separate from persistent memory, and runs an extraction pass at session end to move stable facts across.

Layer 2: persistent memory. External storage holds user preferences with exact values, identity facts and task state that must survive a session reset. Store exact values ("currency: USD"), not paraphrases ("prefers dollars, I think").

The loop at runtime, following Redis's developer guide, looks like this:

1. Retrieve session memory by session ID.
2. Recall long-term memories that match the user message, scoped to the owner.
3. Build the model context from the session summary, recent events, recalled memories and the current message.
4. Call the model.
5. Save the new messages as session events.

That is five steps and two stores. Everything else in this article is a refinement of one of them.

Write policy: what gets remembered

Most memory systems fail on writes, not reads. If you store everything, retrieval returns noise. If you store too little, the agent repeats questions.

Chroma's agentic memory guide lays out three extraction timings:

A hybrid of these is common: write a small set of high-confidence facts in real time, and run a fuller extraction at the end.

The same guide suggests classifying each memory at write time. Useful fields: memory type (fact, instruction, past result), phase relevance, scope (user-specific or global), confidence and source. Tag these once and every later retrieval gets cheaper and safer.

A practical write checklist:

• Score importance before writing. A one-off typo fix is not a preference.
• Deduplicate. Check whether the fact already exists.
• Resolve conflicts. A new preference should supersede the old one, not sit beside it. TiDB's architecture guide calls this superseding logic and notes that the write policy is the hard part.
• Record provenance. Source, timestamp and confidence travel with every record.
• Redact secrets before any persistent write. API keys and passwords never belong in a memory store.

The OpenAI Agents SDK cookbook shows a compact version of this: a state object (user profile plus global memory notes) is injected into the system prompt at session start, updated through a save_memory_note tool call during the session, then consolidated asynchronously at the end through deduplication, conflict resolution and forgetting of stale or low-confidence entries.

Read policy: retrieval that stays relevant

Writing well gets you half way. The other half is pulling back the right three records out of three thousand.

Use different read paths for different memory categories. Exact lookup suits task state. Identity plus filters suits facts and preferences. Semantic similarity, ranked by recency and confidence, suits episodic history, as TiDB describes. Pure vector similarity is rarely enough on its own: the ACL survey notes that effective retrieval depends on adaptive, context-aware access and, for long histories, on temporal, causal or relational structure.

Practical rules:

• Retrieve at decision points, not on every token. Pull memory when the agent chooses a tool, a target or a recipient.
• Filter before you rank. Scope by user, tenant, task and permission first, then rank by relevance and recency.
• Cap what you inject. Three to five records, labeled with source and timestamp, beat thirty unlabeled ones.
• Keep memory out of the instruction channel. Retrieved text is data. Do not let it override system policy.

The last rule matters for security. A memory store that ingests web pages or emails can be poisoned with instructions. See how we approach this in our write-up on prompt injection protection, and how grounding reduces made-up answers in Preventing AI Agent Hallucinations by Design.

Persistence across sessions and across models

To give an agent long-term memory that survives restarts, keep the store outside the model and outside any single provider. Then the model becomes replaceable: Claude today, a local Qwen through Ollama next quarter, with the same memory underneath.

Research shows this is not trivial. A controlled study, Does Your Agent's Memory Survive a Model Upgrade?, found that fixed-schema knowledge graphs transferred reliably when the writing model was swapped: accuracy moved by only +0.0004. Compressed free-text notes were tightly coupled to the model that wrote them, with accuracy shifting by +9.91 or -13.28 percentage points depending on migration direction. For RAG, a partial embedding migration using a 50/50 mixed index recovered only 4.96 points of accuracy versus 11.90 from full re-embedding.

The lesson for builders:

• Prefer structured records (typed fields, exact values) over free-form summaries written by one model.
• If you change embedding models, re-embed the whole index. Do not mix.
• Keep the raw source events so memory can be rebuilt after a model change.

Two research directions point further. RPMem compiles sessions into model-independent latent memory and maps it to backbone-specific LoRA parameters, reaching 85.52% on PERMA with Qwen3-8B. A paper on runtime-independent persistent agents proposes separating identity, memory and executable body from replaceable models and servers, with a quiesce, checkpoint, validate, bind, rehydrate, resume protocol. Both are research, so treat them as direction rather than a drop-in recipe.

Protecting memory: scope, retention, deletion

Memory is stored user data, so it needs the same controls as any other store.

Scope. Filter by actor, tenant, permissions and freshness before ranking. An agent working for one customer must not recall another customer's records.

Label. Attach source, timestamp, confidence and trust level to each retrieved record, so the model and your audit log can tell a verified fact from a guess.

Retain deliberately. Let short-term memory expire quickly. Keep long-term records only for a documented purpose. Application-level guidance suggests a TTL of 30 to 120 minutes for working memory, as AIS recommends.

Build a deletion path. Removing a record from the vector index does not remove it from the source of truth. Delete from both, log the deletion, and design the path before you have a deletion request, not after. If you handle personal data of California residents or EU residents, have your counsel confirm what the CCPA or GDPR requires of your specific memory design.

Gate consequential actions. Memory can be wrong. A stale address or an outdated approval limit is a memory bug, and the cost rises when the agent acts on it. In Melaya, every consequential action (publishing, sending, paying, writing to connected apps) runs behind human approval, and runs are traceable and replayable, so you can see which recalled fact drove which step.

Build it step by step

The order below follows the staged approach AIS recommends: start simple, add layers when tasks demand them.

1. Start stateless. Run the agent with retrieval over your documents only. Measure where it fails.
2. Add a working-memory buffer and an episodic log once tasks span hours or days.
3. Add a semantic store when the same facts and preferences keep recurring. Use typed fields and exact values.
4. Add tool-scoped memory when latency or cost comes from repeated API calls: cache the results with a TTL.
5. Share memory across a team only through curated promotion, with review and policy, once agents collaborate.
6. Set retention and deletion before you ship, and test the deletion path end to end.
7. Replay and inspect. Check traces for wrong recalls, and tune scoring and decay accordingly.

How Melaya handles this

Melaya is bring-your-own-model: you connect any cloud or local AI model (Claude, GPT, Gemini, DeepSeek, Mistral, Qwen, Ollama, LM Studio) and Melaya supplies the tools, memory, schedules, approvals, triggers and multi-agent pipelines around it. We do not sell a model and agents do not require one from us. Because the tool and memory layer sits outside the model, you can change models or run on local infrastructure without rebuilding your workflows.

The visual Agent Builder lets you combine memory with tools, schedules and approvals in one pipeline, and each run is traceable and replayable. For a closer look at the product direction, see our July 2026 recap, the October 2026 recap on open-weight agents, and the Melaya Assistant page for work across connected business systems.

Our view: memory belongs in the tool layer

We think the model is the brain and Melaya is the controlled pair of hands. Memory is the notebook those hands keep, and it should not be locked inside any one brain.

That is why we build memory, permissions and approvals as a layer you own. When a team changes from a cloud model to a local one, the notebook stays. When an agent recalls something wrong, the execution trace shows what it recalled and which step used it, and a person approves the consequential step before it happens.

Our position is simple: memory without governance is a liability, and governance without memory makes agents repeat themselves. Granular permissions, human approval and execution history make memory safe to use at work.

Melaya Team

Start building with Melaya

If you want agents with memory that stay model-independent and sit behind human approval, start in the Agent Builder. Sandbox is $0 per month. Outpost is $20 per month, Forge is $49 per month or $469 per year, Bastion is $129 per month or $1239 per year, and Citadel starts from $490 per month. Add-on modules such as Mobile Agents, Browser Agents and Melaya Assistant are $9.99 per month each, and Melaya Marketing is $14.99 per month.

See the full plan list on the Melaya pricing page. If your agents need to work in a browser, look at Browser AI agents. For Android apps, see the Android AI agent page. Mobile control is Android only.

FAQ

What is the difference between an agent's context window and its memory?

The context window is the set of tokens the model sees on one request, and it is gone when the request ends. Memory is a store outside the model that your application writes to and reads from. You retrieve relevant records from memory and place them into the context window for each call.

How do I give an AI agent long-term memory?

Store facts, preferences and task state in an external store, such as a database or a key-value store. At session start, inject the relevant records into the prompt. During and after the session, write new facts through a controlled write step that scores, deduplicates and resolves conflicts.

Should I store memory as summaries or structured records?

Use structured records for anything you need to rely on. A controlled study found that fixed-schema knowledge graphs survived a model swap with almost no accuracy change, while compressed notes were tightly coupled to the writing model. Summaries are fine as a supplement, with the raw events kept.

How do I stop an agent from using stale memory?

Attach timestamps and confidence to every record, rank by recency as well as relevance, and let new facts supersede old ones. Expire short-term memory quickly, and review long-term records on a schedule. For consequential actions, keep a human approval step so a wrong recall is caught before it acts.

Sources

• What Is AI Agent Memory? (IBM)
• AI Memory: Types, Architecture and Agent Memory (Arize AI)
• How AI Agent Memory Works (Walrus Docs)
• Context vs. memory (Neurals)
• LLM Agent Memory: A Survey from a Unified Perspective (ACL Anthology)
• How to Build Persistent Agent Memory Across Sessions (DEV)
• Redis Agent Memory developer guide
• Agentic Memory (Chroma docs)
• AI Agent Memory Architecture Guide (TiDB)
• Context Engineering for Personalization (OpenAI Cookbook)
• Does Your Agent's Memory Survive a Model Upgrade? (arXiv)
• RPMem: Learning Long-Term Recurrent Parametric Memory (arXiv)
• Runtime-Independent Persistent Agents (arXiv)
• Practical Memory Patterns for Reliable, Longer-Horizon Agent Workflows (AIS)

Recommended

• Preventing AI Agent Hallucinations by Design
• AI Agent Builder: Visual, No-Code, Free to Start