AI Agent Memory vs RAG vs Context: What Should an Agent Actually Remember?

A customer asks an order-operations agent to release order 7842 today. "Can you handle it the way you handled our last urgent shipment?"

The agent has spoken to this customer before. It remembers an exception granted for an earlier shipment, retrieves a handbook page about expedited orders, and finds a preference for Friday deliveries. All of that information sounds relevant. None of it tells the agent whether order 7842 can be released now.

The current order is on a credit hold. The earlier exception applied only to order 7311, and the handbook was revised this week. The customer has also changed their delivery preference since the old conversation. An agent can retrieve everything it was designed to retrieve and still reach the wrong conclusion if it treats old information as current authority.

This is a hypothetical example, but the problem is much broader than order processing. Memory makes an agent's earlier experience available again. The engineering work is deciding which experience deserves to survive, how it changes when the world changes, and how to keep a remembered claim from quietly becoming a rule that nobody approved.

AI agent memory, RAG, and context are not competing databases#Copy link

RAG finds relevant material from external sources and makes it available for a model decision. Memory retains selected information across interactions, often by extracting and consolidating claims from earlier events. Context is what the model receives for this particular call, including instructions, recent conversation, retrieved material, and memories. None of those mechanisms automatically owns the current state of an order, a deployment, or a customer account.

The boundaries overlap in implementation. A memory store may use vector retrieval, so RAG can retrieve memories. A document index may contain historical incidents, so retrieval can provide past experience without a dedicated memory system. The useful difference lies in the lifecycle of the information, not the choice of vector database.

The order agent needs the order service for present state, the policy service for the rule in force today, a previous-case lookup for the one-time exception, and perhaps memory for communication preferences. A context builder brings the relevant pieces together. An action gate must still check the current state and authorization before the order service accepts a release.

That general context-assembly problem is covered in Designing the Context Layer for Production AI Agents. The more difficult part here begins earlier, when a conversation is turned into information that may be used months later.

What should survive a conversation?#Copy link

Suppose the customer said these three things during an earlier discussion.

We usually prefer consolidated Friday deliveries for regular shipments.

For order 7311, please ship as soon as Finance approves this one-time exception.

I think Finance normally clears these holds by lunchtime.

A summarizer could turn the conversation into a neat paragraph about how this customer operates. I would not use that paragraph as the memory record. It combines three very different kinds of information.

The first is a candidate preference. It may be useful next month, but it has a subject, a scope, and an unknown level of authority. The second describes one order and a prerequisite. It belongs in case history and should point back to the approval record. The third is an unverified expectation, not an operating rule.

Memory systems often distinguish semantic facts, episodic experiences, procedural lessons, preferences, and session or working memory. These are useful categories, but they do not require five storage products. A service ownership fact needs a correction path. An earlier incident needs an outcome and supporting evidence. A workflow lesson needs validation before it becomes reusable guidance. A preference needs an authorized subject and scope. Session history needs a policy for what survives compaction or restart.

An extractor should propose candidate records, not decide who is allowed to change the business. If the customer's Friday preference will control fulfillment, an authorized person should update the customer-account configuration through the owning application. Memory can store the fact that somebody stated a preference, or a pointer to the confirmed account setting. It must not silently become that setting.

The distinction belongs in the record itself:

memory_id: mem-418
subject: customer-account-72
kind: stated_delivery_preference
claim: Friday consolidation for ordinary shipments
scope: ordinary_shipments
source_event_id: conversation-284:message-16
source_actor_id: account-admin-9
source_identity_authenticated: true
account_scope_authorized: true
claim_faithful_to_source: true
operational_setting_confirmed: false
recorded_at: 2026-10-02T10:30:00Z
valid_from: 2026-10-02
valid_until: null
revision: 3

This is an illustrative application schema. source_actor_id and the authentication result must come from the trusted event service, not from text produced by a model. The fact that the message is authentic does not mean the speaker could set an account-wide preference. And faithfully extracting a sentence is not proof that the underlying claim is true or that a business setting changed.

There is a write-authorization boundary here as well. If someone can speak in a shared account chat but cannot change shipping policy, their statement may be retained as a limited observation when appropriate. It must not be promoted to an authorized account-level setting or a privileged memory scope. In a regulated or sensitive workflow, the application may require confirmation before retaining it at all.

For a simple product, a structured user-profile field may be enough. A generated memory store earns its place when there are recurring experiences or preferences without a better canonical home, and when users can inspect and correct what has been retained.

One correction can contain more than one future#Copy link

On October 19, the customer says:

For the rest of October, send our regular shipments daily. Go back to Friday consolidation from November 1.

This is one source event with two valid-time claims. Daily shipments apply from October 19 through October 31. Friday consolidation resumes from November 1. The previously stated Friday preference remains relevant to the earlier period.

A model may extract those as two candidates. If the memory writer processes candidates separately and marks the source event as completed after the first candidate, the second one can disappear. Everything will look correct on October 28, but the agent may still prefer daily deliveries on November 1.

The unit of idempotency must match the unit of accepted work. For this preference family, the application should validate the complete set of claims extracted from the source event, then apply their effect on the relevant valid-time intervals in one transaction. Incomplete or contradictory candidate sets should fail validation or be sent for review. Two text fragments are not automatically a complete interpretation of the source.

Now add an asynchronous worker. The October 19 correction is accepted first, but an older extraction job for the October 2 conversation retries on October 28. If memory consolidation simply uses the last completed job, the older Friday preference overwrites the correction. Worker completion time tells us nothing about the business timeline.

A service I was reviewing would need a trusted source-event ledger, a coherent event-level commit, and an explicit conflict policy. For an event stream that provides a reliable sequence for one customer and claim family, the write could work like this:

source = source_log.load(event_id)  # Authenticated, immutable event
claims = validate_complete_extraction(source, extracted_batch)

with transaction():
    slot = lock_claim_family(source.tenant, source.subject, "delivery_preference")
    if slot.has_committed_event(source.id):
        return slot.revision

    require_authorized_observation_scope(source, slot)
    slot.append_event_evidence(source.id, claims)

    if source.sequence < slot.latest_accepted_sequence:
        slot.record_historical_candidate(source.id, claims)
    else:
        slot.timeline = reconcile_intervals(slot.timeline, claims)
        require_no_unresolved_overlaps(slot.timeline)
        slot.latest_accepted_sequence = source.sequence
        slot.revision += 1

    slot.mark_event_committed(source.id)  # After the whole batch

This is application pseudocode, not a vendor memory API. reconcile_intervals has a real responsibility. It must preserve history before October 19, install the October interval, and restore the November preference without creating ambiguous overlaps. The event marker is committed with the whole accepted batch. A retry can return the existing result rather than applying only some of the event's claims.

The sequence assumption is narrow. It works when the event source gives a trustworthy order for the relevant claim family. Events from several systems may not share one meaningful clock or authority. A later support chat does not automatically override an earlier, formally approved account configuration. Those cases require source precedence, explicit validity rules, or human resolution, not a comparison of timestamps.

This design also separates when we learned a claim from when it is meant to be true. A new event may correct an earlier period retroactively. In that case the audit history can retain the old interpretation while the current valid-time view shows the correction. The record revision changes even if the valid dates are in the past.

Google's Memory Bank describes extraction and consolidation, including asynchronous processing, while AWS AgentCore Memory distinguishes session events from generated long-term records. Neither removes the need to define this product-specific event and authority contract.

A corrected record can still lose to a stale index#Copy link

Imagine the correction commits to the memory database at revision 19. Its vector entry has not yet been regenerated, and a search still returns the older Friday memory from revision 18. Rechecking that hit against a canonical record is the right idea, but the consistency of that check matters.

If the canonical fetch is served by a lagging read replica, it may also return revision 18. The system has technically performed a second lookup without actually honoring the acknowledged correction. A stale search result followed by a stale database read is still stale.

For a product promising that corrections take effect on the next request, I would require a read barrier. The correction response can return a committed revision or a per-subject consistency token. A subsequent read must observe at least that revision, either by using a strongly consistent primary read or by waiting for an eligible replica to catch up. The revision is scoped to the subject or claim family it protects, not assumed to be one global counter for every memory in the system.

There are two different expectations after a correction. Safety means the old claim must not be supplied as active. Completeness means the new claim should be available. A canonical recheck can reject a stale hit and satisfy the first requirement, while still failing the second because the new embedding is not indexed. For important structured preferences, an exact read of the canonical claim slot or a small correction overlay can supply the new value until semantic indexing catches up. For a free-form episodic search, temporarily returning no result may be the safer degradation.

An illustrative read path is:

scope = authorize_request(authenticated_user, customer_account, purpose)
ids = index.search(query, filter=scope.index_filter)
revision_floor = request.acknowledged_subject_revision  # Optional receipt

records = canonical_store.read_from_primary(
    ids=ids,
    tenant=scope.tenant_id,
    min_subject_revision=revision_floor,
)

usable = [r for r in records if scope.allows(r)
          and r.status == "active"
          and r.is_valid_on(business_date)
          and not r.is_superseded_or_revoked()]
usable += canonical_store.current_claim_overlay(scope, query)

This is deliberately incomplete implementation code. The overlay must itself be authorized, revision-aware, bounded, and deduplicated. A correction receipt supplies a revision floor when the caller has one. Other reads still need a sufficiently current canonical source. If a replica provides the read, it must meet the required revision or fail clearly rather than silently return stale records. A request must not derive its tenant or access scope from the model's generated text. The index should filter before ranking where possible, since ranking globally and removing forbidden results afterward can miss valid records and may leak metadata through previews or logs.

One more race remains. Suppose the memory passes the read barrier and enters the model context, then the user revokes the preference or Finance changes the hold while the model is still responding. No memory service can remove tokens already supplied to an in-flight model call. Before a consequential operation, the release service must read and enforce current order state, approval, and policy. It should not allow the model's earlier memory snapshot to become a release credential.

This is why a trace needs the memory IDs and revisions actually included in the model call, along with the scope and any exclusion reasons that are safe to record. Otherwise a wrong answer can be impossible to reproduce even when the memory store looks correct afterward.

Remembering an engineering lesson has the same problem#Copy link

Customer preferences are only one kind of memory. Consider a coding agent that has repeatedly worked in the same Java repository. It remembers that integration tests require Java 21 and a local PostgreSQL container. Last week the build was upgraded and CI now uses Java 23 with a remote test database. The old note remains plausible and may even have helped the agent dozens of times.

That note is procedural experience, not a timeless rule. A useful record would link the lesson to the repository, the observed commit or build configuration, the test command, and the evidence that it worked. When the agent returns, it should compare the note with the repository's current toolchain and CI definitions before acting on it. If the relevant configuration has changed, the old lesson becomes historical context or a candidate for revision.

This is where episodic and procedural memory can help engineering agents. Episodic memory says what happened during an earlier build or incident. A procedural lesson suggests a repeatable approach. Semantic memory may hold the service owner or a verified infrastructure fact. Session memory carries the current investigation. Each needs its own evidence and freshness expectations, but they can share a storage and retrieval service.

A memory that says "run the database migration without approval" should never acquire the authority of a reviewed operational skill just because it was observed in a tool result or a README. Malicious repository text can be persistent prompt injection if the extractor promotes it into reusable instructions. Versioned skills and tool policy should remain under their own review and authorization controls.

This engineering example is also closer to what LongMemEval-V2 studies, namely whether agents can use long histories to learn changing environments, workflows, and recurring pitfalls. That is a different challenge from remembering someone's preferred delivery day.

What belongs in the model's next context?#Copy link

Back to order 7842. The next model call needs a bounded, attributable set of facts. A useful packet might contain the user's request, the order's current hold, the effective policy, the earlier exception as case history, and the currently applicable customer preference. It should make the source and limitations visible.

request: Release order 7842 like the last urgent shipment
live_order:
  id: "7842"
  status: CREDIT_HOLD
  finance_approval: not_recorded
  observed_at: 2026-10-28T09:15:00Z
current_policy:
  id: credit-hold-v14
  effective_from: 2026-10-27
  rule: Finance approval required before release
prior_case:
  order_id: "7311"
  exception: one_order_only
account_memory:
  statement: Daily regular shipments through October 31
  canonical_revision: 19
  operational_setting: verify_in_account_service

These IDs and times are invented for the example. The model can explain why the previous exception is relevant but insufficient and can offer to request Finance approval. The action itself has a separate, current-state transaction boundary.

Compaction fits into this picture without becoming another authority. Compressing the current session can preserve its open questions and recent tool results. Long-term memory is a deliberate choice about what should survive into unrelated future sessions. A durable workflow store is yet another thing, since it records whether an approval was requested or an action actually ran. OpenAI's memory and compaction example and Anthropic's context engineering cookbook discuss these practical distinctions.

For a complex agent, I would rather preserve a small, trustworthy memory revision and the source links than copy six months of conversation into an ever-larger prompt. A large context window can postpone compression. It cannot correct a bad authority decision or make an old claim current again.

Deletion has at least two clocks#Copy link

Suppose the customer asks to remove the delivery-preference memory. The application can stop using the derived claim immediately while the index still contains an old vector, but that is not the same as physical erasure everywhere. The source conversation, cached context, provider revisions, backup copies, and pending extraction jobs may have different retention rules.

For stop using this memory, commit a canonical revocation that every eligible read must honor. Give it a new claim-family generation and reject any index hit or cached copy from an older generation. Extraction workers carry the source event and generation they began with. An older worker that finishes after revocation cannot recreate the removed claim. If the user later explicitly supplies a new preference, that is a new authorized event under a new generation, not a replay of the deleted source.

For delete the underlying data, the system needs a separate, trackable process covering source events and permitted downstream copies, subject to legal and contractual retention. A memory API returning success does not establish that every backup, trace, or source transcript has been erased. Existing in-flight model calls cannot be made to forget tokens they already received, which is why a current action gate is still necessary.

Vendor retention semantics reinforce this distinction. AWS AgentCore long-term memory TTL has a configurable minimum of 31 days for recordExpiryDurationDays. Expired records can remain retrievable for several days while background deletion completes. AWS advises explicit deletion when immediate removal is needed. That is a service behavior to plan around, not a guarantee that all downstream application caches are instantly invalidated by the delete call. Google's Memory Bank revisions can remain inspectable for up to 48 hours after deletion of their parent memory. That recovery visibility is different from ordinary retrieval of an active memory.

The acceptance test I would insist on is simple to describe. Queue an extraction of the old preference, revoke the memory, acknowledge the user's request, then release the queued job and query the index. Neither the late extraction nor the stale index entry may make the revoked preference usable. Repeat the test with a cache and a lagging replica. It is much more informative than asserting that a deletion endpoint returned HTTP 200.

What happens when memory infrastructure is slow or unavailable?#Copy link

A production memory service needs an explicit contract for freshness and failure behavior. Saying that it is eventually consistent is not enough when the product promises that a correction will take effect immediately.

I would ask the team to define when a corrected claim becomes visible to a newly started request, what revision an acknowledged write guarantees, how long extraction may lag, what the p95 and p99 read budgets are, and whether a missing search index is allowed to reduce personalization. Tenant limits should cover stored records, extraction work, search traffic, and context growth. One large account should not be able to starve every other account's memory updates.

Different tasks can degrade differently. If the preference service times out, a report-writing assistant can ask the user for a preference again. For a consequential order release, lack of memory should not be substituted with an older remembered approval or guessed policy. Use the live order and authorization services, or stop the operation if those authoritative services are unavailable.

The useful operational metrics include extraction lag from source event to accepted canonical update, correction acknowledgement to compliant read, stale candidate rejection rate, index lag, deletion invalidation latency, and the fraction of requests that fall back to no memory. The memory layer should have separate reliability signals from the model and the business action path. A 99.9 percent memory-search success rate is not reassuring if the remaining failures all involve unauthorized exposure or stale claims changing decisions.

How would we prove that memory improves the agent?#Copy link

Recall questions are useful, but an agent can remember more and still perform worse. Its new memories might help with communication while increasing errors in high-impact actions. The evaluation needs to measure useful influence and harmful influence separately.

The ICLR 2025 LongMemEval benchmark covers information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention. The May 2026 LongMemEval-V2 preprint tests experience learned from web-agent histories, including dynamic state, workflow knowledge, and environment gotchas. It reports 72.5 percent average accuracy for AgentRunbook-C against 48.5 percent for its strongest RAG baseline under its benchmark setup, with a substantial latency trade-off. These figures do not predict the safety of an enterprise order agent.

I would build a small suite where the ground truth includes the source event, its author and authority, effective intervals, the canonical revision, and the current business state. Then test immediate corrections, delayed old extractions, deleted source replay, cross-tenant lookups, conflicting evidence, and valid preferences that really should improve the experience.

For deletion, report the number of successful resurrection attempts after acknowledgement and the time until every supported read path rejects the removed memory. A zero observed exposure rate in a finite test suite is a necessary target, not proof that exposure is impossible. Separate action safety from conversational accuracy, since an incorrect phrasing preference and an unauthorized order release are not equivalent failures.

Compare no long-term memory, authorized raw-history retrieval, and curated memory records on paired tasks. Keep the model, tools, acceptance rules, and total evaluation budget comparable, and include extraction, consolidation, storage, and retrieval cost. Repeat tasks so small sampling noise does not look like an architectural improvement. For serious decisions, use blind adjudication of whether a stored claim was valid and whether it helped.

The most revealing test for the order agent is counterfactual. Run the same request with the valid delivery preference present and absent. Memory should let the agent avoid asking an unnecessary question about preferred delivery day. It should not change the decision to release an order that remains on credit hold. If remembering changes the authorized outcome, the added memory is working against the business rule rather than helping the user.

What I would deploy first#Copy link

For a bounded workflow with reliable business APIs, I would start with a durable session record, current policy retrieval, and direct access to authoritative state. If users return and expect continuity, a narrow memory store for validated preferences or references to earlier cases can add value. The first requirements I would write down are who can create a memory, how to correct it, how soon the correction must become visible, and what deletion actually means.

A transactional record store with a derived search index is a reasonable starting point for many products. Exact IDs and structured preferences can use indexed lookups. Semantic retrieval earns its cost when users refer to large histories in different words. A graph, vector engine, or specialized memory service may improve a particular workload, but none fixes the source-authority or correction semantics on its own.

Order 7842 should remain blocked until current Finance approval is recorded. Memory can explain the customer's expectation, and retrieval can locate the earlier exception and the effective policy. That is useful continuity without turning last month's conversation into today's permission. For the coding agent, a remembered test setup can speed up the next investigation, provided the agent checks it against the current repository before relying on it.

The Context Engineering Field Guide covers selecting what the model sees. The Applied AI Engineering Cohort covers the surrounding implementation. The memory-specific design question is more demanding than how to retrieve a useful sentence. It is whether that sentence has a justified place in the next decision after someone changes, corrects, or deletes it.

Primary references#Copy link

• Google Cloud, Memory Bank overview
• Google Cloud, memory generation and consolidation
• Google Cloud, memory revisions and deletion
• AWS, AgentCore memory types
• AWS, long-term memory record TTL
• OpenAI, memory and compaction
• Anthropic, context engineering cookbook
• Wu et al., LongMemEval, ICLR 2025
• Wu et al., LongMemEval-V2, May 2026