Aller au contenu principal

Define memory protection requirements for agent development

Implementation Effort: Medium – Requires defining write, isolation, and retrieval requirements for agent memory, agreeing on a data-handling taxonomy, and wiring content-safety and identity controls into the agent development standard before agents ship.
User Impact: Medium – Agents that persist and recall context will behave differently once protection rules constrain what can be written, retained, and recalled, which can change response behavior for the people who rely on them.

Overview​

Memory lets an agent retain and recall information across interactions, which makes it more helpful but also turns it into a durable control layer for the system. A fact written today can shape tool selection, refusal behavior, and reasoning later — often in a different session or application than the one where it was created. That persistence changes the threat model: a threat actor no longer needs to win in a single prompt. By influencing what an agent stores, they can shape its behavior gradually, exploiting the gap between when malicious content is planted and when it is acted on. This activity establishes the protection requirements every agent must meet before it is allowed to use memory, so that memory poisoning, delayed tool invocation, persistent cross-prompt injection, and cross-context leakage are addressed by design rather than discovered in production. The requirements cover three moments in the memory lifecycle: gating writes on legitimate intent and provenance, enforcing isolation with deterministic controls outside the model, and treating every retrieval as a risk decision where recalled content is candidate context rather than authoritative truth.

This supports Verify explicitly because memory must be validated at both write time — confirming the caller is authorized and the content reflects genuine user intent — and at read time, where recalled content is re-evaluated for malicious or sensitive material before it enters the reasoning context. It supports Use least privilege access by requiring that each agent and subagent can write and read only the memory it needs, scoped by verifiable identity rather than by trusting model instructions. It supports Assume breach by isolating memory per user, agent, and tenant with access controls and encryption, and by applying content-safety inspection at retrieval so that content planted during a compromise cannot silently steer later behavior. If these requirements are not set before agents are built, protection becomes an afterthought bolted onto systems that already trust their own memory: poisoned or stale entries flow straight into reasoning, sensitive data leaks across contexts, and there is no architectural boundary to contain a single compromised interaction from shaping all future behavior.

Design principles​

Microsoft's guidance frames the memory protection requirements around five principles. Use them as the checklist your agent development standard must satisfy:

  • Establish intent and provenance before persistence — Only write memory when it reflects legitimate user intent, aligns to the service's purpose, and carries clear metadata about where it came from.
  • Enforce boundaries outside the model — Control memory access and isolation with deterministic systems, not model prompting, so sensitive memory cannot leak across users, agents, or tenants.
  • Treat retrieval as a risk decision — Treat recalled memory as candidate context and re-evaluate it for relevance, freshness, and tampering before it influences agent behavior.
  • Provide full lifecycle visibility for security teams — Keep clear records of what changed, when, why, from where, and any access attempts, so memory can be investigated, trusted, and safely expired during incident response.
  • Keep users in control — Give users transparency into how memory shapes their experience and meaningful controls to review, edit, and delete it.

Reference​