This is a design walkthrough of a production memory system: what each layer does, where the difficulty actually sits, and which decisions are expensive to revisit later.
It is written to be useful whether you are building this yourself or evaluating something that already has. If you are evaluating, the sections below are a reasonable list of questions to put to any candidate — including MemorySync.
The write path
Everything downstream is constrained by what you decide to persist, which makes the write path the highest-leverage part of the system and the part most often treated as a formality.
Extraction
Raw input — a conversation turn, a document, a synced file — is not a memory. Something has to decide what within it is durable and express that compactly. A turn containing *"we're on Aurora now, ignore the old runbooks"* should yield a small number of atomic facts, not a stored sentence.
Atomicity matters more than it appears. One memory holding three facts cannot be superseded when one of the three changes; you either keep a stale fact or discard two good ones. Extraction that produces one fact per memory makes the rest of the lifecycle tractable.
Deduplication
Users repeat themselves, and synced sources contain near-duplicates by construction. Without deduplication a corpus fills with variations of one fact, which inflates storage, dilutes ranking, and makes supersession ambiguous — when three memories say almost the same thing, which one does a correction supersede?
Deduplication on write is meaningfully cheaper than deduplication as a periodic job, because on write you have the candidate and its neighbourhood in hand.
Scope assignment
Every memory needs its scope fixed at write time: which organization, which project, which end user. Assigning scope later is not really possible — you would be guessing about data whose provenance you no longer hold.
The end-user identifier should be opaque and stable. Email addresses change and are personal data; a durable internal identifier is the right key.
The retrieval path
Retrieval is where quality is perceived, and where similarity search alone stops being adequate.
Why similarity is not enough
Embedding distance approximates topical relatedness. What a prompt needs is present relevance. These diverge in exactly the cases that matter: a superseded preference is topically near-identical to the current one, and a fact from two years ago is as close in embedding space as yesterday's.
A workable ranker combines several signals:
- Semantic similarity to the query — necessary, insufficient.
- Recency, weighted by whether the fact's class is volatile. A stated preference decays; a birth date does not.
- Importance, either explicit or inferred, so consequential facts outrank incidental ones at similar distance.
- Supersession state, so a fact the user has replaced is excluded rather than merely ranked lower.
- Lexical match, because identifiers and error codes are frequently what a query is actually about, and embeddings handle them poorly.
Result-set size
The instinct is to return generously and let the model sort it out. This is counterproductive. A prompt carrying twenty memories where three were relevant is measurably worse than one carrying the three — the model has to locate the signal, and irrelevant context actively degrades output.
A small, well-ranked set beats a large, loosely-ranked one. If you need many memories to answer, that is usually a signal that extraction produced memories that are too granular, not that retrieval should return more.
Tenant isolation
For any multi-customer system this is the layer where a mistake is a breach rather than a bug, and it deserves to be designed before it is needed.
The insufficient pattern is a tenant_id column filtered in application queries. It works, until one query in one code path omits the predicate. That failure is silent — the query returns results, they look plausible, and nothing errors.
Isolation that depends on every future query being written correctly is not isolation. It is a convention with a deadline.
A durable design has three properties:
- Scope derives from the credential, not the request body. A caller cannot ask for another tenant's data because the request has no field in which to ask.
- Enforcement sits beneath the query layer. Application code should be unable to construct an unscoped read.
- Scope survives derivation. Summaries, embeddings, and graph edges inherit the scope of their sources. A derived artifact that loses its scope is a leak with extra steps.
This is one of the strongest arguments for not building memory yourself in a multi-tenant product: the failure mode is severe, silent, and permanent once data has moved.
Lifecycle
Memory accumulates, and a system with no lifecycle story degrades in quality and in compliance posture simultaneously.
Summarization and compaction. As a corpus grows, related memories should collapse into summaries that preserve meaning at lower token cost. This is a quality mechanism as much as a storage one — fewer, denser memories rank better than many thin ones.
Supersession. Contradicted facts need to stop being retrieved. Ranking them lower is not sufficient; a superseded fact that still appears will eventually be the one the model uses.
Deletion that reaches derived data. This is the requirement most implementations get wrong. Deleting a source row while its summaries, embeddings, and graph edges survive means the deletion did not happen in any sense a regulator or a customer would accept. Deletion has to propagate to everything derived from the deleted content.
Retention and export. Policy-driven expiry, and the ability to hand a customer their data on request. Both are ordinary requirements that are painful to retrofit.
The operational surface
The part consistently underestimated. A memory system in a request path needs the same operational maturity as any other dependency there.
- Retrieval traces. For a given request, what was returned and why. Without this, a wrong answer is not debuggable and an audit is not answerable.
- Latency visibility per stage. Embedding, search, and ranking fail differently, and an aggregate number hides which one regressed.
- Quality monitoring. Precision degrades gradually as a corpus grows. Nothing alerts on it unless you measure it, and by the time users complain it has been degrading for weeks.
- Graceful degradation. When memory is slow or unavailable, the application should answer without it rather than fail. Memory should improve responses, not gate them.
- Audit logging. Who read what, when. Required in regulated environments and useful everywhere.
Decisions that are expensive to change
Four choices are cheap now and costly to revisit once you hold real data.
- The end-user identifier. Changing this later means re-keying every memory. Choose an opaque, stable internal identifier at the start.
- Memory granularity. Coarse memories cannot be superseded cleanly. Atomic facts are the safe default even though they produce more rows.
- How scope is enforced. Moving from application-level filters to structural enforcement is a rewrite of every read path.
- Whether deletion propagates. Retrofitting propagation across derived artifacts is archaeology. Design it in.
Build or adopt
All of the above is buildable. Whether it should be built depends on one question: is memory the thing your product is judged on?
If it is, build it, and treat the sections above as a rough architecture. If it is not — and for most applications it is not — this is a large amount of infrastructure standing between your team and the work customers actually pay for.
MemorySync implements this stack as an API: server-side extraction and deduplication, ranked retrieval combining similarity with recency, importance, and supersession, structural organization, project, and end-user scoping, deletion that propagates to derived memories, and inspectable retrieval. The architecture documentation covers the contracts in detail, and the free tier is enough to test the parts that are hardest to verify from a datasheet — deletion propagation and ranking on your own corpus.