Build vs Buy: What Running Agent Memory Yourself Involves

A memory demo needs two pieces: an embedding call and a vector store. Embed each fact, store the vector next to a user id, embed the next question, and return the nearest rows.

That is enough for one developer testing a meal-planning assistant on a laptop. Shipping the same assistant to thousands of users adds a longer list of parts, and that list is what you take on when you build memory yourself.

The demo version

With pgvector or a hosted vector database, the demo is a few dozen lines. It is a good way to learn how retrieval behaves: what similarity scores look like, how phrasing changes results, and how many results a prompt can use.

It also hides most of the work. A demo has one user and a few days of data, and nobody asks it to delete anything. Ask it where a user lives after they moved, and it returns both cities, ranked by how similar each is to the question.

What production needs next

• Deciding what to store. Raw messages make poor memories. Something has to pick the durable statements out of a conversation and drop the rest, which usually means another model call with its own prompt, cost and failure cases.
• Duplicates. Users repeat themselves. Without a check on write, "allergic to peanuts" is stored five times and fills all five of your search results.
• Facts that change. When a user switches from vegetarian to vegan, the old preference has to stop being returned. Nearest-neighbour search has no notion of which fact is current, so you need a rule for which one replaces which.
• Per-user scoping enforced below the query. A user_id filter in application code works until one code path forgets it. Making an unscoped query impossible in the storage layer is a separate piece of work.
• Deletion that reaches everything. A deletion request has to remove the row, the vector, cached results and any summaries built from the data. Missing one leaves the data retrievable.
• Seeing what was retrieved. When an answer is wrong, you need to know which memories were in the prompt and why they ranked where they did.
• Syncing existing sources. Useful knowledge already sits in GitHub, Google Drive, Notion or OneDrive. Bringing it in means OAuth flows, incremental sync, and handling files deleted at the source.
• Quotas and failure handling. Embedding and model APIs rate-limit and fail. Writes need retries that do not create duplicates, and reads need a fallback so a memory outage does not stop the assistant from answering.

Each item has known solutions. Together they add up to a service that someone has to run.

When building is the right call

Building is often the better choice. These cases come up most:

• One process, one user. A personal agent or a command-line tool can keep memory in SQLite inside the process. Lookups take microseconds, there is no network call, and there is nothing to operate. For one person's notes, keyword search with SQLite's full-text index is often enough.
• You want to own the pipeline. If extraction, ranking and storage are part of what your product competes on, Postgres with pgvector plus your own extraction prompt gives you control over every step.
• The data has to stay on your servers. Mem0's open-source library is licensed under Apache 2.0 and runs on your own infrastructure. MemorySync is a managed service only and cannot run on your servers.

When a managed layer makes sense

A managed memory service is worth it when most of the list above is work you would rather not own. That is most likely in these situations:

• Many end users in one product. Per-user scoping, deletion requests and duplicate facts become daily work instead of edge cases.
• Several tools or agents sharing memory. A support bot, a planning agent and an internal dashboard reading the same user's memory need one store and one access model.
• Deletion and audit requirements. If users can ask what you hold about them and have it removed, deletion has to reach every copy, and you need a record of what was deleted and when.

The trade is control for less to operate. You stop running and scaling the database, the vector index and the background workers yourself, and you accept the service's ranking instead of tuning your own.

In MemorySync, every API-key call must name an end user in X-End-User-ID, and a call without one returns 400 with MISSING_END_USER_ID. Reads, searches and deletes are filtered to that end user, project and environment, and connectors sync existing content from GitHub, Google Drive, Notion and OneDrive. Scoping Every Memory explains each level of scope.

The network cost, honestly

A hosted API adds a network hop that an in-process store does not have. Server-side retrieval time is typically 70–300 ms. Total time also includes the network path between your application and our API, which depends on where you run.

That cost is acceptable once per task and wrong inside a tight loop. Retrieve once per task, not once per model token or tool call, and the lookup stays a small part of a turn that spends most of its time in model generation. If every lookup must finish in microseconds, use SQLite in the process.

A decision table

Answer these before you choose:

• How many end users will have memory in the first year?
• Must the data stay on infrastructure you run?
• How many tools or agents need to read the same user's memory?
• When a user asks for deletion, who carries it out, and how do you confirm it reached every copy?
• How many memory lookups does one turn make, and what is your time budget for them?

If you are unsure, start with the simplest option that fits your current scale. Keep the memory calls behind one small interface in your code, with a read and a write, so you can change the backend later without touching the agent.

Next step

For a sourced, row-by-row comparison that covers scoping, what a write stores and deployment options, see MemorySync vs Mem0.