ElevenLabs + MemorySync

Overview

Install elevenlabs-memorysync and ElevenLabs agents remember callers across calls — on the web and on the phone. Three tiers in one package: a zero-code webhook tool the agent calls directly (dashboard only), session-start dynamic variables plus an HMAC-verified post-call webhook receiver that sends the caller’s turns from the transcript, and the flagship Custom-LLM proxy that injects memories into every turn under a hard budget (default 1.2s) and sends the caller’s newest turn to fact extraction before the upstream call, with an idempotency seed. Only the durable facts in what the caller said are stored; the model’s replies are relayed to ElevenLabs but not sent to MemorySync. The proxy works with any OpenAI-compatible upstream (competitors hardcode OpenAI), and the novel prompt-tag mechanism — one memorysync-user: {{system__caller_id}} line the proxy extracts and strips — gives phone-caller identity with zero client code. Running proxy and webhook together yields live memory plus an end-of-call sweep: a turn the proxy already sent is recognised and never extracted twice.

Setup status and requirements

Supported version

elevenlabs-memorysync 1.1.0 (PyPI)

Last setup review

2026-08-24

Permissions

A MemorySync API key with read and write scopes; an ElevenLabs workspace with Custom LLM and/or post-call webhooks.

Limits

Python 3.10+. The proxy must be reachable by ElevenLabs over HTTPS. Post-call webhooks are a workspace-level setting (they fire for all agents in the workspace).

Capabilities

• Hard recall budget (default 1.2s) — a reply is never late, tested against a 5s-slow backend
• Any OpenAI-compatible upstream LLM — OpenAI, Azure, Groq, Gemini compat, LiteLLM
• Phone-caller identity via prompt tags reading {{system__caller_id}} — stripped before the model sees them
• No identity → pure passthrough: nothing is sent to MemorySync
• Byte-faithful SSE relay — tool-call deltas (end_call, …) pass through untouched
• Post-call webhook with signature verification pinned to the official SDK scheme
• Live capture + end-of-call sweep share seeds: no turn is extracted twice
• Optional buffer words spoken only when the turn is already slow

Quick Start

# server.py — the agent's Custom LLM
from elevenlabs_memorysync import create_proxy_app

app = create_proxy_app(
    api_key="ms_...",              # MemorySync
    upstream_api_key="sk-...",     # your LLM provider
    proxy_api_key="a-long-random-secret",
)
# uvicorn server:app --port 8013
#
# Agent prompt gains one line for phone identity:
#   memorysync-user: {{system__caller_id}}

Use Cases

Explore More