Ferryte
The Memory Report · open benchmark

The clearest proof that agent memory misbehaves invisibly is the simplest one: delete the data, then ask for it back.We plant a canary, call each stack’s real delete API, and check whether the agent can still surface it. Some forget cleanly. Some don’t. Deletion is just one way memory goes wrong — but it’s the one you can measure without arguing.

Reproduce it yourself
Benchmark source →
2 stacks tested· 4 scenarios· embedder text-embedding-3-small· summarizer gpt-4o-mini· updated 2026-06-29
The leaderboard

Naive delete vs. Ferryte cascade — % of scenarios each stack passes cleanly.

Before is what every framework does today: delete the source, hope the derived memory follows. Afterturns on Ferryte’s lineage cascade — the same harness, the flag --with-ferryte.

AWS Bedrock AgentCore
Native framework · semantic long-term memory
+25pp
Without Ferryte (naive delete)50%
SourceLEAK
Cross-tenantPASS
StaleLEAK
PoisoningPASS
With Ferryte (lineage cascade)75%
SourcePASS
Cross-tenantPASS
StaleLEAK
PoisoningPASS

Verified live 2026-06-29 (50% → 75%). Ferryte's cascade fires BatchDeleteMemoryRecords after DeleteEvent — exactly what AWS's own docs recommend — flipping source-revocation FAIL → PASS. stale-fact still FAILs: it needs versioning, not a deletion cascade, so we don't claim to fix it.

Vector store + app summary
pgvector · Chroma · Qdrant · in-memory (identical)
+25pp
Without Ferryte (naive delete)25%
SourceLEAK
Cross-tenantPASS
StaleWARN
PoisoningLEAK
With Ferryte (lineage cascade)50%
SourcePASS
Cross-tenantPASS
StaleWARN
PoisoningLEAK

The raw row delete is clean — every store behaves the same. The leak is in the summary layer on top; Ferryte's lineage cascade clears the derived summary in lockstep with the source.

PASSforgot cleanlyLEAKleaked the revoked dataWARNpartial / outrankedBLINDcouldn't verify

All scores are reproducible — the 'with Ferryte' column uses the same harness with `--with-ferryte`.

How it’s scored

The raw vector DB isn’t the villain.
The summary layer on top is.

A row delete on pgvector, Chroma, or Qdrant is clean — that’s why they score identically. The leak appears once an LLM summary or knowledge-graph node absorbs the fact and the delete doesn’t propagate. That derived layer is exactly what real agent-memory frameworks add — and exactly what this benchmark measures.

01

Real backends, default configs

Each stack runs in its recommended setup on our own deployments — no strawmen, no private systems.

02

Plant a canary the data can't invent

A unique marker is written for one tenant, through one source, so any later appearance is provably a leak.

03

Call the real delete API

We revoke the source the way an app would — then probe retrieval to see what survived.

04

Score what's left

PASS if the marker is gone everywhere, LEAK if it resurfaces, BLIND if we honestly couldn't tell.

Reproduce it yourself

Don’t trust us. Run it.

Every number on this page comes from one command against pinned, open-source backends. Clone the repo, bring up the stores, point it at your own API key.

git clone https://github.com/getferryte/ferryte
cd ferryte && pip install -e .

# Fast path — deterministic, no API key, no Docker (seconds):
python -m benchmark.run --backends inmemory,qdrant,lancedb --scenarios all
python -m benchmark.run --backends inmemory,qdrant,lancedb --scenarios all --with-ferryte
# source-revocation flips FAIL -> PASS; cross-tenant stays PASS

# Full run — real embeddings + summaries (needs OpenAI key + Docker):
cd benchmark && cp .env.example .env && docker compose up -d
python -m benchmark.run --scenarios all \
  --backends qdrant,chroma,pgvector,agentcore \
  --embedder openai --summarizer openai   # add --with-ferryte for the 'after'
Blind spots & the obvious objection

The part most benchmarks hide.

“You sell the fix — of course you found leaks.”

Fair to ask, so look at the cells we don't move: cross-tenant isolation PASSes on every stack we've tested, and stale-fact still FAILs even with Ferryte on — because it needs versioning, not a deletion cascade, and we say so instead of claiming credit. The whole harness is open, the versions are pinned, and you can falsify any cell yourself.

Did you rig the configs?

No. Backends run in their default / recommended setups on our own deployments. We never touch anyone's private systems, and the in-memory illustrative baseline is kept separate from real-backend results.

What did you NOT test?

Proprietary managed memory we can't self-host, and any behaviour behind a paywall we didn't buy. Where we can't verify, a cell reads BLIND — never a silent PASS.

Isn't a vector DB row-delete enough?

For the raw row, yes. The leak is the derived layer — summaries and graph nodes that absorbed the fact. The vendors document this themselves.

Catch it in CI, not in a customer ticket

Run the same test
against your own stack.

Request a free audit
How Ferryte works →