Add VectorSearchEngine backed by turbovec by edwinyyyu · Pull Request #1499 · MemMachine/MemMachine · GitHub
Skip to content

Add VectorSearchEngine backed by turbovec - #1499

Open
edwinyyyu wants to merge 10 commits into
MemMachine:mainfrom
edwinyyyu:turbovec_engine_atomic
Open

Add VectorSearchEngine backed by turbovec#1499
edwinyyyu wants to merge 10 commits into
MemMachine:mainfrom
edwinyyyu:turbovec_engine_atomic

Conversation

@edwinyyyu

@edwinyyyu edwinyyyu commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Depends on #1460 and is branched from it, so everything here except
Add VectorSearchEngine backed by turbovec and Update turbovec to 1.0.0 and let it publish the index is either that PR's or a merge of main (taken to
resolve uv.lock against the Milvus backend that landed since). Supersedes
#1448.

Purpose of the change

SQLiteVectorStore backed by turbovec is smaller and faster than
SQLiteVecVectorStore backed by sqlite-vec. Both are linear full scans, but
turbovec keeps TurboQuant-compressed vectors in RAM, so its scaling constant is
much smaller and an index is a fraction of the size of the f32 engines'.

Description

Adds TurboVecVectorSearchEngine, a VectorSearchEngine backed by turbovec,
behind a new turbovec optional extra floored at 1.0.0. turbovec indexes a
dimensionality that is a multiple of 8, so any other width is zero-padded up to
one -- exact for both metrics here, since a zero coordinate adds nothing to an
inner product or to an L2 norm -- and every width the other engines accept
works.

Publication is turbovec's own. 1.0.0 is upstream's first stable release, and
what it commits to is the on-disk format: v7 is the only container turbovec
reads or writes, and a file written by 1.0.0 stays readable by later releases.
v7 is also what makes sync possible, and save calls it -- a checkpoint
appends what changed since the last one rather than restating the whole index,
and commits it durably, so a crash at any byte leaves the previous commit
intact and the publication survives a power failure. The engine therefore does
not route through the shared atomic_index_write helper that the hnswlib and
usearch engines use on #1460: wrapping sync would restate the index on every
checkpoint and publish it through the weaker of the two protocols, a rename the
helper's own docstring declines to make durable.

That is what supersedes #1448, which added the same engine against turbovec
0.x, whose write had no publication protocol of its own and wrote straight to
the final path -- an interrupted save left a truncated file where the store
expects a loadable index, and because index_saved makes a published index a
durable contract, that is a hard IndexLoadError rather than a silent rebuild.

Known limitations, both consequences of storing only compressed vectors:

  • get_vectors raises NotImplementedError, since the original vectors are not
    recoverable. That makes the engine usable by EventMemory but not by semantic
    memory, whose feature updates read stored embeddings back.
  • Search is approximate. A self-match lands near the exact score rather than on
    it, so the tests assert ranking and membership rather than magnitudes. The
    quantized inner product also overshoots the cosine range -- 40 of 50
    self-matches at 8 dimensions, and every dimension measured at bit width 2 --
    so SearchMatch scores are clamped to [-1, 1] under a cosine metric, which
    is what its docstring promises and what score_threshold compares against.
    A dot product is unbounded by contract and is not clamped.

Removal is a strong point by comparison: turbovec drops the id from its map
rather than tombstoning it, so deletions stay cheap and a removed key cannot
resurface in results.

Type of change

  • New feature (non-breaking change which adds functionality)

How Has This Been Tested?

  • Unit Test

test_turbovec_engine.py covers construction, add, remove, cosine and dot
search, filtered search, get_vectors raising, score range (clamped under
cosine, untouched under dot), unaligned widths (search, save/load, and a
wrong-width vector refused), and persistence -- a save/load
round-trip, a load replacing a live index, a save leaving nothing but the index
behind, and what a later checkpoint carries, reading a bulk removal and an
append back through a fresh engine. The module is importorskip-guarded, so the
suite still runs without the extra installed.

Test Results: uv run pytest packages/server/server_tests/memmachine_server/common/vector_store
-> 346 passed. ruff check and ruff format --check clean. The ty jobs are
red on main itself (nebulagraph_python.client lost the members the graph
store imports), fixed by #1519 rather than here; nothing ty reports is in
this diff.

Checklist

  • I have signed the commit(s) within this pull request
  • My code follows the style guidelines of this project (See STYLE_GUIDE.md)
  • I have performed a self-review of my own code
  • I have commented my code
  • My changes generate no new warnings
  • I have added unit tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • Any dependent changes have been merged and published in downstream modules (Publish vector index files atomically, not durably #1460 is still open)
  • I have checked my code and corrected any misspellings

Maintainer Checklist

  • Confirmed all checks passed
  • Contributor has signed the commit(s)
  • Reviewed the code
  • Run, Tested, and Verified the change(s) work as expected

edwinyyyu and others added 5 commits August 25, 2026 16:01
SQLiteVectorStore persists each collection's index by calling the search
engine's save(), which wrote directly to the final path. A crash mid-write
left a truncated/corrupt file. Because index_saved=True makes the on-disk
index a durable contract (missing/corrupt is a hard IndexLoadError, not a
silent empty rebuild), an interrupted save could render a collection
unrecoverable.

Write the index to a sibling temp file and swap it into place with
os.replace (atomic on POSIX and Windows on the same filesystem), so a reader
sees either the old or new index, never a partial write; a failed save leaves
the previous index intact. Leftover temp files are cleared on load so a crash
does not leak them across restarts.

Implemented in the engines (shared index_persistence helper) rather than in
SQLiteVectorStore/SQLiteVectorStoreCollection, since the index save location
and number of files written differ across engine implementations.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The swap protects a reader from a torn index, but the vector store also
trims its pending-operation log once `save` returns -- and that log is the
only other copy of those vectors, since the records table stores no vector
column. So the swap reaching disk is load-bearing rather than a bonus:

- fsync the parent directory after the replace, since POSIX `rename(2)`
  leaves the new directory entry in the page cache. Best-effort and ignored
  on failure, matching SQLite's `unixSync`; a no-op on Windows, which has no
  equivalent operation.
- stop swallowing a failed fsync of the temp file. SQLite draws the same
  line -- a file fsync failure raises SQLITE_IOERR_FSYNC while a directory
  fsync failure is ignored -- and `EIO` means the writeback already failed
  and the dirty pages were dropped, which is exactly when the save must not
  be reported as committed. The existing cleanup then leaves the previous
  index in place with the log untrimmed, so the next save retries.
- use F_FULLFSYNC on macOS, where plain `fsync` leaves the data in the
  drive's volatile write cache, falling back when a filesystem refuses it.

State the resulting obligation on `VectorSearchEngine.save` itself, since
that is what the store now relies on: replace atomically, then make the
replacement as durable as the platform allows. An engine whose backend
already implements the whole protocol can delegate to it and skip these
helpers; the rest use `atomic_index_write`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The pending log holds the only durable copy of a vector between
checkpoints -- the records table has no vector column -- so trimming it is
safe only at an instant when the index provably holds those vectors. The
temp-write + rename protocol this PR shipped could not provide that
instant. A rename changes a directory entry, and Windows exposes no way
to flush one: os.fsync is _commit, which is FlushFileBuffers, which is
for file data, and you cannot open a directory to fsync it. The decisive
evidence is SQLite's own -- it threads a directory-sync flag through
every commit-relevant directory operation, honors it in unixDelete, and
declares it /* Not used on win32 */ in winDelete. So os.replace could
return, _save_collection_index could commit its trim durably behind it,
and a power cut could still roll the rename back: records forward, index
back, no copy of the difference left. MOVEFILE_WRITE_THROUGH is not a
fix; its documented guarantee covers copy-and-delete (cross-volume)
moves, not same-volume renames.

Take SQLite's answer, which was not to harden the directory operation
but to stop using one as a commit point (PERSIST commits by zeroing a
header, TRUNCATE by truncating, WAL by appending frames).

A base path now expands into two index slots plus a generation record
each, created once and thereafter only overwritten. A checkpoint writes
the index over the inactive slot and flushes it, then writes that slot's
generation record and flushes that. The record is the commit, and it is
a write into a file that already exists. It holds the generation and its
bitwise complement, so a torn write reads as absent rather than as some
other generation -- all or nothing without needing single-sector
atomicity from the hardware. load takes the highest believable
generation, and deliberately does not fall back to the older slot when
the published index will not parse: the log was trimmed against the
newer one, so the older is stale by exactly the ops that can no longer
be replayed.

Both backends already write straight to the path they are given, which
is what this protocol wants -- verified that repeated saves preserve the
inode and leave no stray files -- so no engine gains a temp file, a
buffer, or a rename.

Durability is entirely the engine's, including which artifact is live.
The store keeps no slot pointer, manifest, or generation, so no schema
change and no migration: what remains is one rule, never trim past what
save says is durable, and _save_collection_index already had that order.
index_path becomes index_base_path since it no longer names a file, and
discarding a collection asks the engine layer which files that covers.

BREAKING CHANGE: an index written by the previous protocol is not
published under the new one, so a collection with index_saved=True
raises IndexLoadError until its index directory is cleared and the
records re-ingested.

Anomaly tests walk every crash point in the publish sequence by
constructing the on-disk state each would leave, plus one that pins the
ordering itself (a failed index write must publish nothing) since
state-based tests cannot observe it. Verified against three deliberate
breaks -- dropping the complement check, writing the record first, and
reusing one slot instead of alternating -- each caught.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The two-slot generation-record protocol bought a guarantee we have
decided not to make: that a save survives a power failure. Every engine
would have to implement and maintain that protocol, and the failure it
buys out is bounded -- search recall for the records applied since the
last checkpoint, repaired by re-ingesting them. The direction that
actually costs, a published index that will not parse, is closed by the
atomic swap on its own.

So this returns to the temp-file-plus-rename publication and spends the
difference on stating the contract instead of strengthening it: `save`
publishes atomically, never durably; the store trims the pending log
behind a publication a power failure can revert; a record whose vector
is lost that way still resolves by uuid, is absent from search until it
is upserted again, and nothing here detects the gap for the caller.

Reverts the durability and engine-owned-publication commits, keeps the
atomic swap, and adds a store-level test that reconstructs a reverted
publication deterministically -- restore the previous index bytes after
the trim -- to pin the direction it fails in.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`update_feature` reads the stored embedding back when a caller updates a
feature without supplying one, and that is the only place in the server
that depends on the index still holding a vector. With publication now
atomic rather than durable, a power failure can leave a feature whose
row is intact and whose vector is not -- a state this path reported as
"Vector record not found", which points the caller at the wrong thing
and hides the repair.

Split the two cases. A record that is genuinely absent keeps the old
message; a record whose embedding the index no longer holds says so and
names the fix, which is to pass a fresh embedding.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@edwinyyyu
edwinyyyu force-pushed the turbovec_engine_atomic branch from 04a827c to 2239abd Compare August 25, 2026 23:13
turbovec keeps TurboQuant-compressed vectors in RAM, so an index is a
fraction of the size of the f32 engines' and a full scan has a much
smaller scaling constant than sqlite-vec's on-disk one. Search is
approximate as a result: scores land near the exact value rather than on
it, and the tests assert ranking and membership instead of magnitudes.

Two consequences of storing only compressed vectors are worth naming.
`get_vectors` raises `NotImplementedError`, since the originals are not
recoverable -- which makes this engine usable by EventMemory but not by
semantic memory, whose feature updates read stored embeddings back. And
removal is exact and cheap: turbovec drops the id from its map rather
than tombstoning, so a deleted key cannot resurface in results.

`save` publishes through the shared atomic-write helper, like the other
engines, so an interrupted save leaves the previously published index
intact rather than a truncated file the store would treat as a hard
load error. `load` clears any temp file a previous save left behind.

Signed-off-by: Edwin Yu <edwinyyyu@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1.0.0 is upstream's first stable release, and what it commits to is the
on-disk format: v7 is the only container turbovec reads or writes, and a
file written by this release stays readable by later ones. The extra
floors there rather than at 0.7.0.

v7 is also what makes `sync` possible, and that changes how this engine
saves. `save` no longer routes through the shared `atomic_index_write`
helper. turbovec publishes the index itself: a checkpoint appends what
changed since the last one rather than restating the whole index, and
commits it durably -- a crash at any byte leaves the previous commit
intact, and unlike the helper's rename the publication survives a power
failure. Wrapping that would restate the index on every checkpoint and
publish it through the weaker of the two protocols. `load` drops
`clear_stale_index_temp` with it, since the engine no longer writes the
`<path>.tmp` sibling it cleared.

The tests follow. Save leaving no temp file is now an assertion about
the whole directory, since turbovec names its own temp; the stale-temp
test goes with the protocol it tested; and a new test pins what a later
checkpoint carries, reading a bulk removal and an append back through a
fresh engine.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Edwin Yu <edwinyyyu@gmail.com>
`SearchMatch` documents a cosine score as a cosine similarity in [-1, 1],
and `SQLiteVectorStoreCollection` compares `score_threshold` against it
directly. turbovec's score is a quantized inner product, so it lands a
hair outside that range: at 8 dimensions and the default bit width, 40 of
50 self-matches score above 1.0 (max 1.0066), and at bit width 2 they
exceed it at every dimension measured (1.0038 at 768, 1.0025 at 1536).
Publishing 1.0066 as a cosine similarity is a broken promise, however
small, so the hair is clamped where the match is built.

A dot product is unbounded by contract and passes through untouched.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Edwin Yu <edwinyyyu@gmail.com>
turbovec indexes a dimensionality that is a positive multiple of 8, so the
engine refused a width every other engine in the tree accepts -- 300, the
width of a spaCy or word2vec vector, among them -- with turbovec's own
error rather than a store-level one.

The width is now rounded up and vectors are written into the leading
columns of a zeroed buffer. Padding is exact for both metrics this engine
serves: a zero coordinate adds nothing to an inner product and nothing to
an L2 norm, so the padded index answers as the unpadded one would. The
same write is what rejects a wrong-width vector, which previously reached
turbovec as a shape it would report in its own terms.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Edwin Yu <edwinyyyu@gmail.com>
`prepare()` warms turbovec's per-index caches and its lazy id-to-slot map
so the first search, contains or remove after a write does not pay a
one-time cost. Against turbovec 0.8.0 that cost was the whole SIMD-blocked
code layout, rebuilt per add and proportional to the index rather than the
batch -- 199 ms at 100k vectors, which the first unlucky search paid if
this call did not. Upstream made that re-convergence incremental in 1.0.0,
and the call now costs 12-20 us and buys nothing measurable: at 100k a
search after a one-vector add reads 0.52 ms with it and 0.51 ms without,
and in a cold process the first search reads 0.10 ms against 0.09 ms.

The remaining warm it offers -- the id-to-slot map after a load, worth
0.5 ms off the first remove at 100k -- is not on this path, and the map
materializes safely under concurrent readers regardless.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Edwin Yu <edwinyyyu@gmail.com>
@edwinyyyu
edwinyyyu force-pushed the turbovec_engine_atomic branch from 2239abd to d38b296 Compare August 25, 2026 23:52
edwinyyyu added a commit to edwinyyyu/MemMachine that referenced this pull request Aug 26, 2026
…rototype

1.0.0 is upstream's first stable release, and what it commits to is the
on-disk format: v7 is the only container turbovec reads or writes, and a
file written by this release stays readable by later ones. The floor goes
there rather than at 0.7.0.

v7 is also what makes `sync` possible, and that changes how this engine
saves. `save` no longer routes through `atomic_index_write`. turbovec
publishes the index itself: a checkpoint appends what changed since the
last one rather than restating the whole index, and commits it durably --
a crash at any byte leaves the previous commit intact, and unlike the
helper's rename the publication survives a power failure. Wrapping that
would restate the index on every checkpoint and publish it through the
weaker of the two protocols. `load` drops `clear_stale_index_temp` with
it, since the engine no longer writes the `<path>.tmp` sibling it cleared.
Measured on the deployed store: a checkpoint carrying 512 new 768-d
vectors grew a 42MB container by 202,752 bytes -- the codes plus a commit
header, where the old path rewrote all 42MB. This follows MemMachine
PR MemMachine#1499, and the two persistence tests come from it.

The deployed index was a v3 container, which 1.0.0 refuses rather than
mis-decodes: v3 predates the v5 rotation change that altered every encoded
byte, so no converter can read it, and nothing here holds the float32
vectors -- the records table has no vector column and turbovec keeps only
codes. `claude_memory.migrate_index_v7` rebuilds it from the one source
that survives, the segment text, re-deriving each record's embedding anchor
through the ingest path's own deriver and embedder. It preserves row ids,
so vector.db, segment.db, state/ and demotions.json keep resolving, and it
reapplies the demotion deltas, which a naive rebuild would silently undo.
It is resumable, and every checkpoint is an incremental sync: on the live
store the first pass embedded 104,161 records in 39 minutes and the
catch-up pass committed the 71 that arrived behind it in seconds.

`DiskIndex` and `FreshIndex` are prototype work that lives only in the
local fork wheel; PyPI has never shipped them, and depending on the
released package means giving them up. So the `turbovecdisk` backend, the
two engines behind it, their tests, the `_index_search` hook they
subclassed, and the directory-shaped branch of `_remove_index_artifact`
(which existed for FreshIndex alone) all go.

The publication contract is documented where it lives, now that it is
engine-dependent: turbovec commits durably, the engines that publish by
rename do not, and `SQLiteVectorStore` trims its pending log behind either.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant