{{ message }}
Add VectorSearchEngine backed by turbovec - #1499
Open
edwinyyyu wants to merge 10 commits into
Open
Conversation
26 tasks
SQLiteVectorStore persists each collection's index by calling the search engine's save(), which wrote directly to the final path. A crash mid-write left a truncated/corrupt file. Because index_saved=True makes the on-disk index a durable contract (missing/corrupt is a hard IndexLoadError, not a silent empty rebuild), an interrupted save could render a collection unrecoverable. Write the index to a sibling temp file and swap it into place with os.replace (atomic on POSIX and Windows on the same filesystem), so a reader sees either the old or new index, never a partial write; a failed save leaves the previous index intact. Leftover temp files are cleared on load so a crash does not leak them across restarts. Implemented in the engines (shared index_persistence helper) rather than in SQLiteVectorStore/SQLiteVectorStoreCollection, since the index save location and number of files written differ across engine implementations. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The swap protects a reader from a torn index, but the vector store also trims its pending-operation log once `save` returns -- and that log is the only other copy of those vectors, since the records table stores no vector column. So the swap reaching disk is load-bearing rather than a bonus: - fsync the parent directory after the replace, since POSIX `rename(2)` leaves the new directory entry in the page cache. Best-effort and ignored on failure, matching SQLite's `unixSync`; a no-op on Windows, which has no equivalent operation. - stop swallowing a failed fsync of the temp file. SQLite draws the same line -- a file fsync failure raises SQLITE_IOERR_FSYNC while a directory fsync failure is ignored -- and `EIO` means the writeback already failed and the dirty pages were dropped, which is exactly when the save must not be reported as committed. The existing cleanup then leaves the previous index in place with the log untrimmed, so the next save retries. - use F_FULLFSYNC on macOS, where plain `fsync` leaves the data in the drive's volatile write cache, falling back when a filesystem refuses it. State the resulting obligation on `VectorSearchEngine.save` itself, since that is what the store now relies on: replace atomically, then make the replacement as durable as the platform allows. An engine whose backend already implements the whole protocol can delegate to it and skip these helpers; the rest use `atomic_index_write`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The pending log holds the only durable copy of a vector between checkpoints -- the records table has no vector column -- so trimming it is safe only at an instant when the index provably holds those vectors. The temp-write + rename protocol this PR shipped could not provide that instant. A rename changes a directory entry, and Windows exposes no way to flush one: os.fsync is _commit, which is FlushFileBuffers, which is for file data, and you cannot open a directory to fsync it. The decisive evidence is SQLite's own -- it threads a directory-sync flag through every commit-relevant directory operation, honors it in unixDelete, and declares it /* Not used on win32 */ in winDelete. So os.replace could return, _save_collection_index could commit its trim durably behind it, and a power cut could still roll the rename back: records forward, index back, no copy of the difference left. MOVEFILE_WRITE_THROUGH is not a fix; its documented guarantee covers copy-and-delete (cross-volume) moves, not same-volume renames. Take SQLite's answer, which was not to harden the directory operation but to stop using one as a commit point (PERSIST commits by zeroing a header, TRUNCATE by truncating, WAL by appending frames). A base path now expands into two index slots plus a generation record each, created once and thereafter only overwritten. A checkpoint writes the index over the inactive slot and flushes it, then writes that slot's generation record and flushes that. The record is the commit, and it is a write into a file that already exists. It holds the generation and its bitwise complement, so a torn write reads as absent rather than as some other generation -- all or nothing without needing single-sector atomicity from the hardware. load takes the highest believable generation, and deliberately does not fall back to the older slot when the published index will not parse: the log was trimmed against the newer one, so the older is stale by exactly the ops that can no longer be replayed. Both backends already write straight to the path they are given, which is what this protocol wants -- verified that repeated saves preserve the inode and leave no stray files -- so no engine gains a temp file, a buffer, or a rename. Durability is entirely the engine's, including which artifact is live. The store keeps no slot pointer, manifest, or generation, so no schema change and no migration: what remains is one rule, never trim past what save says is durable, and _save_collection_index already had that order. index_path becomes index_base_path since it no longer names a file, and discarding a collection asks the engine layer which files that covers. BREAKING CHANGE: an index written by the previous protocol is not published under the new one, so a collection with index_saved=True raises IndexLoadError until its index directory is cleared and the records re-ingested. Anomaly tests walk every crash point in the publish sequence by constructing the on-disk state each would leave, plus one that pins the ordering itself (a failed index write must publish nothing) since state-based tests cannot observe it. Verified against three deliberate breaks -- dropping the complement check, writing the record first, and reusing one slot instead of alternating -- each caught. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The two-slot generation-record protocol bought a guarantee we have decided not to make: that a save survives a power failure. Every engine would have to implement and maintain that protocol, and the failure it buys out is bounded -- search recall for the records applied since the last checkpoint, repaired by re-ingesting them. The direction that actually costs, a published index that will not parse, is closed by the atomic swap on its own. So this returns to the temp-file-plus-rename publication and spends the difference on stating the contract instead of strengthening it: `save` publishes atomically, never durably; the store trims the pending log behind a publication a power failure can revert; a record whose vector is lost that way still resolves by uuid, is absent from search until it is upserted again, and nothing here detects the gap for the caller. Reverts the durability and engine-owned-publication commits, keeps the atomic swap, and adds a store-level test that reconstructs a reverted publication deterministically -- restore the previous index bytes after the trim -- to pin the direction it fails in. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`update_feature` reads the stored embedding back when a caller updates a feature without supplying one, and that is the only place in the server that depends on the index still holding a vector. With publication now atomic rather than durable, a power failure can leave a feature whose row is intact and whose vector is not -- a state this path reported as "Vector record not found", which points the caller at the wrong thing and hides the repair. Split the two cases. A record that is genuinely absent keeps the old message; a record whose embedding the index no longer holds says so and names the fix, which is to pass a fresh embedding. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
edwinyyyu
force-pushed
the
turbovec_engine_atomic
branch
from
August 25, 2026 23:13
04a827c to
2239abd
Compare
turbovec keeps TurboQuant-compressed vectors in RAM, so an index is a fraction of the size of the f32 engines' and a full scan has a much smaller scaling constant than sqlite-vec's on-disk one. Search is approximate as a result: scores land near the exact value rather than on it, and the tests assert ranking and membership instead of magnitudes. Two consequences of storing only compressed vectors are worth naming. `get_vectors` raises `NotImplementedError`, since the originals are not recoverable -- which makes this engine usable by EventMemory but not by semantic memory, whose feature updates read stored embeddings back. And removal is exact and cheap: turbovec drops the id from its map rather than tombstoning, so a deleted key cannot resurface in results. `save` publishes through the shared atomic-write helper, like the other engines, so an interrupted save leaves the previously published index intact rather than a truncated file the store would treat as a hard load error. `load` clears any temp file a previous save left behind. Signed-off-by: Edwin Yu <edwinyyyu@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1.0.0 is upstream's first stable release, and what it commits to is the on-disk format: v7 is the only container turbovec reads or writes, and a file written by this release stays readable by later ones. The extra floors there rather than at 0.7.0. v7 is also what makes `sync` possible, and that changes how this engine saves. `save` no longer routes through the shared `atomic_index_write` helper. turbovec publishes the index itself: a checkpoint appends what changed since the last one rather than restating the whole index, and commits it durably -- a crash at any byte leaves the previous commit intact, and unlike the helper's rename the publication survives a power failure. Wrapping that would restate the index on every checkpoint and publish it through the weaker of the two protocols. `load` drops `clear_stale_index_temp` with it, since the engine no longer writes the `<path>.tmp` sibling it cleared. The tests follow. Save leaving no temp file is now an assertion about the whole directory, since turbovec names its own temp; the stale-temp test goes with the protocol it tested; and a new test pins what a later checkpoint carries, reading a bulk removal and an append back through a fresh engine. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Edwin Yu <edwinyyyu@gmail.com>
`SearchMatch` documents a cosine score as a cosine similarity in [-1, 1], and `SQLiteVectorStoreCollection` compares `score_threshold` against it directly. turbovec's score is a quantized inner product, so it lands a hair outside that range: at 8 dimensions and the default bit width, 40 of 50 self-matches score above 1.0 (max 1.0066), and at bit width 2 they exceed it at every dimension measured (1.0038 at 768, 1.0025 at 1536). Publishing 1.0066 as a cosine similarity is a broken promise, however small, so the hair is clamped where the match is built. A dot product is unbounded by contract and passes through untouched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Edwin Yu <edwinyyyu@gmail.com>
turbovec indexes a dimensionality that is a positive multiple of 8, so the engine refused a width every other engine in the tree accepts -- 300, the width of a spaCy or word2vec vector, among them -- with turbovec's own error rather than a store-level one. The width is now rounded up and vectors are written into the leading columns of a zeroed buffer. Padding is exact for both metrics this engine serves: a zero coordinate adds nothing to an inner product and nothing to an L2 norm, so the padded index answers as the unpadded one would. The same write is what rejects a wrong-width vector, which previously reached turbovec as a shape it would report in its own terms. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Edwin Yu <edwinyyyu@gmail.com>
`prepare()` warms turbovec's per-index caches and its lazy id-to-slot map so the first search, contains or remove after a write does not pay a one-time cost. Against turbovec 0.8.0 that cost was the whole SIMD-blocked code layout, rebuilt per add and proportional to the index rather than the batch -- 199 ms at 100k vectors, which the first unlucky search paid if this call did not. Upstream made that re-convergence incremental in 1.0.0, and the call now costs 12-20 us and buys nothing measurable: at 100k a search after a one-vector add reads 0.52 ms with it and 0.51 ms without, and in a cold process the first search reads 0.10 ms against 0.09 ms. The remaining warm it offers -- the id-to-slot map after a load, worth 0.5 ms off the first remove at 100k -- is not on this path, and the map materializes safely under concurrent readers regardless. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Edwin Yu <edwinyyyu@gmail.com>
edwinyyyu
force-pushed
the
turbovec_engine_atomic
branch
from
August 25, 2026 23:52
2239abd to
d38b296
Compare
edwinyyyu
added a commit
to edwinyyyu/MemMachine
that referenced
this pull request
Aug 26, 2026
…rototype 1.0.0 is upstream's first stable release, and what it commits to is the on-disk format: v7 is the only container turbovec reads or writes, and a file written by this release stays readable by later ones. The floor goes there rather than at 0.7.0. v7 is also what makes `sync` possible, and that changes how this engine saves. `save` no longer routes through `atomic_index_write`. turbovec publishes the index itself: a checkpoint appends what changed since the last one rather than restating the whole index, and commits it durably -- a crash at any byte leaves the previous commit intact, and unlike the helper's rename the publication survives a power failure. Wrapping that would restate the index on every checkpoint and publish it through the weaker of the two protocols. `load` drops `clear_stale_index_temp` with it, since the engine no longer writes the `<path>.tmp` sibling it cleared. Measured on the deployed store: a checkpoint carrying 512 new 768-d vectors grew a 42MB container by 202,752 bytes -- the codes plus a commit header, where the old path rewrote all 42MB. This follows MemMachine PR MemMachine#1499, and the two persistence tests come from it. The deployed index was a v3 container, which 1.0.0 refuses rather than mis-decodes: v3 predates the v5 rotation change that altered every encoded byte, so no converter can read it, and nothing here holds the float32 vectors -- the records table has no vector column and turbovec keeps only codes. `claude_memory.migrate_index_v7` rebuilds it from the one source that survives, the segment text, re-deriving each record's embedding anchor through the ingest path's own deriver and embedder. It preserves row ids, so vector.db, segment.db, state/ and demotions.json keep resolving, and it reapplies the demotion deltas, which a naive rebuild would silently undo. It is resumable, and every checkpoint is an incremental sync: on the live store the first pass embedded 104,161 records in 39 minutes and the catch-up pass committed the 71 that arrived behind it in seconds. `DiskIndex` and `FreshIndex` are prototype work that lives only in the local fork wheel; PyPI has never shipped them, and depending on the released package means giving them up. So the `turbovecdisk` backend, the two engines behind it, their tests, the `_index_search` hook they subclassed, and the directory-shaped branch of `_remove_index_artifact` (which existed for FreshIndex alone) all go. The publication contract is documented where it lives, now that it is engine-dependent: turbovec commits durably, the engines that publish by rename do not, and `SQLiteVectorStore` trims its pending log behind either. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Purpose of the change
SQLiteVectorStorebacked by turbovec is smaller and faster thanSQLiteVecVectorStorebacked by sqlite-vec. Both are linear full scans, butturbovec keeps TurboQuant-compressed vectors in RAM, so its scaling constant is
much smaller and an index is a fraction of the size of the f32 engines'.
Description
Adds
TurboVecVectorSearchEngine, aVectorSearchEnginebacked by turbovec,behind a new
turbovecoptional extra floored at 1.0.0. turbovec indexes adimensionality that is a multiple of 8, so any other width is zero-padded up to
one -- exact for both metrics here, since a zero coordinate adds nothing to an
inner product or to an L2 norm -- and every width the other engines accept
works.
Publication is turbovec's own. 1.0.0 is upstream's first stable release, and
what it commits to is the on-disk format: v7 is the only container turbovec
reads or writes, and a file written by 1.0.0 stays readable by later releases.
v7 is also what makes
syncpossible, andsavecalls it -- a checkpointappends what changed since the last one rather than restating the whole index,
and commits it durably, so a crash at any byte leaves the previous commit
intact and the publication survives a power failure. The engine therefore does
not route through the shared
atomic_index_writehelper that the hnswlib andusearch engines use on #1460: wrapping
syncwould restate the index on everycheckpoint and publish it through the weaker of the two protocols, a rename the
helper's own docstring declines to make durable.
That is what supersedes #1448, which added the same engine against turbovec
0.x, whose
writehad no publication protocol of its own and wrote straight tothe final path -- an interrupted save left a truncated file where the store
expects a loadable index, and because
index_savedmakes a published index adurable contract, that is a hard
IndexLoadErrorrather than a silent rebuild.Known limitations, both consequences of storing only compressed vectors:
get_vectorsraisesNotImplementedError, since the original vectors are notrecoverable. That makes the engine usable by EventMemory but not by semantic
memory, whose feature updates read stored embeddings back.
it, so the tests assert ranking and membership rather than magnitudes. The
quantized inner product also overshoots the cosine range -- 40 of 50
self-matches at 8 dimensions, and every dimension measured at bit width 2 --
so
SearchMatchscores are clamped to[-1, 1]under a cosine metric, whichis what its docstring promises and what
score_thresholdcompares against.A dot product is unbounded by contract and is not clamped.
Removal is a strong point by comparison: turbovec drops the id from its map
rather than tombstoning it, so deletions stay cheap and a removed key cannot
resurface in results.
Type of change
How Has This Been Tested?
test_turbovec_engine.pycovers construction, add, remove, cosine and dotsearch, filtered search,
get_vectorsraising, score range (clamped undercosine, untouched under dot), unaligned widths (search, save/load, and a
wrong-width vector refused), and persistence -- a save/load
round-trip, a load replacing a live index, a save leaving nothing but the index
behind, and what a later checkpoint carries, reading a bulk removal and an
append back through a fresh engine. The module is
importorskip-guarded, so thesuite still runs without the extra installed.
Test Results:
uv run pytest packages/server/server_tests/memmachine_server/common/vector_store-> 346 passed.
ruff checkandruff format --checkclean. Thetyjobs arered on
mainitself (nebulagraph_python.clientlost the members the graphstore imports), fixed by #1519 rather than here; nothing
tyreports is inthis diff.
Checklist
Maintainer Checklist