You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit f97c83b
Browse filesBrowse the repository at this point in the historyBrowse files
Merged PR 2194508: Fix bson_aggregation_pipeline_tests vector test broken by pgvector 0.8.5
## Why
The `numLists > data size` IVFFLAT block in `bson_aggregation_pipeline_tests` hardcoded a **pgvector build-memory pre-flight estimate** (`ERROR: memory required is 394 MB, maintenance_work_mem is 64 MB`) in its expected output.
That number is a pgvector implementation detail, not documentdb behavior. PGDG shipped **pgvector 0.8.5** (commit `b383e4d1`), which changed IVFFLAT sampling to reduce build memory for small tables. The estimate dropped **394 MB -> 383 MB**, and the test started failing at `make check` in the Debian packaging path:
```
expected: memory required is 394 MB
actual: memory required is 383 MB
```
Crucially, the two CI paths pin **different** pgvector versions:
- regress source build -> pgvector **0.8.1** (394 MB)
- deb / PGDG -> pgvector **0.8.5** (383 MB)
So hardcoding any single MB number cannot satisfy both paths, and pinning a pgvector version is brittle. The fix has to be **version independent**.
The block also used `numLists: 10000` against only **two** vector rows. That always exceeded `maintenance_work_mem` and asserted the memory-rejection *failure* path, so the test never actually exercised the "successful randomized-centroid search" path its own comment describes.
## How
Change `numLists` **10000 -> 3** on one SQL line. `3` is the smallest value still greater than the two-row data size, so it preserves the block's stated intent (`numLists > data size`) while letting the index build cheaply. With the index building, the volatile `memory required is N MB` assertion disappears entirely and both searches return real, deterministic results.
- **Only one SQL line changes.** The expected `.out` is regenerated because the now-building index turns the former error lines into real result rows and EXPLAIN plans.
- `nProbes` values are **intentionally unchanged**: query1 `nProbes:10000` auto-clamps to `numLists`, and query2 `nProbes:1` is deterministic because its query vector exactly equals a stored vector (score `1.0`). Both were proven run-to-run stable and byte-identical across pgvector versions.
## Validation
- Full `make check` passes on pgvector **0.8.1** and **0.8.5** (`# All 100 tests passed.`).
- Cross-version expected output is **byte-identical** (0 diff), verified with 10x determinism loops per version.
- documentdb OSS fork CI green across **PG15-18** on **arm64 and amd64** (sql regress + functional deb `make check`).
## Scope / sync note
This edits the OSS test under `oss/pg_documentdb/...`, which syncs to the public `documentdb/documentdb` repo. No separate public PR is needed. The distributed variant (`oss/internal/pg_documentdb_distributed/...`) has no IVFFLAT `numLists` block and is unaffected.
Signed-off-by: Guanzhou Song <guanzhousong@microsoft.com>
HINT: Drop the index until the table has more data.
943
-
ERROR: memory required is 394 MB, maintenance_work_mem is 64 MB
944
-
CONTEXT: SQL statement "CREATE INDEX documents_rum_index_3502 ON documentdb_data.documents_3500 USING ivfflat(CAST(documentdb_api_internal.bson_extract_vector(document, 'v'::text) AS public.vector(3)) public.vector_cosine_ops) WITH (lists = 10000) WHERE documentdb_api_internal.bson_extract_vector(document, 'v'::text) IS NOT NULL"
0 commit comments