perf(core): collect multiword FTS operand matches through GIN by phernandez · Pull Request #1802 · basicmachines-co/basic-memory · GitHub
Skip to content

perf(core): collect multiword FTS operand matches through GIN - #1802

Merged
phernandez merged 3 commits into
mainfrom
perf/postgres-search-large-projects
Oct 21, 2026
Merged

phernandez merged 3 commits into
mainfrom
perf/postgres-search-large-projects

Conversation

@phernandez

Copy link
Copy Markdown
Member

Why

Part of #1791. Multiword Postgres full-text searches repeatedly fetch and decompress chunk vectors to determine which query words each document contains. This is expensive on large projects even after GIN candidate selection.

What Changed

Collect each operand's parent/chunk matches through GIN, group by project and document identity, and project a document-level operand vector once behind the existing candidate fence. Preserve Boolean semantics, cross-chunk matches, scores, and exact counts. Add semantic and query-plan regression tests plus a reproducible local benchmark.

Implementation Details

The grouped operand matches also provide candidate keys, avoiding a second candidate scan. Pure/optional negation retains the scoped document universe. Single-word and mixed-script queries retain their existing path.

On a generated 32,000-note / 320,000-chunk fixture, compared with main 976a10c89, second-run inclusive buffer touches fell:

Query Page Exact count
meeting budget review 225,890 → 94,910 (58%) 70,162 → 3,999 (94%)
missingword budget 114,076 → 2,081 (98%) 114,076 → 1,302 (99%)

These are local SQL measurements, not production latency guarantees. See docs/postgres-fts-large-project-benchmark.md for methodology and reproduction.

Testing

  • BASIC_MEMORY_TEST_POSTGRES=1 LOGFIRE_IGNORE_NO_CONFIG=1 uv run pytest -q tests/repository/test_postgres_search_repository.py --no-cov — 40 passed.
  • BASIC_MEMORY_TEST_POSTGRES=1 LOGFIRE_IGNORE_NO_CONFIG=1 uv run pytest -q tests/repository/test_postgres_operand_search.py tests/repository/test_search_repository.py tests/repository/test_script_ngrams.py tests/services/test_search_service.py tests/api/v2/test_scoped_search_router.py --no-cov — 211 passed, 19 skipped.
  • just fast-check, just doctor, and uv run ty check scripts/benchmark_postgres_fts.py — passed.
  • The query-plan regression fails against the baseline compiler and passes with this change.
  • Full result identities and score floats matched baseline for six queries on the large fixture.
  • Benchmark seed/measure smoke passed with 20 notes and 200 chunks.

Risks / Follow-ups

Broad single-word ranking remains expensive: archive matched most fixture documents and took 30–40 seconds with roughly 3.1 million buffer touches. This PR does not close #1791. Further work should reduce chunk ranking reads without truncating candidates or changing scores. Production/Neon cold-cache and end-to-end API latency remain unverified.

Signed-off-by: phernandez <paul@basicmachines.co>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-21T15:42:40.002669Z 048c50e Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 85c57c36d2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/basic_memory/repository/postgres_search_query.py
Signed-off-by: phernandez <paul@basicmachines.co>
@phernandez

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. More of your lovely PRs please.

Reviewed commit: f7d876dd06

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Signed-off-by: phernandez <paul@basicmachines.co>
@phernandez

Copy link
Copy Markdown
Member Author

Fixed the CI plan-test failure in 048c50e. The no-correlated-chunk-probe assertion passed in CI; only the exact Bitmap Heap Scan assertion failed on the tiny fixture. The test now exercises bitmap scans enabled and disabled, allowing both valid index scan types while retaining the at-most-one-loop invariant. All 12 operand tests and fast-check/typecheck pass. @codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

@phernandez
phernandez merged commit 8df7e90 into main Oct 21, 2026
23 checks passed
@phernandez
phernandez deleted the perf/postgres-search-large-projects branch October 21, 2026 16:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Postgres search reads whole tables per query on large projects (admission count, FTS predicate placement, count)

1 participant