Repository navigation
perf(core): collect multiword FTS operand matches through GIN - #1802
Conversation
Signed-off-by: phernandez <paul@basicmachines.co>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 85c57c36d2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: phernandez <paul@basicmachines.co>
|
@codex review |
|
Codex Review: Didn't find any major issues. More of your lovely PRs please. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Signed-off-by: phernandez <paul@basicmachines.co>
|
Fixed the CI plan-test failure in 048c50e. The no-correlated-chunk-probe assertion passed in CI; only the exact Bitmap Heap Scan assertion failed on the tiny fixture. The test now exercises bitmap scans enabled and disabled, allowing both valid index scan types while retaining the at-most-one-loop invariant. All 12 operand tests and fast-check/typecheck pass. @codex review |

Why
Part of #1791. Multiword Postgres full-text searches repeatedly fetch and decompress chunk vectors to determine which query words each document contains. This is expensive on large projects even after GIN candidate selection.
What Changed
Collect each operand's parent/chunk matches through GIN, group by project and document identity, and project a document-level operand vector once behind the existing candidate fence. Preserve Boolean semantics, cross-chunk matches, scores, and exact counts. Add semantic and query-plan regression tests plus a reproducible local benchmark.
Implementation Details
The grouped operand matches also provide candidate keys, avoiding a second candidate scan. Pure/optional negation retains the scoped document universe. Single-word and mixed-script queries retain their existing path.
On a generated 32,000-note / 320,000-chunk fixture, compared with main
976a10c89, second-run inclusive buffer touches fell:These are local SQL measurements, not production latency guarantees. See
docs/postgres-fts-large-project-benchmark.mdfor methodology and reproduction.Testing
BASIC_MEMORY_TEST_POSTGRES=1 LOGFIRE_IGNORE_NO_CONFIG=1 uv run pytest -q tests/repository/test_postgres_search_repository.py --no-cov— 40 passed.BASIC_MEMORY_TEST_POSTGRES=1 LOGFIRE_IGNORE_NO_CONFIG=1 uv run pytest -q tests/repository/test_postgres_operand_search.py tests/repository/test_search_repository.py tests/repository/test_script_ngrams.py tests/services/test_search_service.py tests/api/v2/test_scoped_search_router.py --no-cov— 211 passed, 19 skipped.just fast-check,just doctor, anduv run ty check scripts/benchmark_postgres_fts.py— passed.Risks / Follow-ups
Broad single-word ranking remains expensive:
archivematched most fixture documents and took 30–40 seconds with roughly 3.1 million buffer touches. This PR does not close #1791. Further work should reduce chunk ranking reads without truncating candidates or changing scores. Production/Neon cold-cache and end-to-end API latency remain unverified.