[MOD-20011] Keep Intersection and Union aggregates sound across revalidation by gdesmott · Pull Request #11940 · RediSearch/RediSearch · GitHub
Skip to content

[MOD-20011] Keep Intersection and Union aggregates sound across revalidation - #11940

Open
gdesmott wants to merge 2 commits into
masterfrom
gdesmott-intersection-revalidate-consensus
Open

gdesmott wants to merge 2 commits into
masterfrom
gdesmott-intersection-revalidate-consensus

Conversation

@gdesmott

@gdesmott gdesmott commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

Describe the changes in the pull request

  1. Current: Intersection, UnionFlat and UnionHeap can publish an aggregate after revalidate whose borrowed child entries were invalidated by the children's own revalidation: reading the result after an Ok revalidation is undefined behavior under Miri. The intersection can also publish a document that is not a consensus it yielded after a child error (children on different documents, a document that failed the slop / in-order check, or an error during the first read), and its re-seek could seek a child that stayed ahead backwards.
  2. Change: When every child still backs the published document, revalidation rebuilds the aggregate's entries in place and keeps its metrics. The unions reuse the rebuild their resume path already performs, so both paths behave identically. Otherwise the intersection re-seeks from the furthest child, as a read would, and reports Moved. A result swapped in through current() that cannot be rebuilt in place aborts. Children may now only be added to an intersection before iteration (debug-asserted; the query optimizer already adds them at plan time).
  3. Outcome: Internal hardening, with no user-visible change today. An intersection's or union's result is sound after an Ok revalidation, and after a child error the intersection continues from the next consensus instead of republishing a stale document or skipping one. None of these cases can currently be triggered from a test:
    • The stale entries are only read after the next Read has rebuilt them, except under a union, where the pointers still reach unchanged results. Miri flags this, but the data read is correct.
    • No production code swaps a result through current(), and the query optimizer only adds intersection children before iteration.
    • The child-error cases need an iterator to time out mid-consensus, inside a cursor kept alive by ON_TIMEOUT RETURN. That can only happen on a real deadline; no debug hook triggers it deterministically.

Which additional issues this PR fixes

N/A

Main objects this PR modified

  1. Intersection::revalidate (rqe_iterators): refreshes the aggregate when it is still backed, otherwise re-seeks from the furthest child.
  2. UnionFlat::revalidate / UnionHeap::revalidate: rebuild the aggregate's entries when no child changed, sharing resume's rebuild.

Mark if applicable

  • This PR introduces API changes
  • This PR introduces serialization changes

Release Notes

  • This PR requires release notes
  • This PR does not require release notes

If a release note is required (bug fix / new feature / enhancement), describe the user impact of this PR in the title.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 41b5ff9d2c

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/redisearch_rs/rqe_iterators/src/intersection.rs Outdated
Comment thread src/redisearch_rs/rqe_iterators/src/intersection.rs
@gdesmott
gdesmott force-pushed the gdesmott-intersection-revalidate-consensus branch from 41b5ff9 to c7d402c Compare October 20, 2026 09:00
@gdesmott gdesmott changed the title Abort Intersection revalidation on a result its children no longer back Keep Intersection's aggregate sound and backed by its children across revalidation Oct 20, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c7d402cf89

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/redisearch_rs/rqe_iterators/src/intersection.rs Outdated
Comment thread src/redisearch_rs/rqe_iterators/src/intersection.rs
@gdesmott
gdesmott force-pushed the gdesmott-intersection-revalidate-consensus branch from c7d402c to 1c3c89b Compare October 20, 2026 09:28

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1c3c89b89e

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/redisearch_rs/rqe_iterators/src/intersection.rs
@gdesmott
gdesmott force-pushed the gdesmott-intersection-revalidate-consensus branch 2 times, most recently from 9799796 to 22b1e8b Compare October 20, 2026 09:46

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 22b1e8bdd4

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/redisearch_rs/rqe_iterators/src/intersection.rs Outdated
@gdesmott
gdesmott force-pushed the gdesmott-intersection-revalidate-consensus branch from 22b1e8b to e966b42 Compare October 20, 2026 09:55

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e966b42ff8

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/redisearch_rs/rqe_iterators/src/intersection.rs
@codecov

codecov Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.87179% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.76%. Comparing base (debcd43) to head (81b161e).
⚠️ Report is 3 commits behind head on master.

Files with missing lines Patch % Lines
...rc/redisearch_rs/rqe_iterators/src/intersection.rs 92.45% 4 Missing ⚠️
src/redisearch_rs/rqe_iterators/src/union_flat.rs 95.23% 1 Missing and 1 partial ⚠️
src/redisearch_rs/rqe_iterators/src/union_heap.rs 96.36% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master   #11940      +/-   ##
==========================================
- Coverage   83.77%   83.76%   -0.02%     
==========================================
  Files         682      683       +1     
  Lines       88675    88774      +99     
  Branches    11471    11480       +9     
==========================================
+ Hits        74291    74361      +70     
- Misses      10740    10777      +37     
+ Partials     3644     3636       -8     
Flag Coverage Δ
flow 72.27% <70.51%> (+0.02%) ⬆️
unit 58.21% <94.87%> (-0.10%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@gdesmott
gdesmott force-pushed the gdesmott-intersection-revalidate-consensus branch from e966b42 to ab2a8ed Compare October 20, 2026 11:33
@gdesmott gdesmott changed the title Keep Intersection's aggregate sound and backed by its children across revalidation Keep Intersection and Union aggregates sound across revalidation Oct 20, 2026
@github-actions github-actions Bot added size:L and removed size:M labels Oct 20, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ab2a8ed1c1

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/redisearch_rs/rqe_iterators/src/union_flat.rs
Comment thread src/redisearch_rs/rqe_iterators/src/union_heap.rs
@gdesmott
gdesmott force-pushed the gdesmott-intersection-revalidate-consensus branch 2 times, most recently from ff76621 to 8216430 Compare October 20, 2026 12:22
@github-actions github-actions Bot added size:XL and removed size:L labels Oct 20, 2026
@github-actions

Copy link
Copy Markdown

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8216430c11

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/redisearch_rs/rqe_iterators/src/union.rs Outdated
@gdesmott
gdesmott force-pushed the gdesmott-intersection-revalidate-consensus branch from 8216430 to 9f61b4d Compare October 20, 2026 12:39
@github-actions github-actions Bot added size:L and removed size:XL labels Oct 20, 2026
@gdesmott
gdesmott requested a review from GuyAv46 October 20, 2026 13:34
@gdesmott gdesmott changed the title Keep Intersection and Union aggregates sound across revalidation [MOD-20011] Keep Intersection and Union aggregates sound across revalidation Oct 20, 2026
Revalidation could publish an aggregate that is not a consensus the
intersection yielded. A child error mid-consensus can leave the children
on different documents, or leave the result on a document that failed
the proximity check; an error during the first read can leave children
positioned with nothing yielded. And when no child moved, the aggregate's
child entries are stale either way: revalidating a child borrows it
mutably, which invalidates the entries the aggregate borrowed from it
(undefined behavior under Miri when the result is read afterwards).

When every child still sits on the document last yielded, rebuild the
aggregate's entries for it in place, keeping its metrics; a child may
report a move while keeping its position, so this does not depend on
whether one did. Otherwise re-seek from the furthest child, as a read would;
re-seeking from the furthest moved child alone could seek a child that
stayed ahead backwards. A result swapped in through current() that
cannot be rebuilt in place aborts.

Children may only be pushed before iteration: a child added later sits
behind the documents already yielded, which no re-seek can reconcile.
…hing

When no child changed, revalidate published the aggregate as it stood,
but revalidating each child borrows it mutably, which invalidates the
entries the aggregate borrowed from it: reading the result after an Ok
revalidation is undefined behavior under Miri.

Rebuild the entries on that exit with the same helper resume uses on
its own no-change exit, so the two paths offer children in the same
order and keep the same quick-mode backer, metrics and doc ID. As in
resume, an unbacked rebuild aborts, and a result swapped in through
current() that is not a borrowed union aggregate aborts rather than
being rebuilt or settled in place.
@gdesmott
gdesmott force-pushed the gdesmott-intersection-revalidate-consensus branch from 9f61b4d to 81b161e Compare October 20, 2026 15:11
@sonarqubecloud

Copy link
Copy Markdown

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant