fix(query): drop memory-heavy state when processors finish by dqhl76 · Pull Request #20716 · databendlabs/databend · GitHub
Skip to content

fix(query): drop memory-heavy state when processors finish - #20716

Open
dqhl76 wants to merge 8 commits into
databendlabs:mainfrom
dqhl76:fix-range-join-release-state
Open

dqhl76 wants to merge 8 commits into
databendlabs:mainfrom
dqhl76:fix-range-join-release-state

Conversation

@dqhl76

@dqhl76 dqhl76 commented Oct 16, 2026 •

Copy link
Copy Markdown
Member

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

The pipeline executor keeps every processor in the graph until the query's RunningGraph is dropped; returning Event::Finished only marks the node as finished. So any state an operator does not release itself when it finishes stays allocated for the rest of the query, while the operators above it keep running. Several operators keep state that grows with their input this way. This PR releases it when the operator finishes, one commit per operator:

Commit Operator What was kept until query end
range join TransformRangeJoinLeft/Right Both fully buffered inputs and sort keys (RangeJoinState). The transforms now hold Option<Arc<RangeJoinState>> and take it on finish, like the new hash join's FinishedJoin swap.
single-state final aggregate FinalSingleStateAggregator The serialized partial states after merging; for median, array_agg, uniq and similar that is the whole input.
legacy hash join TransformHashJoinProbe / HashJoinBuildState With a final scan (full, mark, single), wait_probe_counter is already zero before the probes finish, so the last probe never saw 1 in finish_build and the build side plus hash table were never released. Unfinished probes are now counted separately. The hash table entry storage lived on the build state and was never cleared (growing across spill rounds); it now sits next to the table and is cleared whenever the table is replaced or released.
REPLACE INTO ReplaceIntoProcessor The set of seen key digests (one per input row), through the segment rewrite and commit that follow.
window TransformWindow The last partition's blocks; without PARTITION BY that is the whole input.
materialized CTE MaterializedCteSink Closing the channel keeps queued payloads alive while any sender exists, so a consumer that stopped early held its unread payloads. Senders are now dropped.
spatial join TransformSpatialJoinProbe/Build The shared state with the concatenated build side and packed R-tree. Taken on finish like range join.
aggregate spill SharedPartitionStream After a flush each bucket reserves its last flushed size again, and finishing only finalized buckets that still held rows, so the reserved capacity (bounded by max_block_bytes per bucket) stayed until the query ended.

Bounded per-processor buffers (filter selection vectors, hash table initial capacity, file writer buffers, exchange partition builders) are left as they are.

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

AI assistance

  • AI usage: An AI coding assistant traced operator state lifetimes in the executor, surveyed which operators keep state after finishing with a temporary debug probe, wrote the changes, and ran the local memory measurements.
  • Responsible human: @dqhl76
  • The responsible human has read every line of this diff and can explain each change

This change is Reviewable

TransformRangeJoinLeft/Right kept Arc<RangeJoinState> until the executor
graph was dropped at query end, so every range/IE join held its fully
buffered inputs and sort keys after producing all output. Take the state
when each processor finishes, like the new hash join does, so the tables
are freed once the last range join processor is done.
@github-actions github-actions Bot added the pr-bugfix this PR patches a bug in codebase label Oct 16, 2026
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-16T10:50:19.563929Z 166e52b Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e0807cd65e

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

⏱ CI timing · Dev

run 38891661211 · attempt 2 (wall time includes re-run gap) · 82.4 min (main p50 39.9 min, +42.5 min ⚠️)

stage this run main p50 Δ
total 82.4 min 39.9 min +42.5 min ⚠️
build (aarch64, ci profile) 26.1 min 25.4 min +0.6 min
├ deps + crates (parallel) 13.1 min — —
├ databend-query codegen 3.3 min — —
├ databend-binaries link 7.8 min — —
└ sccache hit rate 97.9 % — —
test_unit 14.8 min 17.3 min −2.5 min
mac_check — 14.1 min —
tests after build (49 jobs) 11.8 min 13.2 min −1.4 min
├ queue p50 / max 2.7 min / 5.7 min 1.5 min / 7.7 min +1.2 min
├ sqllogic standalone (8 jobs, max) 4.3 min 4.0 min +0.2 min
├ sqllogic cluster (8 jobs, max) 7.3 min 6.9 min +0.4 min
├ sqllogic stage (2 jobs, max) 4.6 min 4.8 min −0.3 min
├ sqllogic other (10 jobs, max) 6.5 min 5.6 min +0.9 min
├ stateless (2 jobs, max) 6.7 min 6.1 min +0.6 min
├ stateful (4 jobs, max) 4.6 min 4.6 min −0.1 min
├ compat (6 jobs, max) 4.7 min 4.8 min −0.1 min
├ meta / logs / lineage / udf (5 jobs, max) 4.4 min 4.4 min −0.0 min
├ ee (2 jobs, max) 1.1 min 1.1 min +0.0 min
├ private_tasks (1 jobs, max) 8.8 min 8.8 min −0.0 min
└ fuzz (1 jobs, max) 1.1 min 1.0 min +0.1 min

Minutes unless stated; durations exclude queue time unless stated; "max" = longest job in the group. Baseline = median of the last 10 successful runs (Dev: Main on main; Cloud: recent Cloud runs). Build breakdown has no baseline. Updated on each push.

@dqhl76
dqhl76 marked this pull request as draft October 16, 2026 07:42
@dqhl76
dqhl76 force-pushed the fix-range-join-release-state branch from c38a40f to e0807cd Compare October 16, 2026 08:46
…gate

FinalSingleStateAggregator kept the serialized partial states after merging
them. For aggregates whose state holds every value (median, array_agg, uniq)
that is the whole input, kept until the query ends.
Joins with a final scan count wait_probe_counter down to zero before the
probes finish, so the last probe never saw 1 in finish_build and the build
side and hash table were kept until the query ended. Count unfinished probes
separately. The hash table entry storage also lived on the build state and was
never cleared, growing across spill rounds; keep it next to the table and
clear it whenever the table is replaced or released.
The set of seen key digests grows with the input and was kept after the
processor finished, through the segment rewrite and commit that follow.
The last partition's blocks are only released when a new partition starts,
so they were kept until the query ended. Without PARTITION BY that is the
whole input.
Closing the channel keeps queued payloads alive while any sender exists, so a
consumer that stopped early held its unread payloads until the query ended.
Also drop buffered payloads once every consumer is gone.
The probe and build processors kept the shared state, with the concatenated
build side and the packed R-tree, until the query ended. Take it when each
processor finishes so it is freed when the last one is done.
…inish

The partition stream shared by partial aggregates reserves each bucket's last
flushed size again after a flush, and finishing only finalized buckets that
still held rows, so the reserved capacity stayed until the query ended. Drop
every bucket builder once the last partial finishes.
@dqhl76 dqhl76 changed the title fix(query): release range join state when processors finish fix(query): release operator state when processors finish Oct 16, 2026
@dqhl76 dqhl76 changed the title fix(query): release operator state when processors finish fix(query): drop memory-heavy operator state when processors finish Oct 16, 2026
@dqhl76 dqhl76 changed the title fix(query): drop memory-heavy operator state when processors finish fix(query): drop memory-heavy state when processors finish Oct 16, 2026
@dqhl76
dqhl76 marked this pull request as ready for review October 16, 2026 10:12
@dqhl76

dqhl76 commented Oct 16, 2026

Copy link
Copy Markdown
Member Author

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 166e52bdc4

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@dqhl76
dqhl76 force-pushed the fix-range-join-release-state branch from 369a9a3 to 166e52b Compare October 16, 2026 11:07
@dqhl76
dqhl76 requested a review from zhang2014 October 18, 2026 01:33

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-bugfix this PR patches a bug in codebase

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant