Repository navigation
Conversation
TransformRangeJoinLeft/Right kept Arc<RangeJoinState> until the executor graph was dropped at query end, so every range/IE join held its fully buffered inputs and sort keys after producing all output. Take the state when each processor finishes, like the new hash join does, so the tables are freed once the last range join processor is done.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e0807cd65e
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
⏱ CI timing · Devrun 38891661211 · attempt 2 (wall time includes re-run gap) · 82.4 min (main p50 39.9 min, +42.5 min
|
| stage | this run | main p50 | Δ |
|---|---|---|---|
| total | 82.4 min | 39.9 min | +42.5 min |
| build (aarch64, ci profile) | 26.1 min | 25.4 min | +0.6 min |
| ├ deps + crates (parallel) | 13.1 min | — | — |
| ├ databend-query codegen | 3.3 min | — | — |
| ├ databend-binaries link | 7.8 min | — | — |
| └ sccache hit rate | 97.9 % | — | — |
| test_unit | 14.8 min | 17.3 min | −2.5 min |
| mac_check | — | 14.1 min | — |
| tests after build (49 jobs) | 11.8 min | 13.2 min | −1.4 min |
| ├ queue p50 / max | 2.7 min / 5.7 min | 1.5 min / 7.7 min | +1.2 min |
| ├ sqllogic standalone (8 jobs, max) | 4.3 min | 4.0 min | +0.2 min |
| ├ sqllogic cluster (8 jobs, max) | 7.3 min | 6.9 min | +0.4 min |
| ├ sqllogic stage (2 jobs, max) | 4.6 min | 4.8 min | −0.3 min |
| ├ sqllogic other (10 jobs, max) | 6.5 min | 5.6 min | +0.9 min |
| ├ stateless (2 jobs, max) | 6.7 min | 6.1 min | +0.6 min |
| ├ stateful (4 jobs, max) | 4.6 min | 4.6 min | −0.1 min |
| ├ compat (6 jobs, max) | 4.7 min | 4.8 min | −0.1 min |
| ├ meta / logs / lineage / udf (5 jobs, max) | 4.4 min | 4.4 min | −0.0 min |
| ├ ee (2 jobs, max) | 1.1 min | 1.1 min | +0.0 min |
| ├ private_tasks (1 jobs, max) | 8.8 min | 8.8 min | −0.0 min |
| └ fuzz (1 jobs, max) | 1.1 min | 1.0 min | +0.1 min |
Minutes unless stated; durations exclude queue time unless stated; "max" = longest job in the group. Baseline = median of the last 10 successful runs (Dev: Main on main; Cloud: recent Cloud runs). Build breakdown has no baseline. Updated on each push.
c38a40f to
e0807cd
Compare
…gate FinalSingleStateAggregator kept the serialized partial states after merging them. For aggregates whose state holds every value (median, array_agg, uniq) that is the whole input, kept until the query ends.
Joins with a final scan count wait_probe_counter down to zero before the probes finish, so the last probe never saw 1 in finish_build and the build side and hash table were kept until the query ended. Count unfinished probes separately. The hash table entry storage also lived on the build state and was never cleared, growing across spill rounds; keep it next to the table and clear it whenever the table is replaced or released.
The set of seen key digests grows with the input and was kept after the processor finished, through the segment rewrite and commit that follow.
The last partition's blocks are only released when a new partition starts, so they were kept until the query ended. Without PARTITION BY that is the whole input.
Closing the channel keeps queued payloads alive while any sender exists, so a consumer that stopped early held its unread payloads until the query ended. Also drop buffered payloads once every consumer is gone.
The probe and build processors kept the shared state, with the concatenated build side and the packed R-tree, until the query ended. Take it when each processor finishes so it is freed when the last one is done.
…inish The partition stream shared by partial aggregates reserves each bucket's last flushed size again after a flush, and finishing only finalized buckets that still held rows, so the reserved capacity stayed until the query ended. Drop every bucket builder once the last partial finishes.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 166e52bdc4
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
369a9a3 to
166e52b
Compare

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/
Summary
The pipeline executor keeps every processor in the graph until the query's
RunningGraphis dropped; returningEvent::Finishedonly marks the node as finished. So any state an operator does not release itself when it finishes stays allocated for the rest of the query, while the operators above it keep running. Several operators keep state that grows with their input this way. This PR releases it when the operator finishes, one commit per operator:TransformRangeJoinLeft/RightRangeJoinState). The transforms now holdOption<Arc<RangeJoinState>>and take it on finish, like the new hash join'sFinishedJoinswap.FinalSingleStateAggregatormedian,array_agg,uniqand similar that is the whole input.TransformHashJoinProbe/HashJoinBuildStatewait_probe_counteris already zero before the probes finish, so the last probe never saw1infinish_buildand the build side plus hash table were never released. Unfinished probes are now counted separately. The hash table entry storage lived on the build state and was never cleared (growing across spill rounds); it now sits next to the table and is cleared whenever the table is replaced or released.ReplaceIntoProcessorTransformWindowPARTITION BYthat is the whole input.MaterializedCteSinkTransformSpatialJoinProbe/BuildSharedPartitionStreammax_block_bytesper bucket) stayed until the query ended.Bounded per-processor buffers (filter selection vectors, hash table initial capacity, file writer buffers, exchange partition builders) are left as they are.
Tests
Type of change
AI assistance
This change is