Conversation
34e3018 to
93b3061
Compare
sunchao
left a comment
There was a problem hiding this comment.
Reviewed 93b3061 against 39da850. No actionable findings after five independent review scopes covering native join wiring, Spark metrics, memory/error cleanup, Parquet reader compatibility, and test/edge-case preservation.
The extraction preserves join eligibility, build/probe selection, independent execution state, reset/child-replacement behavior, and EOF/error/cancellation cleanup. Reader remapping and boundaries around limits, projections, and stateful residual expressions are unchanged. All six renamed metric keys match across native producers, Spark consumers, tests, and documentation. The existing native and Scala test assertions remain intact.
Validation:
- Inspected the native CI job: 1,447 tests passed, 5 skipped, including all 18 runtime-filter operator tests and the planner metric-export test.
- Verified 52 passing CometJoinSuite tests each in the Spark 3.4 and Spark 4.2 execution jobs. Those whole-job runs had zero failures, with 15 canceled/5 ignored and 7 canceled/5 ignored tests respectively.
- CI ran merge commit e2155db. Its source tree is identical to the reviewed head.
- Local formatting and diff-whitespace checks passed. Local native compilation stopped during dependency resolution because the configured package mirror lacks the locked DataFusion 55.1.0 dependency, so native/JVM runtime tests were not reproduced locally.
At submission, CI has 43 successful checks, 8 running, 1 queued, 9 skipped, and no failures. The remaining workflow results are pending.
LGTM. Approving the reviewed revision.
sunchao
left a comment
There was a problem hiding this comment.
Rechecked unchanged 93b306177abeb5ab72cd10bfb63c0ab35fc4735c against 39da85081b140dae57492f96e3478337ee345a8f. My existing approval remains. The decoded-batch filtering and eligibility bodies are byte-identical to the base. The moved lifecycle and reader logic retain their behavior, and all 18 operator tests retain their assertions. The four formatted metric names are constructed when an execution opens, outside the batch loop. All six renamed keys match the native producers, Spark consumers and documentation.
CI has now completed with 53 successful checks and 10 skipped. I verified 1,447 native tests passing, including all 18 runtime-filter operator tests and the metric-export test, with 5 tests skipped. The logs also show 52 passing CometJoinSuite tests on each of Spark 3.4, 3.5, 4.0 and 4.1. The executed merge e2155db8 has the same source tree as this head, and the native artifact's upload/download hashes agree.
Source-equivalence and diff-whitespace checks pass. I found no new P1/P2 finding. I did not run native/JVM tests or benchmarks locally. Maintained Spark 3.5/4.0 source was checked for null-key and duplicate-match semantics. Maintained 3.4/4.1 sources remain unavailable, independently of the CI results.
peterxcli
left a comment
There was a problem hiding this comment.
let's make the structure for dyn filter like:
dynamic_filter/
├─ mod.rs (shared runtime-filter components; from dynamic_filter.rs)
├─ join.rs (DynamicFilterJoinExec)
├─ parquet_reader.rs (try_attach_parquet_reader_filter, null-check logic)
└─ tests/
├─ mod.rs (#[cfg(test)] mod join; mod parquet_reader;)
├─ join.rs
└─ parquet_reader.rs
as we will have topk.rs, agg.rs and their tests
ajsquared
left a comment
There was a problem hiding this comment.
Reviewed all 11 changed files and relevant callers at 93b3061. Runtime-filter behavior and lifetime handling are preserved, and metric names align across Rust and Spark. No independently confirmed P1 findings. Static review only; tests and CI were not run or inspected.

Which issue does this PR close?
Part 1 of the five-PR plan for #5775. The issue remains open for the later steps.
Rationale for this change
Give join wiring, decoded-batch filtering, and Parquet reader attachment clear ownership so the latter two can be reused by TopK.
What changes are included in this PR?
Three commits:
DynamicFilterJoinExecand its existing tests intodynamic_filter_join; keep the decoded-batch consumer indynamic_filterwith an explicit owner-metrics constructor.parquet_reader_filter, with its null-check recognition test.dynamic_filter_join_*for all six join filter metrics across Rust, Spark, tests, and docs. The shared consumer receives its metric prefix from its owner.Filtering behavior, eligibility, and execution lifetime are preserved. Most of the diff moves existing code and tests. Metric consumers must use the renamed keys.
How are these changes tested?