refactor: extract shared runtime filter components by pingzh · Pull Request #5937 · apache/datafusion-comet · GitHub
Skip to content

refactor: extract shared runtime filter components - #5937

Open
pingzh wants to merge 3 commits into
apache:mainfrom
pingzh:pingzh-runtime-filter-refactor
Open

pingzh wants to merge 3 commits into
apache:mainfrom
pingzh:pingzh-runtime-filter-refactor

Conversation

@pingzh

@pingzh pingzh commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Part 1 of the five-PR plan for #5775. The issue remains open for the later steps.

Rationale for this change

Give join wiring, decoded-batch filtering, and Parquet reader attachment clear ownership so the latter two can be reused by TopK.

What changes are included in this PR?

Three commits:

  1. Move DynamicFilterJoinExec and its existing tests into dynamic_filter_join; keep the decoded-batch consumer in dynamic_filter with an explicit owner-metrics constructor.
  2. Extract the existing reader traversal/remapping into parquet_reader_filter, with its null-check recognition test.
  3. Use dynamic_filter_join_* for all six join filter metrics across Rust, Spark, tests, and docs. The shared consumer receives its metric prefix from its owner.

Filtering behavior, eligibility, and execution lifetime are preserved. Most of the diff moves existing code and tests. Metric consumers must use the renamed keys.

How are these changes tested?

  • 19 native tests passed: all 18 existing runtime-filter tests plus the planner metric-export test. Each extraction commit also passed the 18 operator tests independently.
  • 52 Spark 4.1.3 join tests passed on JDK 21. The freshly rebuilt native library and Spark-packaged copy have matching SHA-256 hashes.
  • Whole-workspace Clippy with all targets and warnings denied, Cargo formatting, Spotless, Scalastyle, and Prettier passed. Native build and Spark 4.1 test compilation passed.
  • Three independent sub-agents reviewed the final patch and found no remaining concerns; each commit was reviewed before committing.

@github-actions github-actions Bot added enhancement New feature or request area:joins Join operators and dynamic filter pushdown labels Sep 15, 2026
@pingzh
pingzh marked this pull request as ready for review September 15, 2026 00:27
@pingzh
pingzh force-pushed the pingzh-runtime-filter-refactor branch from 34e3018 to 93b3061 Compare September 15, 2026 00:29

@sunchao sunchao left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 93b3061 against 39da850. No actionable findings after five independent review scopes covering native join wiring, Spark metrics, memory/error cleanup, Parquet reader compatibility, and test/edge-case preservation.

The extraction preserves join eligibility, build/probe selection, independent execution state, reset/child-replacement behavior, and EOF/error/cancellation cleanup. Reader remapping and boundaries around limits, projections, and stateful residual expressions are unchanged. All six renamed metric keys match across native producers, Spark consumers, tests, and documentation. The existing native and Scala test assertions remain intact.

Validation:

  • Inspected the native CI job: 1,447 tests passed, 5 skipped, including all 18 runtime-filter operator tests and the planner metric-export test.
  • Verified 52 passing CometJoinSuite tests each in the Spark 3.4 and Spark 4.2 execution jobs. Those whole-job runs had zero failures, with 15 canceled/5 ignored and 7 canceled/5 ignored tests respectively.
  • CI ran merge commit e2155db. Its source tree is identical to the reviewed head.
  • Local formatting and diff-whitespace checks passed. Local native compilation stopped during dependency resolution because the configured package mirror lacks the locked DataFusion 55.1.0 dependency, so native/JVM runtime tests were not reproduced locally.

At submission, CI has 43 successful checks, 8 running, 1 queued, 9 skipped, and no failures. The remaining workflow results are pending.

LGTM. Approving the reviewed revision.

@rich7420 rich7420 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pingzh thanks for the patch

@sunchao sunchao left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rechecked unchanged 93b306177abeb5ab72cd10bfb63c0ab35fc4735c against 39da85081b140dae57492f96e3478337ee345a8f. My existing approval remains. The decoded-batch filtering and eligibility bodies are byte-identical to the base. The moved lifecycle and reader logic retain their behavior, and all 18 operator tests retain their assertions. The four formatted metric names are constructed when an execution opens, outside the batch loop. All six renamed keys match the native producers, Spark consumers and documentation.

CI has now completed with 53 successful checks and 10 skipped. I verified 1,447 native tests passing, including all 18 runtime-filter operator tests and the metric-export test, with 5 tests skipped. The logs also show 52 passing CometJoinSuite tests on each of Spark 3.4, 3.5, 4.0 and 4.1. The executed merge e2155db8 has the same source tree as this head, and the native artifact's upload/download hashes agree.

Source-equivalence and diff-whitespace checks pass. I found no new P1/P2 finding. I did not run native/JVM tests or benchmarks locally. Maintained Spark 3.5/4.0 source was checked for null-key and duplicate-match semantics. Maintained 3.4/4.1 sources remain unavailable, independently of the CI results.

@peterxcli peterxcli left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's make the structure for dyn filter like:

dynamic_filter/
├─ mod.rs              (shared runtime-filter components; from dynamic_filter.rs)
├─ join.rs             (DynamicFilterJoinExec)
├─ parquet_reader.rs   (try_attach_parquet_reader_filter, null-check logic)
└─ tests/
   ├─ mod.rs           (#[cfg(test)] mod join; mod parquet_reader;)
   ├─ join.rs
   └─ parquet_reader.rs

as we will have topk.rs, agg.rs and their tests

@ajsquared ajsquared left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all 11 changed files and relevant callers at 93b3061. Runtime-filter behavior and lifetime handling are preserved, and metric names align across Rust and Spark. No independently confirmed P1 findings. Static review only; tests and CI were not run or inspected.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:joins Join operators and dynamic filter pushdown enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants