fix(query): seed filter domains with boolean identities by dantengsky · Pull Request #20803 · databendlabs/databend · GitHub
Skip to content

fix(query): seed filter domains with boolean identities - #20803

Merged
dantengsky merged 3 commits into
databendlabs:mainfrom
dantengsky:fix/m0-nullable-or-fold
Oct 23, 2026
Merged

dantengsky merged 3 commits into
databendlabs:mainfrom
dantengsky:fix/m0-nullable-or-fold

Conversation

@dantengsky

@dantengsky dantengsky commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Motivation

Fix unnecessary block reads for OR predicates on nullable columns. A comparison may be known to return only FALSE or NULL; neither passes WHERE. If every OR branch has that property, the whole filter cannot match.

The old constant folder still marks TRUE as possible, so range/bloom pruning keeps blocks that contain no matches. Query results remain correct, but pruning is lost.

Reproduction

Use default settings in a fresh test schema. No SET or special table option is needed.

CREATE TABLE bloom_test_nullable_or_t(c1 VARCHAR NULL, c2 VARCHAR NULL, c3 INT NULL);
INSERT INTO bloom_test_nullable_or_t VALUES
  ('a1','b1',1), (NULL,'b2',2), ('a3',NULL,NULL);
INSERT INTO bloom_test_nullable_or_t VALUES
  ('a4','b4',4), (NULL,NULL,5), ('a6','b6',NULL);

EXPLAIN SELECT * FROM bloom_test_nullable_or_t
WHERE c1 = 'a2' OR c2 = 'b5';

Each INSERT creates one block. Both searched strings are absent, but lie inside the min/max ranges, so bloom membership could exclude the blocks.

EXPLAIN field Before After
read rows 6 0
partitions total 2 2
partitions scanned 2 0

The SELECT returns no rows in both versions. The difference is that the fix avoids planning reads of both blocks.

Approach

Start domain combination with FALSE for OR and TRUE for AND, rather than with both values possible. Argument handling and ordinary SQL NULL semantics are unchanged.

Tests

  • SQLLogicTest

bloom_filter.test covers impossible OR/IN predicates, nested AND, range pruning, matching branches that must retain a block, and actual returned rows where one OR branch is NULL and the other is TRUE. Comments explain the two-block layout and each expected pruning result. Rust test files are unchanged from main; the SQL coverage is not duplicated there.

The expected result checks retain NULL OR TRUE and TRUE OR NULL rows, while excluding a row where both branches are NULL. On the old implementation, the nullable-OR EXPLAIN reports the 6 rows / 2 scanned blocks above instead of zero.

Type of change

  • Bug Fix (non-breaking change which fixes an issue)

AI assistance

  • AI usage: AI coding agents investigated the domain fold, drafted the minimal initializer change and synthetic regressions, and assisted validation and privacy review.
  • Responsible human: @dantengsky
  • The responsible human has read every line of this diff and can explain each change

This change is Reviewable

@github-actions github-actions Bot added the pr-bugfix this PR patches a bug in codebase label Oct 20, 2026
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-21T08:56:48.451598Z ad36bb2 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

⏱ CI timing · Dev

run 39560811738 · 24.4 min (main p50 30.4 min, −6.0 min)

stage this run main p50 Δ
total 24.4 min 30.4 min −6.0 min
build (aarch64, ci profile) 14.1 min 17.6 min −3.5 min
├ deps + crates (parallel) 0.7 min — —
├ databend-query codegen 3.6 min — —
├ databend-binaries link 7.9 min — —
└ sccache hit rate 99.7 % — —
test_unit 14.7 min 15.3 min −0.5 min
mac_check — 13.6 min —
tests after build (49 jobs) 9.8 min 14.2 min −4.4 min
├ queue p50 / max 1.4 min / 4.1 min 1.8 min / 9.4 min −0.3 min
├ sqllogic standalone (8 jobs, max) 4.0 min 4.1 min −0.1 min
├ sqllogic cluster (8 jobs, max) 7.1 min 7.2 min −0.1 min
├ sqllogic stage (2 jobs, max) 5.0 min 4.8 min +0.2 min
├ sqllogic other (10 jobs, max) 5.8 min 5.9 min −0.1 min
├ stateless (2 jobs, max) 6.4 min 6.2 min +0.2 min
├ stateful (4 jobs, max) 4.6 min 4.7 min −0.1 min
├ compat (6 jobs, max) 4.7 min 4.8 min −0.1 min
├ meta / logs / lineage / udf (5 jobs, max) 4.5 min 4.4 min +0.0 min
├ ee (2 jobs, max) 1.0 min 1.1 min −0.1 min
├ private_tasks (1 jobs, max) 8.8 min 8.8 min +0.0 min
└ fuzz (1 jobs, max) 1.0 min 1.1 min −0.1 min

Minutes unless stated; durations exclude queue time unless stated; "max" = longest job in the group. Baseline = median of the last 10 successful runs (Dev: Main on main; Cloud: recent Cloud runs). Build breakdown has no baseline. Updated on each push.

@dantengsky
dantengsky marked this pull request as draft October 21, 2026 06:40
@dantengsky

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

@dantengsky
dantengsky marked this pull request as ready for review October 21, 2026 08:53
@dantengsky
dantengsky requested review from KKould and sundy-li October 23, 2026 03:49

@sundy-li sundy-li left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated AI-assisted review of ad36bb2514a649f075474905b1e9c28fe88ee811 on behalf of @sundy-li.

No blocking issue found after primary and independent review. The false/true seeds are the correct identities for or_filters/and_filters, whose Boolean output treats NULL as not-TRUE; ordinary nullable SQL AND/OR remain unchanged. Checked unknown domains, existing short-circuit/error handling, and the bloom/range consumers. The new result tests retain NULL OR TRUE and TRUE OR NULL rows as expected.

Validation: 354,288 local combinations of the extracted production initializer/merge closure matched an independent three-valued-logic oracle. Inspected exact-head CI run 39560811738: unit tests passed and the expanded explain/bloom_filter.test passed in HTTP and hybrid/TTC modes. These local checks are isolated domain probes, not a full local engine/workspace run.

This automated review does not replace the responsible human's full-diff review before merge.

@dantengsky
dantengsky added this pull request to the merge queue Oct 23, 2026
Merged via the queue into databendlabs:main with commit b2ff6f3 Oct 23, 2026
77 of 79 checks passed
@dantengsky
dantengsky deleted the fix/m0-nullable-or-fold branch October 23, 2026 07:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-bugfix this PR patches a bug in codebase

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants