Fix join runtime filter crash on nested Variant/Dynamic key#106931
Conversation
A join key that contains a Variant or Dynamic nested inside a compound type (Tuple/Array/Map) crashed with LOGICAL_ERROR 'Unexpected return type from __applyFilter. Expected UInt8. Got Nullable(UInt8)' when exactly one distinct value was present on the right side of the join. The runtime filter computes whether the key can be NULL to decide whether the single-element "equals" lookup path is safe. It used hasNullable() plus a top-level isVariant()/isDynamic() check. hasNullable() recurses into Array/Tuple/Map but does not treat Variant/Dynamic as null-bearing, and the isVariant()/isDynamic() checks only inspect the top-level type, so a nested Variant was not detected. The "equals" path was then chosen and produced a Nullable(UInt8), tripping the __applyFilter return-type assertion. Detect Variant/Dynamic anywhere in the key type via a new recursive hasVariantOrDynamic() helper and use it in the runtime filter. hasNullable() is intentionally left unchanged because it is also used for MergeTree key validation, where Variant is rejected by a separate check. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
cc @vdimir @Avogar — could you review this? It fixes a join runtime filter crash ( |
|
Workflow [PR], commit [0d70b12] Summary: ❌
AI ReviewSummaryThis PR fixes the PR Metadata
Fix a `LOGICAL_ERROR` exception in the `JOIN` runtime filter when the join key contains a `Variant` or `Dynamic` type nested inside a `Tuple`, `Array`, or `Map` and the right side of the join has a single distinct value.Final VerdictStatus: ✅ Approve |
The fix detects Variant or Dynamic nested in Tuple/Array/Map, but the test only exercised nested Variant. A regression that stopped detecting nested Dynamic in the one-distinct-value runtime filter path would have passed. Add Tuple(Dynamic), Array(Dynamic) and Map(String, Dynamic) join keys under allow_dynamic_type_in_join_keys, covering both the single-value "equals" path and the multi-value set-lookup path. Verified the single-value Dynamic cases abort with the same 'Unexpected return type from __applyFilter' logical error before the fix. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Added nested- Verified both directions on a debug build: the single-value nested- New HEAD: e6b6796. Session: cron:clickhouse-worker-slot-0:20260610-101200 |
…ntainNulls Per review feedback, replace the hasNullable() || hasVariantOrDynamic() pair at the runtime-filter call site with a single recursive helper hasTypeThatCanContainNulls() that reports any type which can hold a NULL: Nullable, LowCardinality(Nullable), Variant, and Dynamic, at the top level or nested inside Array/Tuple/Map. The helper is built on the existing canContainNull() predicate (Nullable || LowCardinality(Nullable) || Variant || Dynamic), so it subsumes both former checks: hasNullable()'s Nullable/LowCardinality(Nullable) coverage and the nested Variant/Dynamic coverage. This also closes a gap for keys nesting LowCardinality(Nullable) inside a compound type: equals() over Tuple(LowCardinality(Nullable(...)), ...) returns Nullable(UInt8), so the single-distinct-value "equals" runtime-filter path would otherwise be chosen and trip the __applyFilter return-type assertion, exactly as for nested Variant/Dynamic. hasNullable() itself is left unchanged because it is also consumed by MergeTree key validation and default-expression handling, where Variant / Dynamic / LowCardinality semantics must not change. Extends test 04327 with LowCardinality(Nullable) nested-key cases for the one-distinct-value and set-lookup paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pre-PR validation gate
Session id: cron:clickhouse-worker-slot-3:20260610-123700 |
|
@Avogar done in ff72865. Replaced the
This also closes the |
|
CI status for ff72865: the only red check is |
|
CI fully finished on HEAD
All other checks pass. Ready for review. |
…apable keys Covers the ExactNotContainsRuntimeFilter (LEFT ANTI JOIN) path for keys that carry a NULL the top-level isNull bypass does not see: Variant/Dynamic, and nested Nullable/LowCardinality(Nullable) inside Tuple/Array/Map. These keys compare null-safely in the join (tuple(NULL,1) = tuple(NULL,1) matches; a top-level Variant/Dynamic NULL also matches NULL), so a nested-NULL key present on both sides is correctly dropped by the anti-join, which is what the exclusion runtime filter with transform_null_in does. The existing addNullBypassForAntiJoin therefore only needs to bypass top-level Nullable/LowCardinality(Nullable), where NULL = NULL is false in the join. Each case asserts the LEFT ANTI JOIN result with enable_join_runtime_filters=0 equals the result with it = 1, on the equals path and the set (negative lookup) path, for Variant/Dynamic/LowCardinality(Nullable) nested keys with NULLs on both sides. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pre-PR validation gateFor the LEFT ANTI JOIN runtime-filter regression added in commit 62f84fe (new HEAD
Session id: cron:clickhouse-worker-slot-3:20260610-200000 |
hasTypeThatCanContainNulls() recurses into both the Map key type and the value type (DataTypeMap::getKeyType || getValueType), but 04327 only covered Map(String, Variant) / Map(String, Dynamic), exercising the value branch. A Variant/Dynamic Map key (Map(Variant(...), String), Map(Dynamic, String)) is a valid join key that reaches the same runtime-filter path; the older value-only hasNullable() would report it as not null-capable, so without the key-side recursion the single-distinct-value "equals" path would be chosen and trip the __applyFilter return-type assertion. Add single-distinct-value equals-path cases and LEFT ANTI JOIN runtime-filters off==on cases for Variant and Dynamic Map keys, so the Map coverage exercises both key and value positions. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pre-PR validation gateCovers HEAD 0d70b12 (fixup over 62f84fe adding Map key-position coverage per the review comment).
Session id: cron:clickhouse-worker-slot-21:20260610-221400 |
LLVM Coverage Report
Changed lines: Changed C/C++ lines covered by tests: 18/18 (100.00%) | Lost baseline coverage: none · Uncovered code |
CI status — all failures triaged (commit 0d70b12)CI has fully finished (Finish Workflow + Mergeable both green). The single red functional check is not caused by this PR (which touches
All other 163 checks passed. Nothing in this PR's code path failed. Ready for review (cc @vdimir @Avogar — the |
|
Stateless tests (arm_asan_ubsan, azure, sequential) - fixed in master |
f9c3eda
|
Hi @groeneai @Avogar — while reviewing this PR I found the following:
Happy to discuss — close anything that's wrong or already addressed. |

Fixes a server crash (LOGICAL_ERROR) when a JOIN key contains a Variant or Dynamic nested inside a Tuple, Array, or Map and exactly one distinct value is present on the right side of the join.
The join runtime filter decides whether the single-element "equals" lookup path is safe by checking if the key can hold a NULL. It used
hasNullable()plus a top-levelisVariant()/isDynamic()check.hasNullable()recurses into Array/Tuple/Map but does not treat Variant/Dynamic as null-bearing, and theisVariant()/isDynamic()checks only inspect the top-level type. So a nested Variant was not detected, the "equals" path was chosen, andequalsover a Variant argument returnsNullable(UInt8), tripping the__applyFilterreturn-type assertion.A new recursive
hasTypeThatCanContainNulls()helper detects any null-capable type (Nullable,LowCardinality(Nullable),Variant,Dynamic) anywhere in the key type.hasNullable()is left unchanged on purpose because it is also used for MergeTree key validation, where Variant is rejected by a separate check.Reproducer (crashed before, returns
(true,1)now):Closes #101415
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):
Fix a server crash (
LOGICAL_ERROR) in the join runtime filter when the join key contains aVariantorDynamictype nested inside aTuple,Array, orMapand the right side of the join has a single distinct value.Documentation entry for user-facing changes
Version info
26.6.1.864