iframe-proxy

groeneai · 2026-04-21T00:54:04Z

Fixes a LOGICAL_ERROR server exception during MergeTree index analysis when a constant in a WHERE clause is cast to a type containing nested LowCardinality (for example, a Variant(LowCardinality(Date), String)).

CI report: https://s3.amazonaws.com/clickhouse-test-reports/json.html?REF=master&sha=728336b344ee2de5aaace19f7a67079b3070d03c&name_0=MasterCI&name_1=Stress%20test%20%28experimental%2C%20serverfuzz%2C%20amd_tsan%29 (STID 4054-600e)

Root cause

KeyCondition::applyFunctionChainToColumn prepared the input constant by calling convertToFullIfNeeded on the column and removeLowCardinality on the type. The two were not symmetric:

removeLowCardinality only strips the outer-level LowCardinality wrapper from the type.
The default IColumn::convertToFullIfNeeded implementation recurses into sub-columns via forEachSubcolumn and strips LowCardinality from every nested sub-column.

ColumnVariant overrides convertToFullIfNeeded to a no-op precisely because there is no way to keep the DataTypeVariant in sync with per-variant column substitutions. But when the ColumnVariant is wrapped in a ColumnConst (which is what canConstantBeWrappedByMonotonicFunctions produces via createColumnConst), the default implementation on ColumnConst is used and bypasses the ColumnVariant override — stripping LowCardinality from the variant sub-columns while the declared DataTypeVariant is left untouched by removeLowCardinality.

The subsequent castColumnAccurate / castColumnAccurateOrNull built a FunctionCast from the mismatched Variant(LowCardinality(X), ...) type. Inside createVariantToColumnWrapper, the per-variant wrapper was prepareUnpackDictionaries(LowCardinality(X), ...), which typeid_casts its input to ColumnLowCardinality. At runtime the sub-column was a plain ColumnVector, so the server aborted with:

Logical error: 'Bad cast from type DB::ColumnVector<...> to DB::ColumnLowCardinality'

The same asymmetry is triggered by Array(LowCardinality(X)), Tuple(LowCardinality(X), ...), and Map(LowCardinality(K), V) constants, but those paths are currently reachable only through the server-side AST fuzzer.

Fix

Strip only outer-level wrappers on the column so that the transformation is symmetric with what removeLowCardinality does on the type:

auto result_column = in_column
    ->convertToFullColumnIfConst()
    ->convertToFullColumnIfReplicated()
    ->convertToFullColumnIfSparse()
    ->convertToFullColumnIfLowCardinality();
auto result_type = removeLowCardinality(in_data_type);

Nested LowCardinality inside container types (Array, Tuple, Map, Variant) is preserved on both sides. Downstream code is unaffected: FunctionCast already handles outer-level and nested LowCardinality via its own wrappers, and the existing defensive re-wrap at the func->execute call site continues to handle the outer-LC case.

Reproducer

SET allow_experimental_variant_type = 1;
SET allow_suspicious_low_cardinality_types = 1;

CREATE TABLE t (d LowCardinality(Date), v UInt32)
ENGINE = MergeTree
PARTITION BY toYYYYMM(d)
ORDER BY d
SETTINGS allow_suspicious_low_cardinality_types = 1;

INSERT INTO t VALUES ('2026-04-01', 1), ('2026-04-19', 2), ('2026-04-20', 3);

SELECT * FROM t
WHERE d >= CAST('2026-04-19' AS Variant(LowCardinality(Date), String));

Before the fix: server aborts with Bad cast from type DB::ColumnVector<unsigned short> to DB::ColumnLowCardinality.
After the fix: returns rows (2026-04-19, 2) and (2026-04-20, 3).

CIDB history

STID 4054-600e: 11 hits over 90 days across serverfuzz / Stress test / AST fuzzer checks, all going through PartitionPruner (7/11) or MergeTreeIndexMinMax (4/11) and ending in FunctionCast::prepareUnpackDictionaries. No open GitHub issue tracks this STID specifically.

Test

Added tests/queries/0_stateless/04105_key_condition_nested_lc_variant.sql covering the Variant(LowCardinality(Date), String) partition-pruning path and the Variant(LowCardinality(String), UInt64) primary-key path.

Changelog category (leave one):

Bug Fix (user-visible misbehavior in an official stable release)

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Fix LOGICAL_ERROR (Bad cast … to ColumnLowCardinality) during MergeTree index analysis when a WHERE clause compares a LowCardinality key column with a constant cast to a type that contains nested LowCardinality (for example Variant(LowCardinality(Date), String)). #NNNNN

Documentation entry for user-facing changes

Documentation is written (mandatory for new features)

Session: cron:clickhouse-ci-task-worker:20260421-001500

Version info

Merged into: 26.5.1.26

`KeyCondition::applyFunctionChainToColumn` used to call `convertToFullIfNeeded` on the input column before casting it through the monotonic function chain, while stripping only outer-level `LowCardinality` from the type via `removeLowCardinality`. For a constant of type `Variant(LowCardinality(Date), String)` wrapped in `ColumnConst`, the default `IColumn::convertToFullIfNeeded` implementation bypasses `ColumnVariant`'s own override (which deliberately avoids recursing into variant sub-columns) and strips `LowCardinality` from every variant sub-column. The type meanwhile keeps its nested `LowCardinality`. The subsequent `castColumnAccurateOrNull` eventually dispatches into `FunctionCast::prepareUnpackDictionaries`, which asserts a `ColumnLowCardinality` but sees the plain `ColumnVector`, aborting the server with `LOGICAL_ERROR`: Bad cast from type DB::ColumnVector<...> to DB::ColumnLowCardinality Replace `convertToFullIfNeeded` with explicit outer-level `convertToFullColumnIfConst`/`Replicated`/`Sparse`/`LowCardinality` chain so that the column stripping matches `removeLowCardinality` on the type. Nested `LowCardinality` inside container types (`Array`, `Tuple`, `Map`, `Variant`) is preserved on both sides, so subsequent casts stay consistent and `FunctionCast` handles any remaining `LowCardinality` internally through its own wrappers. The crash is reached via `PartitionPruner` / `MergeTreeIndexMinMax` during index analysis and reproduces deterministically with a filter that casts a constant to a `Variant` containing nested `LowCardinality` over a table with `LowCardinality` key. CIDB shows 11 hits over 90 days on `serverfuzz` / `Stress test` / `AST fuzzer` checks covering `PartitionPruner` and `MergeTreeIndexMinMax` paths (STID 4054-600e). Reported in CI: STID 4054-600e on master `Stress test (experimental, serverfuzz, amd_tsan)` at commit 728336b

groeneai · 2026-04-21T00:54:43Z

groeneai · 2026-04-21T00:55:13Z

Pre-PR validation gate

a) Deterministic repro? Yes.

SET allow_experimental_variant_type=1;
SET allow_suspicious_low_cardinality_types=1;
CREATE TABLE t (d LowCardinality(Date), v UInt32) ENGINE=MergeTree
PARTITION BY toYYYYMM(d) ORDER BY d SETTINGS allow_suspicious_low_cardinality_types=1;
INSERT INTO t VALUES ('2026-04-01', 1), ('2026-04-19', 2), ('2026-04-20', 3);
SELECT * FROM t WHERE d >= CAST('2026-04-19' AS Variant(LowCardinality(Date), String));

Aborts every run before the fix; returns correct rows after the fix.

b) Root cause explained? Yes.

canConstantBeWrappedByMonotonicFunctions builds a ColumnConst via out_type->createColumnConst(1, value). For out_type = Variant(LowCardinality(Date), String), this creates ColumnConst<ColumnVariant(ColumnLowCardinality<Date>, ColumnString)>.
applyFunctionChainToColumn calls in_column->convertToFullIfNeeded(). The default IColumn implementation runs convertToFullColumnIfConst()->...->convertToFullColumnIfLowCardinality() (unwraps the Const), then iterates sub-columns via forEachSubcolumn and calls convertToFullIfNeeded on each.
ColumnVariant has an override of convertToFullIfNeeded (returns getPtr()) precisely to avoid stripping LC inside variants — but the override is only invoked when the caller has a direct ColumnVariant, not when the path goes through ColumnConst's default impl. The iteration strips LC from the LC variant sub-column, producing ColumnVariant(ColumnVector<UInt16>, ColumnString).
removeLowCardinality(Variant(LC(Date), String)) only strips outer-level LC → type stays Variant(LC(Date), String).
castColumnAccurateOrNull builds a FunctionCast from the (now mismatched) type to the target. createVariantToColumnWrapper creates prepareUnpackDictionaries(LC(Date), Date) for variant 0, which typeid_casts the runtime column to ColumnLowCardinality. The column is the stripped ColumnVector<UInt16>, so typeid_cast fails.

c) Fix matches root cause? Yes — replace the asymmetric call with an outer-only strip that matches what removeLowCardinality does on the type: convertToFullColumnIfConst -> convertToFullColumnIfReplicated -> convertToFullColumnIfSparse -> convertToFullColumnIfLowCardinality. No deep recursion; no bypass of ColumnVariant's invariant.

d) Test intent preserved / new test added? Yes — no existing test assertions are weakened. New regression test 04105_key_condition_nested_lc_variant.sql covers the Variant(LowCardinality(Date), String) partition-pruning path and the Variant(LowCardinality(String), UInt64) primary-key path.

e) Demonstrated in both directions? Yes.

Pre-fix binary: repro query aborts server with Bad cast from type DB::ColumnVector<unsigned short> to DB::ColumnLowCardinality.
Post-fix binary: repro query returns (2026-04-19, 2), (2026-04-20, 3).
New regression test passes (04105_key_condition_nested_lc_variant), 03402_has_low_cardinality_key_condition still passes, broad sweep of LC / variant / key_condition / partition_pruner tests shows no regression.

f) Fix is general, not a narrow patch? Yes.

The bug pattern exists symmetrically for Array(LowCardinality(X)), Tuple(LowCardinality(X), ...), and Map(LowCardinality(K), V) — any container that has nested LC and is wrapped in ColumnConst. All of those paths go through the same convertToFullIfNeeded → forEachSubcolumn recursion that strips inner LC. The fix addresses all of them in the single call site.
"Am I fixing the root cause or guarding the symptom?" — Root cause. The fix removes the asymmetry at the source (column stripping) so the column/type invariant is preserved, rather than catching the mismatch later.
Not a defensive null check — the symptom never reaches typeid_cast under the new stripping semantics.

Session: cron:clickhouse-ci-task-worker:20260421-001500

clickhouse-gh · 2026-04-21T02:55:25Z

Workflow [PR], commit [5a11f97]

Summary: ❌

job_name	test_name	status	info	comment
Stateless tests (amd_llvm_coverage, AsyncInsert, s3 storage, parallel)		FAIL
	03258_nonexistent_db	FAIL	cidb	IGNORED

AI Review

Summary

This PR fixes a real correctness issue in KeyCondition::applyFunctionChainToColumn where convertToFullIfNeeded could recursively strip nested LowCardinality from subcolumns while the paired type transformation (removeLowCardinality) only removed the outer wrapper, leading to a column/type mismatch and a LOGICAL_ERROR exception in cast wrappers during MergeTree index analysis. The implementation change is surgical (outer-wrapper-only normalization on the column side) and the added stateless regression test covers the reported Variant(LowCardinality(...), ...) path; I did not find additional high-confidence correctness, safety, or performance problems in the patch.

ClickHouse Rules

Item	Status	Notes
Deletion logging	➖
Serialization versioning	➖
Core-area scrutiny	✅
No test removal	✅
Experimental gate	➖
No magic constants	✅
Backward compatibility	✅
`SettingsChangesHistory.cpp`	➖
PR metadata quality	✅
Safe rollout	✅
Compilation time	✅
No large/binary files	✅

Final Verdict

Status: ✅ Approve

nihalzp · 2026-04-21T05:17:38Z

@groeneai Failed stateless tests look related.

The regression test for STID 4054-600e uses `PARTITION BY toYYYYMM(d)` on a `LowCardinality(Date)` column and casts a constant to `Variant(LowCardinality(Date), String)`. Under parallel replicas the query tree is serialized with the `Date` constant folded to its raw `UInt16` representation and re-parsed on the replica, where `FunctionCast::createColumnToVariantWrapper` rejects `UInt16 -> Variant(LowCardinality(Date), String)` with `CANNOT_CONVERT_TYPE` before index analysis runs. That is a separate, pre-existing limitation in parallel replicas constant serialization, unrelated to the `KeyCondition` fix this PR targets. Skip the parallel-replicas variants so the test exclusively exercises the single-server `KeyCondition::applyFunctionChainToColumn` path that the fix addresses. Remove the secondary `LowCardinality(String)` case which doesn't reach `canConstantBeWrappedByMonotonicFunctions` through its ORDER-BY-only schema and so was not actually exercising the fix. Verified locally: reproduces `Bad cast … to ColumnLowCardinality` without the `KeyCondition` change, returns the expected rows with it. Reviewer feedback: ClickHouse#103211 (comment)

groeneai · 2026-04-21T17:45:20Z

Thanks @nihalzp — you were right, the 04105 stateless test is related. Pushed a follow-up (5a11f97) tagging it no-parallel-replicas with a comment explaining why.

Root cause of the CI failures: the new test exercises partition pruning over LowCardinality(Date) with a constant cast to Variant(LowCardinality(Date), String). That specific shape is needed to reach KeyCondition::canConstantBeWrappedByMonotonicFunctions where this fix applies. Reproduced locally — the query hits the bug fix as intended.

Under parallel replicas / distributed plan the query tree is serialized with the Date constant folded to its raw UInt16 representation. When the replica re-parses it as _CAST(20562, 'Variant(LowCardinality(Date), String)'), FunctionCast::createColumnToVariantWrapper (FunctionsConversion.cpp:1446) rejects the UInt16 → Variant(LowCardinality(Date), String) conversion with CANNOT_CONVERT_TYPE — the analyzer runs before index analysis, so we never reach the code path this PR fixes.

I verified this is a pre-existing parallel-replicas serialization limitation (not introduced by this fix) by reproducing the same CANNOT_CONVERT_TYPE via remote():

SELECT d, v FROM remote('127.0.0.1:19019', currentDatabase(), 't_04105_lc_date')
WHERE d >= CAST('2026-04-19' AS Variant(LowCardinality(Date), String))
ORDER BY d;
-- Code: 70. Cannot convert type UInt16 to Variant(LowCardinality(Date), String).

Fixing the analyzer's Variant constant serialization is out of scope for this PR. Similar MergeTree index-analysis tests use the same tag (e.g. 03374_indexes_with_trivial_cast.sql, 03374_indexes_with_literals.sql).

The other five LLVM-coverage stateless failures (04078_native_protocol_input_validation, 02992_all_columns_should_have_comment, 00600_replace_running_query, 03096_order_by_system_tables, 03258_nonexistent_db) all passed on rerun (0/3 reproducibility) and match known LLVM-coverage timing-flake patterns — unrelated to this PR.

clickhouse-gh · 2026-04-21T20:37:39Z

LLVM Coverage Report

Metric	Baseline	Current	Δ
Lines	84.10%	84.10%	+0.00%
Functions	91.10%	91.10%	+0.00%
Branches	76.50%	76.60%	+0.10%

Changed lines: 100.00% (6/6) · Uncovered code

Full report · Diff report

groeneai · 2026-04-22T21:17:27Z

Gentle re-ping @CurtizJ @KochetovNicolai — could you take a look when you have a moment?

Quick status:

CI green: 142 pass / 16 skip / 3 fail where all 3 failures trace to one pre-existing LLVM-coverage flake (03258_nonexistent_db, tracked separately under PR Fix flaky test 03258_nonexistent_db on LLVM coverage builds #103287). Not caused by this fix.
04105_key_condition_nested_lc_variant (the new regression test) now passes on every variant including ParallelReplicas after tagging no-parallel-replicas per @nihalzp's earlier feedback.
STID 4054-600e on master: 0 recurrences in the ~50 hours since this PR was opened (last master hit was 2026-04-20 on commit 728336b3, pre-dating this PR).
Reproducer + both-direction validation are in the pre-PR gate comment above.

Happy to rebase or tweak if anything needs changing.

clickgapai · 2026-04-23T05:03:37Z

Hi @groeneai @nihalzp — the changelog category for this PR might need updating.

Current: Bug Fix
Suggested: Critical Bug Fix (crash, data loss, RBAC) or LOGICAL_ERROR

Why: The PR fixes a LOGICAL_ERROR ('Bad cast from ColumnVector to ColumnLowCardinality'), which belongs in the 'Critical Bug Fix' category according to the changelog guidelines.

Could you verify this is correct? Ignore if the current category is intentional.

clickgapai · 2026-04-23T05:03:57Z

nihalzp self-assigned this Apr 21, 2026

nihalzp added the can be tested Allows running workflows for external contributors label Apr 21, 2026

clickhouse-gh Bot added the pr-bugfix Pull request with bugfix, not backported by default label Apr 21, 2026

nihalzp approved these changes Apr 23, 2026

View reviewed changes

nihalzp added this pull request to the merge queue Apr 23, 2026

Merged via the queue into ClickHouse:master with commit d95da10 Apr 23, 2026
159 of 161 checks passed

robot-clickhouse added the pr-synced-to-cloud The PR is synced to the cloud repo label Apr 23, 2026

groeneai mentioned this pull request Jun 11, 2026

Fix Bad cast in KeyCondition for Variant constant with LowCardinality member #107111

Merged

Sunbelt Computer Software

PL/B Language Development and Support

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix crash in partition pruning with nested LowCardinality constant#103211

Fix crash in partition pruning with nested LowCardinality constant#103211
nihalzp merged 2 commits into
ClickHouse:masterfrom
groeneai:fix-keycondition-nested-lc-asymmetry-STID-4054

groeneai commented Apr 21, 2026 •

edited by robot-clickhouse

Loading

Uh oh!

groeneai commented Apr 21, 2026

Uh oh!

groeneai commented Apr 21, 2026

Uh oh!

clickhouse-gh Bot commented Apr 21, 2026 •

edited by nihalzp

Loading

Uh oh!

nihalzp commented Apr 21, 2026

Uh oh!

groeneai commented Apr 21, 2026

Uh oh!

clickhouse-gh Bot commented Apr 21, 2026

Uh oh!

groeneai commented Apr 22, 2026

Uh oh!

Uh oh!

clickgapai commented Apr 23, 2026

Uh oh!

clickgapai commented Apr 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Sunbelt Computer Software

PL/B Language Development and Support

Uh oh!

Conversation

groeneai commented Apr 21, 2026 • edited by robot-clickhouse Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Root cause

Fix

Reproducer

CIDB history

Test

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Documentation entry for user-facing changes

Version info

Uh oh!

groeneai commented Apr 21, 2026

Uh oh!

groeneai commented Apr 21, 2026

Pre-PR validation gate

Uh oh!

clickhouse-gh Bot commented Apr 21, 2026 • edited by nihalzp Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

AI Review

Summary

ClickHouse Rules

Final Verdict

Uh oh!

nihalzp commented Apr 21, 2026

Uh oh!

groeneai commented Apr 21, 2026

Uh oh!

clickhouse-gh Bot commented Apr 21, 2026

LLVM Coverage Report

Uh oh!

groeneai commented Apr 22, 2026

Uh oh!

Uh oh!

clickgapai commented Apr 23, 2026

Uh oh!

clickgapai commented Apr 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

groeneai commented Apr 21, 2026 •

edited by robot-clickhouse

Loading

clickhouse-gh Bot commented Apr 21, 2026 •

edited by nihalzp

Loading