Fix `UPDATE` with parallel replicas in `MergeTree` by LambertArm · Pull Request #104255 · ClickHouse/ClickHouse · GitHub
Skip to content

Fix UPDATE with parallel replicas in MergeTree#104255

Closed
LambertArm wants to merge 7 commits into
ClickHouse:masterfrom
LambertArm:fix-lwu-parallel-replicas
Closed

Fix UPDATE with parallel replicas in MergeTree#104255
LambertArm wants to merge 7 commits into
ClickHouse:masterfrom
LambertArm:fix-lwu-parallel-replicas

Conversation

@LambertArm

@LambertArm LambertArm commented May 6, 2026

Copy link
Copy Markdown

Closes #104197.

Mutation source reads build a synthetic SELECT without a table expression, so legacy parallel-replica query rewriting must not be used for them.

Mark these reads through SelectQueryInfo::parallel_replicas_disabled and honor that flag in StorageMergeTree and StorageReplicatedMergeTree before entering the legacy parallel-replica paths.

Adds a regression test for #104197.

Changelog category (leave one):

  • Critical Bug Fix (crash, data loss, RBAC)

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Fix lightweight UPDATE queries with legacy parallel replicas enabled for non-replicated MergeTree tables.

Documentation entry for user-facing changes

  • Documentation is not required.

@CLAassistant

CLAassistant commented May 6, 2026

Copy link
Copy Markdown

@alexey-milovidov

Copy link
Copy Markdown
Member

Could you please edit your commits to include your email from GitHub?

Mutation source reads build a synthetic `SELECT` without a table expression, so legacy parallel-replica query rewriting must not be used for them.

Mark these reads through `SelectQueryInfo::parallel_replicas_disabled` and honor that flag in `StorageMergeTree` and `StorageReplicatedMergeTree` before entering the legacy parallel-replica paths.

Adds a regression test for #104197.
@LambertArm LambertArm force-pushed the fix-lwu-parallel-replicas branch from 982a8c2 to cd14db7 Compare May 6, 2026 18:26
@LambertArm LambertArm marked this pull request as draft May 6, 2026 18:27
@LambertArm LambertArm marked this pull request as ready for review May 6, 2026 18:27
@LambertArm LambertArm marked this pull request as draft May 6, 2026 18:32
@LambertArm LambertArm marked this pull request as ready for review May 6, 2026 18:32
@PedroTadim PedroTadim added the can be tested Allows running workflows for external contributors label May 19, 2026
@clickhouse-gh

clickhouse-gh Bot commented May 19, 2026

Copy link
Copy Markdown
Contributor

Workflow [PR], commit [7911ab3]

Summary:


AI Review

Summary

This PR disables enable_parallel_replicas on the copied MutationsInterpreter context, so mutation source reads keep using the normal storage read path instead of entering legacy parallel-replica query rewriting with a synthetic table-less SELECT. The regression test now pins both enable_analyzer = 0 and automatic_parallel_replicas_mode = 0, so it exercises the legacy path that caused the reported SEGV. I found no remaining blockers or major issues.

Final Verdict

✅ No blocking or major review findings. The earlier inline concern about automatic_parallel_replicas_mode is addressed in the current code.

@clickhouse-gh clickhouse-gh Bot added pr-critical-bugfix pr-must-backport Pull request should be backported intentionally. Use this label with great care! labels May 19, 2026

@devcrafter devcrafter left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems all you need to disable parallel replicas for mutations. AFAIS, the proper place for it is this one -

context = std::move(new_context);

alexey-milovidov and others added 3 commits June 8, 2026 02:09
Per review feedback from @devcrafter: instead of plumbing a new
`SelectQueryInfo` flag through `StorageMergeTree` and
`StorageReplicatedMergeTree`, disable parallel replicas directly on the
per-mutation context in `MutationsInterpreter`.

Mutation source reads build a synthetic `SELECT` without a table
expression, so the legacy parallel-replica query rewriting must not be
used for them. Setting `enable_parallel_replicas` to 0 on the mutation
context covers both the task-based and custom-key paths uniformly.

The `parallel_replicas_disabled` field of `SelectQueryInfo` that the
previous approach relied on has since been removed from master.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The crash only happens on the legacy (non-analyzer) parallel-replica
path in `StorageMergeTree`. Force `enable_analyzer = 0` so the test
reliably reproduces it instead of depending on randomized settings.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@alexey-milovidov

Copy link
Copy Markdown
Member

Addressed @devcrafter's feedback and rebased onto current master.

Reworked the fix as suggested. Instead of plumbing a SelectQueryInfo::parallel_replicas_disabled flag through StorageMergeTree and StorageReplicatedMergeTree, parallel replicas are now disabled directly on the per-mutation context in the MutationsInterpreter constructor:

/// Mutation source reads build a synthetic `SELECT` without a table expression,
/// so parallel replicas must not be used for them.
new_context->setSetting("enable_parallel_replicas", Field(0));

This is the single chokepoint all MutationsInterpreter constructors delegate to, and it covers both the task-based and custom-key paths uniformly (both canUseParallelReplicasOnInitiator and canUseParallelReplicasCustomKey gate on enable_parallel_replicas). The storage read methods are now untouched. This also became necessary because the parallel_replicas_disabled field the previous approach relied on was removed from SelectQueryInfo in 274427fb0de in the meantime.

Verification (local non-replicated MergeTree, three-replica cluster):

  • Without the fix, enable_analyzer = 0: reproduces the exact SEGV from LWU with parallel replicas SEGV #104197 (rewriteSelectQueryDatabaseAndTableWithAlias on the table-less synthetic SELECT).
  • With the fix: the UPDATE succeeds and the server stays alive.

I also added enable_analyzer = 0 to the regression test so it deterministically exercises the legacy path that crashes, rather than depending on randomized settings.

(@www-dev-1 — the request to set your GitHub email on the commits is still on the original commit; that one needs to be done from your side.)

Comment thread tests/queries/0_stateless/04207_lwu_parallel_replicas.sql
alexey-milovidov and others added 2 commits June 9, 2026 14:34
`StorageMergeTree::read` only enters the legacy parallel-replica path
that crashed when `canUseTaskBasedParallelReplicas` holds, which requires
`automatic_parallel_replicas_mode == 0`. `clickhouse-test` randomizes
`automatic_parallel_replicas_mode` to `2` in some runs, which would bypass
the path under test and let the regression test pass without the fix.
Pin it to `0` so the test deterministically exercises the legacy path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@alexey-milovidov

Copy link
Copy Markdown
Member

Merged current master and addressed the remaining review thread by pinning automatic_parallel_replicas_mode = 0 in the regression test (7911ab39259), so it deterministically exercises the legacy parallel-replica path.

The sole CI red — AST fuzzer (amd_debug): Logical error: Column identifier A is already registered (STID: 4697-4326) — is unrelated to this change. The fuzzer crash is an analyzer bug in subquery set building (buildSubqueryPlansForSetsAndAddPlannerGlobalPlannerContext::createColumnIdentifier) triggered by fuzzed ALTER ... UPDATE ... if(in(t0__fuzz_21, (SELECT DISTINCT ...))) queries. It has nothing to do with parallel replicas — this PR's only source change disables enable_parallel_replicas in the mutation context. It is tracked by the open issue #106649 (and historically by the closed #99931, exact same STID).

@groeneai, please investigate the failure https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=104255&sha=a171461d9fa04f80eee52bf67ca38389bebd3e16&name_0=PR&name_1=AST%20fuzzer%20%28amd_debug%29 and provide a fix in a separate PR. If a fix for #106649 is already in progress, link it here.

@groeneai

groeneai commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Linking the in-flight fix: PR #106025 addresses STID 4697-4326 / Issue #106649 (Logical error: Column identifier ... is already registered). It is a chronic trunk bug in the analyzer's subquery set planning (buildSubqueryPlansForSetsAndAdd -> Planner -> GlobalPlannerContext::createColumnIdentifier), unrelated to the parallel-replicas mutation context disabled in this PR. PR #106025 is currently OPEN with CHANGES_REQUESTED; PR #106414 (sibling, parallel attempt for the same root cause) is also in flight. Neither failure here is caused by this PR's changes.

@clickhouse-gh

clickhouse-gh Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

LLVM Coverage Report

Metric Baseline Current Δ
Lines 84.60% 84.60% +0.00%
Functions 92.30% 92.30% +0.00%
Branches 77.30% 77.20% -0.10%

Changed lines: Changed C/C++ lines covered by tests: 5/5 (100.00%) | Lost baseline coverage: none · Uncovered code

Full report · Diff report

@LambertArm LambertArm closed this by deleting the head repository Jun 11, 2026
@devcrafter

Copy link
Copy Markdown
Member

@groeneai could you create new PR with the same fix? I can't simply reopen PR since @LambertArm has deleted his repository.

@groeneai

Copy link
Copy Markdown
Contributor

alexey-milovidov pushed a commit to hinata34/ClickHouse that referenced this pull request Jun 13, 2026
Lightweight UPDATE on a non-replicated MergeTree table with legacy
parallel replicas enabled crashed the server with a SEGV (issue ClickHouse#104197).

Mutation source reads build a synthetic SELECT without a table
expression. When the legacy parallel-replica path runs for such a query,
ClusterProxy::rewriteSelectQuery constructs a DatabaseAndTableWithAlias
from a missing table expression and dereferences a null AST pointer
(DatabaseAndTableWithAlias.cpp:60), so the server crashes inside
MutationsInterpreter::Source::read -> StorageMergeTree::read ->
executeQueryWithParallelReplicas.

Disable parallel replicas directly on the per-mutation context in the
MutationsInterpreter constructor, which is the single chokepoint all
constructors delegate to. Setting enable_parallel_replicas to 0 there
covers both the task-based and custom-key paths uniformly, and leaves the
storage read methods untouched.

Adds a regression test pinning enable_analyzer = 0 (the crash is on the
legacy non-analyzer path) and automatic_parallel_replicas_mode = 0 (so
canUseTaskBasedParallelReplicas enters that path; clickhouse-test
otherwise randomizes it and could bypass the code under test).

Recreates PR ClickHouse#104255 by @LambertArm (approved by @devcrafter); the
original author deleted their repository, so it could not be reopened.

Closes ClickHouse#104197.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

can be tested Allows running workflows for external contributors pr-critical-bugfix pr-must-backport Pull request should be backported intentionally. Use this label with great care!

Projects

None yet

Development

Successfully merging this pull request may close these issues.

LWU with parallel replicas SEGV

6 participants