Add debug virtual row assertions to MergingSortedAlgorithm by vdimir · Pull Request #106565 · ClickHouse/ClickHouse · GitHub
Skip to content

Add debug virtual row assertions to MergingSortedAlgorithm#106565

Merged
alexey-milovidov merged 1 commit into
masterfrom
vdimir/virtual_row_partial_read_assert
Jun 6, 2026
Merged

Add debug virtual row assertions to MergingSortedAlgorithm#106565
alexey-milovidov merged 1 commit into
masterfrom
vdimir/virtual_row_partial_read_assert

Conversation

@vdimir

@vdimir vdimir commented Jun 5, 2026

Copy link
Copy Markdown
Member

Changelog category (leave one):

  • Not for changelog (changelog entry is not required)

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

  • Add debug virtual row assertions to MergingSortedAlgorithm

Followup for #106429 submitting separately so it's easier to backport prev PR with actual fix

@clickhouse-gh

clickhouse-gh Bot commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

@clickhouse-gh clickhouse-gh Bot added the pr-not-for-changelog This PR should not be mentioned in the changelog label Jun 5, 2026
@clickhouse-gh

clickhouse-gh Bot commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

LLVM Coverage Report

Metric Baseline Current Δ
Lines 84.40% 84.40% +0.00%
Functions 92.30% 92.30% +0.00%
Branches 77.00% 77.00% +0.00%

Changed lines: Changed C/C++ lines covered by tests: 22/43 (51.16%) | Lost baseline coverage: none · Uncovered code

Full report · Diff report

@alexey-milovidov alexey-milovidov self-assigned this Jun 6, 2026
@alexey-milovidov alexey-milovidov added this pull request to the merge queue Jun 6, 2026
Merged via the queue into master with commit 5677a25 Jun 6, 2026
166 checks passed
@alexey-milovidov alexey-milovidov deleted the vdimir/virtual_row_partial_read_assert branch June 6, 2026 05:48
@robot-ch-test-poll robot-ch-test-poll added the pr-synced-to-cloud The PR is synced to the cloud repo label Jun 6, 2026
groeneai added a commit to groeneai/ClickHouse that referenced this pull request Jun 7, 2026
- src/.../MergingSortedAlgorithm.cpp: collapse the 6-line gate explanation
  next to `if (apply_virtual_row_conversions)` into a one-line invariant
  note. The full root-cause explanation (Union of read-in-order subtrees,
  per-source conversions, default-zero fallback in setVirtualRow) lives in
  the PR description and the original commit message of 28c0151.
- tests/.../04323_virtual_row_two_shards_pk_alias.sql: collapse the
  11-line header comment into a one-line "Regression test for STID
  2651-3359 (debug assertion added in ClickHouse#106565)" - test name and the SET
  list make the purpose self-evident, full context lives in ClickHouse#106664.

Also corrects the outdated test-file PR reference noted by clickhouse-gh
inline (ClickHouse#106565 introduced the assertion; the test reproduces the
assertion path covered by this PR, not ClickHouse#106654 which is unrelated).

No code-behavior changes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
grantholly-clickhouse pushed a commit to grantholly-clickhouse/ClickHouse that referenced this pull request Jun 8, 2026
The debug assertions added in ClickHouse#106565 fired across many unrelated PRs
(STID 2651-3359), including master, with the message
`Virtual row boundary violated in MergingSortedAlgorithm ... the virtual row
announced Decimal64_'0.000000000' but the source then produced ...`.

Trigger query (`tests/queries/0_stateless/04229_distributed_cte_timestamp_rename_103508.sql`
under random `read_in_order_use_virtual_row=1`):

    WITH A AS (
        SELECT * FROM dist_t_two_shards
        WHERE timestamp >= toDateTime64('2023-01-01', 9)
          AND timestamp <  toDateTime64('2025-01-01', 9)
    )
    SELECT timestamp, id FROM A ORDER BY timestamp DESC LIMIT 10
    SETTINGS distributed_product_mode = 'allow',
             optimize_read_in_order = 1,
             read_in_order_use_virtual_row = 1;

The pipeline that hits the assertion is

    MergingSortedTransform 2 -> 1
      BufferChunks * 2
        Union
          ExpressionTransform * 2 -> ReverseTransform -> VirtualRowTransform -> MergeTreeSelect (shard 1)
          ExpressionTransform * 2 -> ReverseTransform -> VirtualRowTransform -> MergeTreeSelect (shard 2)

The outer `MergingSortedTransform` is built by `SortingStep::mergingSorted`
from the Union branch in `optimizeReadInOrder.cpp` (the case at line 1613,
`sorting->convertToFinishSorting(*max_sort_descr, use_buffering, false);`),
which deliberately passes `apply_virtual_row_conversions = false` because no
single conversion fits all children of the Union. Each child, however, still
emits virtual-row chunks carrying its own per-source conversion in
`MergeTreeReadInfo::pk_block`, with the storage's primary-key column names
(`timestamp`).

Inside `setVirtualRow`, when `apply_virtual_row_conversions` is false the
per-source rename is skipped, so `pk_block` keeps the inner name `timestamp`
while the outer merge header has the analyzer-renamed `__table1.timestamp`.
The lookup `findByName(col.name)` fails and the column falls back to
`createColumnConstWithDefaultValue(1)` -- a Decimal64 zero. The debug
assertions in `rememberVirtualRowBoundary` / `checkVirtualRowBoundary` then
treat that zero as the announced boundary and abort once the source produces
its real first row. Outside debug builds the merge still produces correct
output because the virtual-row 1-row chunk is consumed via
`skip_last_row = true` in `mergeImpl` / `mergeBatchImpl` and never reaches
the output, so the only visible damage is the debug abort.

Gate `rememberVirtualRowBoundary` and `checkVirtualRowBoundary` on
`apply_virtual_row_conversions = true`. When the flag is false the recorded
boundary is meaningless, so simply do not compare against the next chunk.
The runtime path (`setVirtualRow` + `skip_last_row`) is unchanged.

Add `tests/queries/0_stateless/04323_virtual_row_two_shards_pk_alias.sql`
covering the two-shard union shape with `read_in_order_use_virtual_row = 1`
both ASC and DESC, and with `read_in_order_use_virtual_row_per_block = 1`.
The test fails on master with the assertion and passes with this fix.

Caused by: ClickHouse#106565

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-not-for-changelog This PR should not be mentioned in the changelog pr-synced-to-cloud The PR is synced to the cloud repo

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants