iframe-proxy

groeneai · 2026-06-26T04:18:19Z

Changelog category (leave one):

Bug Fix (user-visible misbehavior in an official stable release)

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Fix a logical error (Equal values are not contiguous within the range assumed to be sorted) when running DISTINCT over a STREAM read (SELECT DISTINCT ... FROM table STREAM). Read-in-order optimizations are no longer applied to STREAM reads, which produce rows in commit order rather than sorting-key order.

Description

A STREAM read (FROM ... STREAM, enable_streaming_queries = 1) returns parts in commit order via MergeTreeCommitOrderSequentialSource, not in sorting-key order, so its output is not globally sorted by the table sorting key.

checkSupportedReadingStep in optimizeReadInOrder.cpp still accepted such a ReadFromMergeTree (the sorting key is non-empty), so the query plan optimizer requested read-in-order on it. The read then advertised the sorting-key order (input_order_info -> getSortDescription()), and order-dependent transforms were applied on top of unsorted data.

For DISTINCT in order this feeds unsorted data to DistinctSortedStreamTransform, which aborts the server in debug/sanitizer builds:

Logical error: 'Equal values are not contiguous within the range assumed to be sorted'.
  src/Columns/findEqualRangeEndAssumeSorted.h:85
  src/Columns/ColumnString.cpp:816  DB::ColumnString::getEqualRangeEndAssumeSorted
  src/Core/SortCursor.h:776
  src/Processors/Transforms/DistinctSortedStreamTransform.cpp:184  DB::DistinctSortedStreamTransform::transform

The same phantom order silently affects aggregation-in-order and LIMIT BY-in-order on STREAM reads.

Fix: reject streaming reads in checkSupportedReadingStep, so none of the in-order optimizations (read-in-order, aggregation-in-order, distinct-in-order, limit-by-in-order) are applied to a STREAM read. STREAM is only valid on StorageMergeTree/StorageReplicatedMergeTree (both produce ReadFromMergeTree), so the single guard covers all streaming reads; the ReadFromMerge/ReadFromObjectStorageStep branches cannot be streaming.

Reproducer (crashes before the fix, returns correctly after):

CREATE TABLE t (a String, b UInt64) ENGINE = MergeTree ORDER BY a;
INSERT INTO t SELECT toString(number % 100), number FROM numbers(5000);
INSERT INTO t SELECT toString(number % 100), number FROM numbers(5000);
INSERT INTO t SELECT toString(number % 100), number FROM numbers(5000);
SELECT count() FROM (SELECT DISTINCT a FROM t STREAM LIMIT 50) SETTINGS enable_streaming_queries = 1;

Found by the AST fuzzer on the Stress test (arm_asan_ubsan, s3) check (STID 2508-4156, query SELECT count() FROM (SELECT DISTINCT * FROM t_streaming_test STREAM LIMIT 10)):
https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=108554&sha=bdbb85719cfc04f07cc5084b3e4a932df073b419&name_0=PR&name_1=Stress%20test%20%28arm_asan_ubsan%2C%20s3%29

Version info

Merged into: 26.7.1.273

A STREAM read returns parts in commit order, not sorting-key order, so its output is not globally sorted by the sorting key even though the key is non-empty. checkSupportedReadingStep still accepted such a ReadFromMergeTree, so the query plan optimizer requested read-in-order on it. The read then advertised the sorting-key order (via input_order_info) and fed unsorted data to order-dependent transforms. For DISTINCT in order this aborts the server in debug/sanitizer builds with 'Equal values are not contiguous within the range assumed to be sorted' (DistinctSortedStreamTransform). The same phantom order silently affects aggregation-in-order and LIMIT BY-in-order. Reject streaming reads in checkSupportedReadingStep so none of the in-order optimizations (read-in-order, aggregation-in-order, distinct-in-order, limit-by-in-order) are applied to a STREAM read. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

groeneai · 2026-06-26T04:18:49Z

Pre-PR validation gate (click to expand)

#	Question	Answer
a	Deterministic repro?	Yes. `CREATE TABLE t (a String, b UInt64) ENGINE = MergeTree ORDER BY a`, three `INSERT ... SELECT toString(number % 100), number FROM numbers(5000)`, then `SELECT count() FROM (SELECT DISTINCT a FROM t STREAM LIMIT 50) SETTINGS enable_streaming_queries = 1` aborts the server every time on the unfixed binary.
b	Root cause explained?	A `STREAM` read returns parts in commit order (`MergeTreeCommitOrderSequentialSource`), not sorting-key order, so its output is not globally sorted by the key. `checkSupportedReadingStep` accepted it (key non-empty), the optimizer requested read-in-order, the read advertised the sorting-key order via `input_order_info`, and `DistinctSortedStreamTransform` then ran `getEqualRangeEndAssumeSorted` on unsorted data, tripping the contiguity assertion.
c	Fix matches root cause?	Yes. The fix rejects streaming reads at the source of the phantom order (`checkSupportedReadingStep`), so no in-order optimization is ever applied to a `STREAM` read. It does not widen/silence the assertion.
d	Test intent preserved / new tests added?	New regression test `04411_streaming_distinct_in_order` added (DISTINCT a / DISTINCT * over STREAM). No existing test weakened.
e	Both directions demonstrated?	Yes. Unfixed binary: the repro and the new test abort with `Equal values are not contiguous`. Fixed binary: query returns 50, new test passes 20/20 with randomization; pipeline switches from `DistinctSortedStreamTransform` to plain `DistinctTransform`.
f	Fix is general across code paths?	Yes. All four in-order optimizations (read-in-order, aggregation-in-order, distinct-in-order, limit-by-in-order) funnel through `findReadingStep` -> `checkSupportedReadingStep`, so the single guard covers them all, not just DISTINCT.
g	Fix generalizes across inputs?	`STREAM` is only valid on `StorageMergeTree`/`StorageReplicatedMergeTree` (both produce `ReadFromMergeTree`); every other storage returns `supportsStreaming() == false`. So the `ReadFromMerge`/`ReadFromObjectStorageStep` branches cannot be streaming, and the one `ReadFromMergeTree` guard covers the full input space. Verified the crash family covers `DISTINCT a` and `DISTINCT *`.
h	Backward compatible?	Yes. `STREAM` (`enable_streaming_queries`) is an experimental feature; this only disables an unsound optimization for it. No setting default, on-disk/wire format, or existing-data validation changes. No `SettingsChangesHistory.cpp` change needed.
i	Invariants and contracts preserved?	Yes. `DistinctSortedStreamTransform`/`getEqualRangeEndAssumeSorted` require globally-sorted input; the fix stops feeding them unsorted streaming data instead of weakening that contract. Non-streaming DISTINCT-in-order, aggregation-in-order and read-in-order still fire (verified via `EXPLAIN PIPELINE` and the dataset-free tests `02317_distinct_in_order_optimization{,_explain}`, `02481_aggregation_in_order_plan`, `00940_order_by_read_in_order_query_plan`, all OK).

Session id: cron:clickhouse-worker-slot-0:20260626-034100

groeneai · 2026-06-26T04:19:33Z

cc @KochetovNicolai @novikd — could you review this? A STREAM read returns parts in commit order (not sorting-key order), but checkSupportedReadingStep still accepted it for read-in-order, so the read advertised the sorting-key order and fed unsorted data to DISTINCT-in-order (DistinctSortedStreamTransform), aborting with 'Equal values are not contiguous'. The fix excludes streaming reads from all in-order optimizations at checkSupportedReadingStep.

clickhouse-gh · 2026-06-26T13:25:04Z

Workflow [PR], commit [9732127]

Summary: ✅

Performance Comparison: Performance dashboard

AI Review

Summary

This PR disables in-order read optimizations for STREAM reads by rejecting streaming ReadFromMergeTree steps before read-in-order, aggregation-in-order, distinct-in-order, and limit-by-in-order can request input_order_info. The current code matches the stated root cause: valid STREAM queries only reach ReadFromMergeTree, ReadFromMergeTree advertises storage order only after requestReadingInOrder, and the old window-ordering helper cannot process STREAM because the old analyzer rejects it before planning. The previous inline test concern has been addressed by pinning the optimizer settings.

Final Verdict

Status: ✅ Approve

STREAM reads are Linux-only (the server raises SUPPORT_IS_DISABLED for hasStream() under #ifndef OS_LINUX in QueryAnalyzer). The test was running on the macOS Fast test (arm_darwin) and failing with SUPPORT_IS_DISABLED. no-darwin skips it only on macOS while keeping full coverage on every Linux job, including the Linux Fast test where the STREAM crash repro is valid. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

groeneai · 2026-06-26T16:29:04Z

Pushed 12998180cac: tag 04411_streaming_distinct_in_order as no-darwin.

The previous head failed Fast test (arm_darwin) with Code: 344. Streaming requests are supported only on Linux. (SUPPORT_IS_DISABLED) — the macOS Fast test ran the new test, but STREAM reads are Linux-only (#ifndef OS_LINUX guard in QueryAnalyzer). no-darwin skips it only on macOS; it still runs (and the crash repro is still exercised) on every Linux job, including the Linux Fast test. Verified 5/5 OK on Linux with randomization; the tag is a no-op there.

Session id: cron:clickhouse-worker-slot-1:20260626-162000

The stateless runner randomizes optimizer settings. The regression depends on the query-plan DISTINCT-in-order pass being enabled; if optimize_distinct_in_order or query_plan_enable_optimizations is randomized off, the query uses the generic distinct path and the unfixed build never reaches the bad input_order_info, so the test could pass without the fix. Pin enable_analyzer, query_plan_enable_optimizations and optimize_distinct_in_order so the test deterministically exercises the path the PR guards. Verified: with the pins the test crashes on the pre-fix binary and passes on the fixed binary; with the pins removed the pre-fix binary no longer crashes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

groeneai · 2026-06-26T20:43:03Z

Addressed the review on 04411_streaming_distinct_in_order.sql: pinned the three trigger settings so the test deterministically exercises the query-plan DISTINCT-in-order pass this PR guards, instead of only the default configuration.

SET enable_analyzer = 1;
SET query_plan_enable_optimizations = 1;
SET optimize_distinct_in_order = 1;

Re-verified both directions locally (debug build):

Fixed binary, pins on: returns 50/50, no crash.
Pre-fix binary (guard reverted), pins on: aborts with the original Equal values are not contiguous within the range assumed to be sorted in DistinctSortedStreamTransform. So the pinned test catches the regression.
Pre-fix binary, optimize_distinct_in_order=0 or query_plan_enable_optimizations=0: returns 50/50, no crash. This confirms those settings are exactly what route the query into the bad path, which is why the test would have silently passed without the fix when CI randomized them off.
Runner with randomization on (fixed binary): 10/10 OK.

Pushed in 1e00156.

groeneai · 2026-06-27T00:50:07Z

CI finish ledger - `1e00156`

CI fully finished (Finish Workflow and Style check passed; no checks queued or in progress; >20 min since the last check completed). Every failure below has an owner.

Check / test	Reason	Owner / fixing PR
Integration tests (arm_binary, distributed plan, 2/4) / test_keeper_disks::test_snapshots_with_disks	flaky (chronic; 18+ unrelated PRs + master in 30d; a Keeper integration test, unrelated to this PR's query-optimizer change in optimizeReadInOrder.cpp)	#108513 (ours, open)
CH Inc sync	private sync	CH Inc sync (private, not actionable)

Session id: cron:our-pr-ci-monitor:20260627-003000

alexey-milovidov · 2026-06-29T03:45:15Z

@groeneai, as #108513 was merged, update the branch here.

@alexey-milovidov

Pick up merged PR ClickHouse#108513 (fix flaky test_keeper_disks::test_snapshots_with_disks) per @alexey-milovidov request on PR ClickHouse#108568.

groeneai · 2026-06-29T04:24:40Z

@alexey-milovidov done. Merged current master into the branch (head 9732127), so it now includes the merged #108513. CI is re-running.

clickhouse-gh · 2026-06-29T08:08:58Z

LLVM Coverage Report

Metric	Baseline	Current	Δ
Lines	85.40%	85.50%	+0.10%
Functions	92.60%	92.60%	+0.00%
Branches	77.70%	77.70%	+0.00%

Changed lines: Changed C/C++ lines covered by tests: 4/5 (80.00%) | Lost baseline coverage: none · Uncovered code

Full report · Diff report

groeneai · 2026-06-29T09:32:17Z

CI finish ledger — `9732127`

CI fully finished on this head (Finish Workflow + Mergeable Check pass; no checks queued/in_progress; >20-min ingestion buffer). After merging master to pick up the now-merged #108513, every check is green and the previously-noted flaky test_keeper_disks::test_snapshots_with_disks ledger line is cleared.

Session id: cron:our-pr-ci-monitor:20260629-090000

groeneai mentioned this pull request Jun 26, 2026

Fix logical error in correlated subquery decorrelation with set operations #108554

Open

nikitamikhaylov added the can be tested Allows running workflows for external contributors label Jun 26, 2026

clickhouse-gh Bot added the pr-bugfix Pull request with bugfix, not backported by default label Jun 26, 2026

clickhouse-gh Bot reviewed Jun 26, 2026

View reviewed changes

Comment thread tests/queries/0_stateless/04411_streaming_distinct_in_order.sql

groeneai mentioned this pull request Jun 26, 2026

Fix text index direct read losing NULL semantics for Nullable columns #107078

Open

groeneai mentioned this pull request Jun 26, 2026

Use a monotonic clock in scheduler EventQueue #108647

Merged

groeneai mentioned this pull request Jun 29, 2026

Speed up ZSTD decompression on AArch64 (small-offset overlapping copies) #108049

Merged

Merge branch 'master' into fix-streaming-distinct-in-order-crash

9732127

Pick up merged PR ClickHouse#108513 (fix flaky test_keeper_disks::test_snapshots_with_disks) per @alexey-milovidov request on PR ClickHouse#108568.

alexey-milovidov approved these changes Jun 29, 2026

View reviewed changes

alexey-milovidov requested a review from Michicosun June 29, 2026 23:34

alexey-milovidov self-assigned this Jun 29, 2026

alexey-milovidov added this pull request to the merge queue Jun 29, 2026

Merged via the queue into ClickHouse:master with commit fac0764 Jun 29, 2026
174 checks passed

robot-clickhouse added the pr-synced-to-cloud The PR is synced to the cloud repo label Jun 30, 2026

groeneai mentioned this pull request Jun 30, 2026

Add GradualResizeProcessor to limit effective parallelism for GROUP BY on small data volumes #99495

Draft

Sunbelt Computer Software

PL/B Language Development and Support

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Do not apply read-in-order optimizations to STREAM reads#108568

Do not apply read-in-order optimizations to STREAM reads#108568
alexey-milovidov merged 4 commits into
ClickHouse:masterfrom
groeneai:fix-streaming-distinct-in-order-crash

groeneai commented Jun 26, 2026 •

edited by robot-ch-test-poll4

Loading

Uh oh!

groeneai commented Jun 26, 2026

Uh oh!

groeneai commented Jun 26, 2026

Uh oh!

clickhouse-gh Bot commented Jun 26, 2026 •

edited

Loading

Uh oh!

Uh oh!

groeneai commented Jun 26, 2026

Uh oh!

groeneai commented Jun 26, 2026

Uh oh!

groeneai commented Jun 27, 2026

Uh oh!

alexey-milovidov commented Jun 29, 2026

Uh oh!

groeneai commented Jun 29, 2026

Uh oh!

clickhouse-gh Bot commented Jun 29, 2026

Uh oh!

groeneai commented Jun 29, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Sunbelt Computer Software

PL/B Language Development and Support

Uh oh!

Conversation

groeneai commented Jun 26, 2026 • edited by robot-ch-test-poll4 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Description

Version info

Uh oh!

groeneai commented Jun 26, 2026

Uh oh!

groeneai commented Jun 26, 2026

Uh oh!

clickhouse-gh Bot commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

AI Review

Summary

Final Verdict

Uh oh!

Uh oh!

groeneai commented Jun 26, 2026

Uh oh!

groeneai commented Jun 26, 2026

Uh oh!

groeneai commented Jun 27, 2026

CI finish ledger - 1e00156

Uh oh!

alexey-milovidov commented Jun 29, 2026

Uh oh!

groeneai commented Jun 29, 2026

Uh oh!

clickhouse-gh Bot commented Jun 29, 2026

LLVM Coverage Report

Uh oh!

groeneai commented Jun 29, 2026

CI finish ledger — 9732127

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

groeneai commented Jun 26, 2026 •

edited by robot-ch-test-poll4

Loading

clickhouse-gh Bot commented Jun 26, 2026 •

edited

Loading

CI finish ledger - `1e00156`

CI finish ledger — `9732127`