Skip mark file read for parts with zero granules#106675
Conversation
`MergeTreeMarksLoader::loadMarksImpl` opens the marks file unconditionally and verifies that the underlying reader is at EOF after the requested number of marks has been consumed. For a part with zero granules (`marks_count == 0`) the for-loop runs 0 iterations and the EOF check fires immediately. If the marks file on disk happens to contain any bytes, the check throws `Too many marks in file <path>, marks expected 0 (bytes size 0)` during table ATTACH or `SYSTEM PREWARM MARK CACHE`, which prevents the server from starting up. This shape was observed in stress tests where a `text(tokenizer = array())` skip-index part on a deeply-mutated empty result (`all_1_7_2_8` with zero rows) ended up with a non-empty `skp_idx_idx.cmrk4` file. Independent hits on master, PR ClickHouse#103961 and PR ClickHouse#106621 within ~17 hours pointed to a trunk-side issue rather than a PR regression. Return an empty marks structure when `marks_count == 0` without opening the file. The check is moved before the existing `file_size == 0` validation so the empty-marks-count path is fully decoupled from file content. The pre-existing `file_size == 0` check is preserved with the now-redundant `marks_count != 0` predicate dropped: the new early-return makes that branch unreachable when `marks_count` is zero. Adds a regression test that creates an empty part with a `text` skip-index and exercises `SYSTEM PREWARM MARK CACHE`. Tracked by ClickHouse#106621 (replied). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pre-PR validation gate
CIDB last 30 days for
Session id: cron:clickhouse-ci-task-worker:20260607-110600 |
|
cc @CurtizJ @KochetovNicolai - could you review this? It returns early from |
|
Workflow [PR], commit [d550776] Summary: ❌
AI ReviewSummaryThis PR makes Final VerdictStatus: ✅ Approve |
The previous .sql variant used `OPTIMIZE FINAL + ALTER DELETE WHERE 1` to produce a part with `marks_count == 0`, but `MergeTreeData::createEmptyPart` emits a zero-byte `skp_idx_idx.cmrk4` for such parts and the old `loadMarksImpl` already short-circuited that case via `file_size == 0 && marks_count != 0` (false because `marks_count == 0`). The test therefore passed both with and without the fix. This rewrite uses a `.sh` test that drops `compress_marks` to produce uncompressed `mrk4` files (so the size validation at the top of `loadMarksImpl` is the trigger), then deterministically: 1. Builds a part with `marks_count == 0` (`OPTIMIZE FINAL` then `ALTER ... DELETE WHERE 1`). 2. `DETACH TABLE`, appends one byte to `skp_idx_idx.mrk4`, removes `checksums.txt` so re-attach recomputes them from disk. 3. `ATTACH TABLE`. On master HEAD (no fix), step 3 throws `Bad size of marks file ...skp_idx_idx.mrk4: 1, must be: 0` (`CORRUPTED_DATA`, code 246). With the fix, `loadMarksImpl` returns immediately on `marks_count == 0` and `ATTACH TABLE` succeeds. The compressed-marks (`cmrk4`) variant from the original CIDB report reaches the same fix via the same early-return; we exercise the uncompressed path here because it is reproducible deterministically with a single byte append (the compressed path needs a synthetic valid LZ4 frame, which is a strictly weaker test of the same code). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pre-PR validation gate
The compressed-marks variant from the original CIDB report reaches the same fix via the same early-return; we exercise the uncompressed path in the test because it is reproducible deterministically with a single byte append (the compressed path would need a synthetic valid LZ4 frame, which is a strictly weaker test of the same code). Session id: cron:clickhouse-ci-task-worker:20260607-194300 |
The previous version of `04323_text_index_marks_empty_part.sh` worked by detaching the table, appending a byte to `skp_idx_idx.mrk4`, removing `checksums.txt`, and re-attaching. On the Fast test runner that path emitted a Warning to stderr (`Checksums for part ... not found. Will calculate them from data on disk.`), and the part loader also failed an unrelated `columns_substreams.txt` precondition before reaching the marks loader at all, so the test no longer exercised the fix it was supposed to demonstrate. The simpler path is enough. After `OPTIMIZE FINAL` and `ALTER ... DELETE WHERE 1` the active part already has `marks_count == 0` and a zero-byte `data.cmrk4` (or `data.mrk4` with `compress_marks = 0`). Without the `marks_count == 0` early-return in `loadMarksImpl` the zero-byte file fails the `file_size == 0` validation immediately on `SYSTEM PREWARM MARK CACHE` with `Empty marks file ...: 0, must be: 0` (`CORRUPTED_DATA`, code 246). With the fix the early-return skips that validation and prewarm succeeds. Verified locally in both directions: - without the `marks_count == 0` short-circuit reverted from `MergeTreeMarksLoader::loadMarksImpl`, `SYSTEM PREWARM MARK CACHE` throws `Empty marks file '.../data.cmrk4': 0, must be: 0` - with the fix restored, the test passes (50/50 with `--no-random-settings`) The detach/append/checksum dance and the verbose block comment are gone; the same comment about why the test exists belongs in the commit message, not in the file. Addresses clickhouse-gh[bot]'s review concern that the prior shape did not actually exercise the regression. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pre-PR validation gate (commit
|
| # | Question | Answer |
|---|---|---|
| a | Deterministic repro? | Yes. The local repro is just CREATE … TYPE text(tokenizer = array()) GRANULARITY 100000000 ENGINE = MergeTree ORDER BY tuple() SETTINGS prewarm_mark_cache = true, two tiny INSERTs, OPTIMIZE FINAL, ALTER … DELETE WHERE 1 SETTINGS mutations_sync = 2, then SYSTEM DROP MARK CACHE; SYSTEM PREWARM MARK CACHE. With the marks_count == 0 early-return reverted out of MergeTreeMarksLoader::loadMarksImpl, the prewarm step throws DB::Exception: Empty marks file '.../data.cmrk4': 0, must be: 0. (CORRUPTED_DATA) (code 246) on every run. Direct rebuild + restart, no randomization. |
| b | Root cause explained? | Yes. After ALTER … DELETE WHERE 1 the active part is the empty mutated part with marks_count == 0 and a zero-byte data.cmrk4 (or data.mrk4 with compress_marks = 0). The pre-fix loadMarksImpl opens the file unconditionally and runs the file_size == 0 validation, which throws CORRUPTED_DATA (the same root cause as the original Too many marks in file …cmrk4 shape from CI — same code path, just a different post-EOF check on top). The fix returns early on marks_count == 0 before either validation runs. |
| c | Fix matches root cause? | Yes. Commit 873efbe89db4 short-circuits loadMarksImpl on marks_count == 0 before any file is opened. That is exactly what makes the data.cmrk4 zero-byte case stop throwing. No bounds widening, no no-random-* blanket. |
| d | Test intent preserved / new tests added? | Yes. The previous shape of 04323_text_index_marks_empty_part did not actually exercise the fix (clickhouse-gh[bot] flagged this in 4547429947): the ALTER … DELETE WHERE 1 empty-part path produced a zero-byte mark file that the pre-existing file_size == 0 short-circuit already tolerated for the case the prior test set up, AND the post-detach ATTACH was failing on an unrelated columns_substreams.txt precondition before ever reaching the marks loader (and emitting a Checksums for part ... not found Warning that broke Fast test). The new shape removes the detach/append/checksum dance and exercises loadMarksImpl directly via SYSTEM PREWARM MARK CACHE. The test now both demonstrates the regression on master HEAD without the fix and silently passes with the fix. |
| e | Both directions demonstrated? | Yes. Locally rebuilt with 873efbe89db4 reverted out of MergeTreeMarksLoader.cpp: SYSTEM PREWARM MARK CACHE t_text_idx_empty throws CORRUPTED_DATA: Empty marks file '.../data.cmrk4': 0, must be: 0. Restored the fix and rebuilt: clickhouse-test --no-random-settings --test-runs 50 is 50/50 PASS, no stderr emitted, reference matches 0\t0\n0\n0. |
| f | Fix is general, not a narrow patch? | Yes. loadMarksImpl is the single shared code path used by every mark-loader caller (table prewarm, ATTACH-time index marks, IndexMarkCache prewarm, async-load future paths) — both compressed (*.cmrk4) and uncompressed (*.mrk4) variants. The early-return is independent of mark_type.compressed / mark_type.adaptive, so it covers the original CI shape (Too many marks in file …cmrk4 from CompressedReadBufferFromFile::eof() after the loop) and the trivial empty-file shape (Empty marks file … from the file_size == 0 check) in one place. No sibling code paths to fix. The bad state is legitimately possible — marks_count == 0 is a normal post-ALTER DELETE WHERE 1 and post-merge empty-part state — so an early return is correct, not a defensive guard against an upstream bug. |
Session id: cron:clickhouse-ci-task-worker:20260607-210400
The previous simplification (commit deca0bd) reduced 04323 to a plain SYSTEM PREWARM MARK CACHE on a mutated empty part. Reviewer clickhouse-gh[bot] correctly pointed out this does not prove the fix: In the PR base (da698ac) MergeTreeMarksLoader::loadMarksImpl guards the throw with `if (file_size == 0 && marks_count != 0)` -- for a part with `marks_count == 0` and a zero-byte mark file the predicate is false and the loader silently returns. Any test that only creates the empty-part state therefore passes both with and without this PR. The CI failure shape requires `marks_count == 0` AND a non-empty mark file on disk. This commit restores the deterministic byte-append fixture (the shape from the earlier 0546e1b test) but addresses the Fast-test stderr-empty regression that broke that earlier attempt. The ATTACH after the manual mark-file mutation emits a Warning to stderr (`Checksums for part ... not found. Will calculate them from data on disk.`); we suppress it with --send_logs_level=fatal, matching the precedent in 04230_iceberg_optimize_metadata_not_initialized_104711.sh. Verification on the actual PR base da698ac: Without fix: ATTACH throws `Bad size of marks file '...mrk4': 1, must be: 0` (CORRUPTED_DATA, code 246) -- test FAILS. With fix: `loadMarksImpl` early-returns on `marks_count == 0`, ATTACH succeeds, prewarm + queries return 0 -- 10/10 PASS under random settings. The fix in this PR (in `MergeTreeMarksLoader::loadMarksImpl`) is unchanged. Related: ClickHouse#106675 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pre-PR validation gate
Session id: cron:clickhouse-ci-task-worker:20260608-041400 |
CI Status — covered SHA
|
| Check | Test | Disposition |
|---|---|---|
Stateless tests (amd_debug, parallel) |
Logical error: Virtual row boundary violated in MergingSortedAlgorithm (STID 2651-3359) |
chronic master regression, NOT PR-caused. Cross-PR query (30d): hits master + 25+ unrelated PRs (#105386, #106215, #104965, #67954, #79393, #99675, etc.). Tracked in umbrella task with fix PR #106664 in flight (vdimir territory, on the heap-position correctness path). The merging path is unrelated to MergeTreeMarksLoader::loadMarksImpl. |
Stateless tests (amd_debug, parallel) |
Server died |
cascade of the above — debug assertion in MergingSortedAlgorithm::consume aborts the server, which terminates whatever test was running. |
Stateless tests (amd_debug, parallel) |
02346_text_index_replacingmergetree1 ERROR |
collateral damage — happened to be the test in flight when the server aborted from the STID 2651-3359 assertion. Cross-PR check: only 1 prior hit (#103706 on 2026-05-20) — not a chronic flake of this test, but a victim of the upstream crash. |
PR diff only changes MergeTreeMarksLoader.cpp + adds a regression test. None of these touch the MergingSortedAlgorithm / virtual-row machinery.
Session: cron:our-pr-ci-monitor:20260608-083000
LLVM Coverage Report
Changed lines: Changed C/C++ lines covered by tests: 11/13 (84.62%) | Lost baseline coverage: none · Uncovered code |
168b5ad

Fixes a stress-test regression where the server cannot start after attaching a
MergeTreetable that has atextskip-index part with zero granules:Too many marks in file ...skp_idx_idx.cmrk4, marks expected 0 (bytes size 0).MergeTreeMarksLoader::loadMarksImplopens the marks file unconditionally and verifies that the underlying reader is at EOF after the requested number of marks has been consumed. Formarks_count == 0the for-loop runs zero iterations and the EOF check fires immediately. If the marks file has any bytes on disk (e.g. trailing bytes from a stress-killed merge or a flushed compression header), the check throws during ATTACH orSYSTEM PREWARM MARK CACHE.A part with zero granules has nothing to load, so this fix returns an empty marks structure without opening the file. The pre-existing
file_size == 0check is preserved with the now-redundantmarks_count != 0predicate dropped.CIDB cross-PR evidence over 30 days for
Too many marks .. text(tokenizer:Three independent hits on three unrelated branches over ~17 hours - trunk-side issue, not caused by any open PR.
Tracked by #106621.
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):
Fix server failing to start with
Too many marks in file ...skp_idx_idx.cmrk4, marks expected 0 (bytes size 0)when aMergeTreetable has a skip-index part with zero granules and a non-empty marks file on disk.Documentation entry for user-facing changes
Version info
26.6.1.1035