Log new RUM split pages as full images; error instead of hanging on zero pages by hari90 · Pull Request #808 · documentdb/documentdb · GitHub
Skip to content

Log new RUM split pages as full images; error instead of hanging on zero pages - #808

Open
hari90 wants to merge 2 commits into
documentdb:mainfrom
hari90:ak_hari90/fix-807
Open

hari90 wants to merge 2 commits into
documentdb:mainfrom
hari90:ak_hari90/fix-807

Conversation

@hari90

@hari90 hari90 commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Description

Fixes #807. When a RUM page that isn't the root splits, the new right page is WAL-logged as a generic delta instead of a full image. With full_page_writes = off, crash recovery then loses the page. The server reports nothing, and the index is left with downlinks and sibling links to all-zero pages. Any later write that descends into one of those pages spins forever and can't be cancelled.

Commit 1: log the new right page as a full image

Root cause: rumPlaceToPage() (the default path) and rumInsertValueOld() (used when documentdb_rum.track_incomplete_split = off) registered the page from RumNewBuffer() with GenericXLogRegisterBuffer(..., 0). A page that was just extended is all-zero on disk. During crash recovery, XLogReadBufferForRedo sees a new page, generic_redo skips the delta, and the invalid-page check doesn't run.

  • Both sites now register the page with GENERIC_XLOG_FULL_IMAGE. The root split, createPostingTree, RumInitBuffer and RumInitMetabuffer already do this for their new pages. I checked every RumNewBuffer() caller; these two were the only ones logging a delta.
  • Only the new page changes. The left page and the right-right sibling already exist, and earlier WAL rebuilds them, so they stay deltas.
  • There's no GUC, and no change to the on-disk or WAL format: it's still an XLOG_GENERIC record. WAL volume is about the same. With full_page_writes = on a never-written page already got a full-page image, and with it off a delta against zeros is already close to page size.
  • RumNewBuffer()'s comment now states that its pages must be logged as full images.
  • Indexes already damaged this way still need REINDEX.

Commit 2: raise index corruption instead of hanging

A zero page has no flags, so the descent treats it as an internal page. findChildPage() then returns block 0, the metapage, which is also non-leaf, so the loop never ends. It never checked for interrupts.

  • New rumCheckTreePage() raises ERRCODE_INDEX_CORRUPTED with a REINDEX hint for an all-zero page or the metapage. It runs under the lock, before the page's flags are read, wherever a page is reached through a tree link:
    • rumTraverseLock(), for every level of the descent
    • rumStepCore() and rumStepEntryForScans(), for sibling steps
    • the rumFindParents() walk
    • the "refind the leftmost page" loops in entryGetNextItem() and entryFindItem()
    • VACUUM's rumFindLeftMostLeafPage()
  • The check has to happen before the flags are read: on an assert-enabled build, reading a zero page's flags trips PageGetSpecialPointer's assertion.
  • rumFindLeafPage() also errors on a page of the other tree kind (entry vs. posting) and on an invalid or self-referencing downlink, which were only Asserts. It also calls CHECK_FOR_INTERRUPTS() on each level.
  • The sibling type-mismatch and deleted-sibling errors use ERRCODE_INDEX_CORRUPTED.
  • "Lost path" deliberately stays an internal error. GetWriteErrorFromErrorData() matches it and turns it into the retryable ERRCODE_INDEX_LOSTPATH.

Tests

New TAP test pg_documentdb/src/test/extended_rum_recovery_tests/t/004_rum_split_full_page_writes_off.pl:

  1. Sets full_page_writes = off, turns off bgwriter writes, checkpoints and autovacuum, and runs one collection per split path (track_incomplete_split on and off).
  2. Before the crash, checks that both indexes really split non-root pages: some internal page has 3 or more downlinks.
  3. Crashes the server with stop('immediate') and lets it recover.
  4. Checks that no index block is all-zero, that an insert completes, and that index scans match sequential scans.
  5. Zeroes the posting-tree leaf pages of one index on disk, then checks that an insert and an index scan fail promptly with "contains unexpected zero page".

Verified in the dev-container image (PG 16, assert-enabled):

Run Result
New test 004 29/29
Commit 1 alone 23/23
WAL fix reverted fails: both split paths leave zero pages after recovery
Error checks reverted fails: the backend aborts on an assertion (production builds loop forever instead)
make -C pg_documentdb check-extended-rum-recovery (001–004) pass
check-extended-rum 83/83
check-extended-rum-builtin-rmgr 83/83
check-regress 138/138 and 5/5

citus_indent is clean.

Type of Change

  • Bug fix
  • New feature
  • Documentation update
  • Refactoring / code cleanup
  • DevOps / tooling
  • Test improvement
  • Other: ___________

AI Disclosure

  • No AI tools were used substantially in this PR
  • AI tools were used to assist with this PR (please complete below)
    • Tool(s) used: Meko, Claude Code (Claude Opus 5.5) and AgentK
    • How it was used: Root-cause analysis, the design, the code changes and the TAP test, and running the test suites in the dev container, under my review.

Checklist

  • I can explain every change in this PR if asked during review
  • I have tested these changes locally
  • I have added or updated tests as appropriate
  • All commits are signed off (DCO: git commit -s)
  • I have read the CONTRIBUTING.md guidelines

🤖 Generated with Claude Code

https://claude.ai/code/session_018uo1BA8DhzNJHxezANds93

cloud.mekodata.ai://48fd42c0-7b11-4c27-a0a9-db6f5be829d7;adlc/proj/documentdb/documentdb
meko_design:design-fix-807.md;62a752e216b8f7a654d2997acffb49da93abf5067a7b5039adc331c7fd376bb7
meko_conversation_id:54978901058b4bafa64ae3b040649dff

hari90 and others added 2 commits October 21, 2026 07:22
A non-root page split (rumPlaceToPage, and rumInsertValueOld when
documentdb_rum.track_incomplete_split is off) registered the page it got
from RumNewBuffer() for generic WAL with flags 0, so it was logged as a
delta against its previous contents.  When the page was just extended it
is all-zero on disk until written out.  With full_page_writes = off, crash
recovery's XLogReadBufferForRedo sees a new page, generic_redo skips the
delta, and crash recovery doesn't run the invalid-page check, so it ends
without an error and leaves downlinks and sibling links pointing at
all-zero pages.  Later writes that descend into one of them never return
(documentdb#807).

Register the new right page with GENERIC_XLOG_FULL_IMAGE, as the root
split, createPostingTree, RumInitBuffer and RumInitMetabuffer already do
for their new pages.  The left page and the right-right sibling already
exist and stay deltas.  With full_page_writes = on nothing changes: a
never-written page already got a full-page image.
RumNewBuffer()'s comment now states that callers must log the page as
a full image.

Adds 004_rum_split_full_page_writes_off to the RUM recovery TAP tests.
It loads two collections, one per split code path, with
full_page_writes = off, no bgwriter writes and no checkpoints, crashes
the server and recovers it, then checks that no index block is all-zero,
that an insert completes, and that index scans match sequential scans.
Without the fix both split paths leave zero pages behind.

Fixes documentdb#807

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018uo1BA8DhzNJHxezANds93
Signed-off-by: Hari Krishna Sunder <hari90@users.noreply.github.com>
A RUM index damaged as in documentdb#807 (or any other way) has links to all-zero
pages.  A zero page has no flags, so the descent in rumFindLeafPage()
takes it for an internal page, findChildPage() returns block 0, the
metapage, which is also not a leaf, and the loop never ends.  It never
checks for interrupts either, so the backend spins at 100% CPU and
ignores pg_cancel_backend and pg_terminate_backend.  (An assert-enabled
build trips an assertion in PageGetSpecialPointer instead.)

Add rumCheckTreePage(), which raises ERRCODE_INDEX_CORRUPTED with a
REINDEX hint for an all-zero page or the metapage, and call it under the
lock, before the page's flags are read, wherever a page is reached
through a tree link: rumTraverseLock() (each level of the descent),
rumStepCore() and rumStepEntryForScans() (sibling steps), the
rumFindParents() walk, the "refind the leftmost page" loops in
entryGetNextItem() and entryFindItem(), and VACUUM's
rumFindLeftMostLeafPage().  rumFindLeafPage() also raises it for a page
of the other tree kind (entry vs. posting) and for an invalid or
self-referencing downlink, which were only Asserts, and calls
CHECK_FOR_INTERRUPTS() on each level.  The sibling type-mismatch and
deleted-sibling errors use ERRCODE_INDEX_CORRUPTED too.  "Lost path"
stays an internal error: GetWriteErrorFromErrorData() turns it into the
retryable ERRCODE_INDEX_LOSTPATH.

Extends 004_rum_split_full_page_writes_off: it zeroes every posting-tree
leaf page of one index on disk and checks that an insert and an index
scan fail promptly with "contains unexpected zero page".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018uo1BA8DhzNJHxezANds93
Signed-off-by: Hari Krishna Sunder <hari90@users.noreply.github.com>
@hari90
hari90 requested a review from a team as a code owner October 21, 2026 19:13

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The rest of the change looks good to me - I'm just curious if this is actually safe - not sure i follow the comment of "where the lock holds off interrupts" - coudl you clarify this please?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RUM non-root page split WAL-logs the new right page as a generic delta, so crash recovery with full_page_writes = off loses it

2 participants