Summary

When a RUM page that isn't the root splits, the newly allocated right page is registered for generic WAL with flags 0. That logs it as a delta against its previous contents instead of a full image:

• pg_documentdb_extended_rum/core/src/rumbtree.c:760, in rumPlaceToPage()
• pg_documentdb_extended_rum/core/src/rumbtree.c:1272, in rumInsertValueOld()

/* split non-root page */
...
lpage = GenericXLogRegisterBuffer(state, stack->buffer, 0);
rpage = GenericXLogRegisterBuffer(state, rbuffer, 0);     /* rbuffer = RumNewBuffer(...) */

The PostgreSQL docs for generic WAL say GENERIC_XLOG_FULL_IMAGE "indicates that a full-page image rather than a delta update should be included in the WAL record. Typically this flag would be set if the page is new or has been rewritten completely." The root-split branch of both functions already does this (rumbtree.c:657 and :1173).

With full_page_writes = on, the delta is harmless, because the first change to a page after a checkpoint carries a full-page image anyway. With full_page_writes = off:

• a page that RumNewBuffer() just extended is all-zero on disk until it's written out;
• in crash recovery, generic_redo → XLogReadBufferForRedo finds a new page and skips the delta;
• crash recovery doesn't run the invalid-page check, so it finishes with no error.

The index then has live downlinks and sibling links pointing at all-zero pages. Every later write that descends into one of them never returns, and doesn't respond to pg_cancel_backend/pg_terminate_backend.

Seen on documentdb 1.0-0 (extended_rum, the default handler) with PostgreSQL 18. The code is the same in 0.116-0.

Repro

This runs against the stock ghcr.io/documentdb/documentdb/documentdb-local:pg18-1.0.0 image and needs Docker and mongosh. It turns off the background writer and checkpoints so the new index pages are still only in shared buffers when the server is killed.

#!/usr/bin/env bash
# Usage: ./repro.sh off   (bug)   |   ./repro.sh on   (control)
set -euo pipefail
FPW=${1:-off}; C=rum-fpw-$FPW; PORT=${PORT:-10260}
IMG=ghcr.io/documentdb/documentdb/documentdb-local:pg18-1.0.0
psql_() { docker exec "$C" psql -h localhost -p 9712 -U documentdb -d postgres -qAtc "$1"; }
wait_up() { for _ in $(seq 60); do psql_ 'select 1' >/dev/null 2>&1 && return; sleep 2; done; echo "server did not start"; exit 1; }
MONGO="mongodb://bench:benchpw@127.0.0.1:$PORT/?tls=true&tlsAllowInvalidCertificates=true&directConnection=true"

docker run -d --name "$C" -p "127.0.0.1:$PORT:10260" "$IMG" \
  --username bench --password benchpw --disable-usage-telemetry >/dev/null
wait_up
# Keep new index pages in shared buffers until the crash: no bgwriter writes, no checkpoints.
psql_ "alter system set full_page_writes = '$FPW'"
psql_ "alter system set bgwriter_lru_maxpages = 0"
psql_ "alter system set checkpoint_timeout = '1h'"
psql_ "alter system set max_wal_size = '20GB'"
docker restart "$C" >/dev/null; wait_up
psql_ "checkpoint"

# 60k documents, each with 2 of 5 tag values: posting trees that split many non-root pages.
mongosh --quiet "$MONGO" --eval '
  const c = db.getSiblingDB("repro").items;
  c.createIndex({ tags: 1 });
  for (let b = 0; b < 60; b++) {
    const docs = [];
    for (let i = 0; i < 1000; i++) { const n = b * 1000 + i; docs.push({ n, tags: ["t" + n % 5, "t" + (n + 1) % 5] }); }
    c.insertMany(docs);
  }
  print("documents:", c.countDocuments());'

docker kill -s KILL "$C" >/dev/null      # crash
docker start "$C" >/dev/null; wait_up      # crash recovery

psql_ "create extension if not exists pageinspect"
IDX=$(psql_ "select indexrelid::regclass from pg_index i join pg_class c on c.oid = i.indexrelid join pg_am a on a.oid = c.relam where a.amname like '%rum%' and c.relnamespace = 'documentdb_data'::regnamespace")
echo "full_page_writes=$FPW  index $IDX: $(psql_ "select count(*) filter (where (page_header(get_raw_page('$IDX', b))).upper = 0) || ' of ' || count(*) || ' pages are all-zero' from generate_series(0, pg_relation_size('$IDX') / 8192 - 1) b")"
echo "recovery log:"; docker exec "$C" grep -E 'redo (starts|done)|invalid page|PANIC' /data/pglog.log | tail -3
echo "insertOne({tags: ['t0']}) with a 15 s timeout:"
timeout 15 mongosh --quiet "$MONGO" --eval 'db.getSiblingDB("repro").items.insertOne({ n: -1, tags: ["t0"] }); print("  ok")' || echo "  no response after 15 s"

Output:

$ ./repro.sh off
documents: 60000
full_page_writes=off  index documentdb_data.documents_rum_index_3: 86 of 108 pages are all-zero
recovery log:
LOG:  redo starts at 0/2679A88
LOG:  redo done at 0/5FB15F8 system usage: CPU: user: 0.14 s, system: 0.03 s, elapsed: 0.19 s
insertOne({tags: ['t0']}) with a 15 s timeout:
  no response after 15 s

$ PORT=10261 ./repro.sh on
documents: 60000
full_page_writes=on  index documentdb_data.documents_rum_index_3: 0 of 108 pages are all-zero
recovery log:
LOG:  redo starts at 0/267F6B8
LOG:  redo done at 0/60CCF78 system usage: CPU: user: 0.13 s, system: 0.05 s, elapsed: 0.21 s
insertOne({tags: ['t0']}) with a 15 s timeout:
  ok

Recovery reports nothing wrong. The zero pages are live: internal posting-tree pages still hold downlinks to them, and leaf pages hold left/right links to them. The hung insertOne backend spins in rumFindLeafPage() (it reads child 0, the metapage, over and over) at 100% CPU and ignores cancel and terminate. Clean up with docker rm -f rum-fpw-off rum-fpw-on.

Why this is a RUM bug

full_page_writes = off is a supported setting for storage that never tears an 8 kB write. No torn page is involved here: the page on disk is a clean, zero-extended block, which is exactly the state crash recovery has to handle. The core index AMs handle it by logging a new page as an init record (BKPBLOCK_WILL_INIT) or a full image, and contrib bloom, which also uses generic WAL, registers new pages with GENERIC_XLOG_FULL_IMAGE (blinsert.c). Only RUM's non-root split depends on full-page writes to be crash-safe.

Suggested fix

Register the new right page with GENERIC_XLOG_FULL_IMAGE in the non-root split at both call sites, as the root split already does for its new pages:

rpage = GenericXLogRegisterBuffer(state, rbuffer, GENERIC_XLOG_FULL_IMAGE);

RumNewBuffer() can also hand back a recycled page from the FSM; a full image is correct there too. Any other path that initializes a page RumNewBuffer() returned should be checked the same way.

Separately, it would help if rumFindLeafPage() raised ERRCODE_INDEX_CORRUPTED on a new or wrong-type page, as nbtree's _bt_checkpage() does, and checked for interrupts in its descent loop. Then an index damaged this way (or any other way) fails with an error instead of hanging the backend.