{{ message }}
Repository navigation
Conversation
A patch series proposed for PostgreSQL (CommitFest 5556, "Expanding HOT updates for expression and partial indexes") adds two server APIs that DocumentDB can use so that updates to a document stay heap-only (HOT) when they leave the document's index entries unchanged: - amcomparedatums, an optional index AM callback that says whether a new column value would give the same index entries as the old one; and - SupportRequestSubattr, a planner support request that says which parts of a structured value a function reads or may change. Neither is in a released PostgreSQL. Detect them from the server's headers, not from its version, and define DOCUMENTDB_HAVE_AMCOMPAREDATUMS and DOCUMENTDB_HAVE_SUBATTR_SUPPORT so code using them builds only where they exist. Nothing uses the definitions yet. Proposal: https://commitfest.postgresql.org/patch/5556/ Assisted-by: Claude Opus 5.5 (claude-opus-5-5) via pi coding agent; human reviewed and tested. AIA HAb CeNc Hin R claude-opus-5-5 v1.0 Signed-off-by: Greg Burd <greg@burd.me>
Every DocumentDB index on a document path is a RUM index on the bson
document column itself, with the path held in operator class options:
CREATE INDEX ... USING documentdb_rum
(document bson_rum_single_path_ops (path = 'a.b'))
PostgreSQL can only update a row as heap-only (HOT), making no new index
entries, if no indexed column changed. Since every write changes the
document column, every update to a collection with any path index has made
new entries in every one of its indexes, including those whose terms did
not change, and left dead entries for VACUUM. That is true of updateOne
and findAndModify, which compute the new document in C and write it back
whole, and of updateMany.
A proposed PostgreSQL index AM callback, amcomparedatums, lets the index
answer instead: would the new value give the same entries as the old?
Implement it for RUM. A RUM index stores, for each row, the keys its
operator class extracts and scans see the row only through them, so the
entries are the same if extractValue gives the same keys, compared as RUM
compares them. For DocumentDB that means the terms for the paths an index
covers, so an update that leaves those paths alone keeps every index's
entries and can be HOT.
The callback is conservative where an entry holds more than its key: it
returns false for columns with per-item additional information (text
indexes, which store positions) and for indexes with attach or add-to
columns. It also returns false if the new document would set opclass
metadata bits, such as a composite index's multi-key paths, that the old
one did not: insertion records those in the metapage, and skipping it must
not lose them.
Same entries are not enough for an exclusion constraint, which is how
DocumentDB builds unique indexes: a HOT update skips the constraint check,
and the constraint compares the new values, not the entries (its terms are
hashes). The server does not call amcomparedatums for an exclusion
constraint's index; the callback refuses such an index anyway. So an
update to a collection with a secondary unique index is not HOT.
The callback is only built when the server has the API (see the previous
commit); documentdb_rum loaded from the public rum library does not get
it. The new test checks HOT counts, which depend on the server, only as
"all or none", and checks on every server that index scans find the same
documents as sequential scans after updates that do and do not change
indexed paths, and that a unique index still rejects an update whose
terms collide with another document's.
Proposal: https://commitfest.postgresql.org/patch/5556/
Assisted-by: Claude Opus 5.5 (claude-opus-5-5) via pi coding agent; human
reviewed and tested. AIA HAb CeNc Hin R claude-opus-5-5 v1.0
Signed-off-by: Greg Burd <greg@burd.me>
gburd
marked this pull request as ready for review
October 17, 2026 09:30
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Lets updates to a document stay heap-only (HOT) when they leave the terms of
every index on the collection unchanged. This needs a PostgreSQL index AM
callback,
amcomparedatums, proposed in CommitFest 5556 ("Expanding HOTupdates for expression and partial indexes",
https://commitfest.postgresql.org/patch/5556/). The callback is not in any
released PostgreSQL. On a server without it this PR compiles to a no-op:
the Makefile looks for the API in the server headers and builds the new code
only when it is there.
Every DocumentDB path index is a RUM index on the
documentcolumn. Everywrite changes that column, so today every update makes new entries in every
index, and leaves dead ones for VACUUM, even when the indexed paths did not
change. With
amcomparedatums, PostgreSQL asks the index whether the new valuegives the same entries as the old. This PR implements it for the extended RUM
AM (
rumcomparedatumsinpg_documentdb_extended_rum/core/src/rumutil.c): itextracts the terms of the old and new documents and compares them as RUM
compares them.
It returns false (not HOT) when:
index has attach / add-to columns;
index's multi-key paths, that the old one did not set;
indexes. A HOT update skips the constraint check, and the constraint
compares values, not index entries (its terms are hashes). The proposed core
patch does not call
amcomparedatumsfor exclusion-constraint indexes; thisguard is defence in depth. Consequence: collections with a secondary unique
index get no HOT updates on the document column. A possible follow-up is
a way for an AM to declare that "same entries" also means "same exclusion
result" (true for DocumentDB's
=?=over generated terms).Commits:
Makefile.globalgreps theserver headers and defines
DOCUMENTDB_HAVE_AMCOMPAREDATUMS/DOCUMENTDB_HAVE_SUBATTR_SUPPORT). This commit is shared with thecompanion subattr PR.
rumcomparedatumsand the testrum_hot_update_tests.Measured on a Debian 13 EC2 host (32 vCPU) with PostgreSQL REL_19_STABLE plus
the CF 5556 backport (debug, cassert). 200 documents, indexes
a_1, compositeb_1_c_1andh_hashed, fillfactor 10:$incof an unindexed field, 200 docs$setof an unindexed field, 50 docs$incof an indexed field, 20 docsMemory: the server calls
amcomparedatumsonce per changed row. An earlierversion of the core patch ran it in the per-query memory context, so peak
backend memory grew with the number of rows updated. The current core patch
(v39, branch
cf-5556)runs it in a per-tuple context. This PR depends on that
fix. Peak backend RSS (VmHWM), updateMany
$incover 60,000 documents withindexes
a_1,t_1(40-element arrays) and compositeb_1_c_1:rumcomparedatumscompiled outWith the memory fix, peak memory matches stock PG19, so the remaining growth
is baseline behaviour, not this PR. Stock PG19 makes 0 HOT updates on this
workload, and so does the patched server: fillfactor is 100 and the pages are
full. The HOT table above uses fillfactor 10. For a pure-core comparison,
updating 100k
int[200]rows under a GIN index so that the GIN keys stay thesame: stock 238 MB, core before the fix 239 → 873 MB, core with the fix 225 →
225 MB.
Tests on the same host. "Branch" is the integration branch containing this PR,
the subattr PR and the alternate-AM EXPLAIN fixes:
make check-basicinextended_rum_tests: main 69/69, branch 71/71 (the 2new tests).
make check-basicinregress: main 125/125, branch 125/125.check-minimalwithrum_hot_update_tests: 2/2 passed.-Werroragainst patched PG19 (flag on,rumcomparedatumsin the library) and against stock REL_19_STABLE (flag off, symbol absent).
citus_indent --checkandscripts/validate_extension_file_state.shpass.The test checks HOT counts only as "all or none", so it passes on servers with
and without the API. On every server it also checks that index scans return
the same documents as sequential scans after updates that do and do not
change indexed paths, and that a unique index still rejects an update whose
terms collide with another document's (
defaultUniqueIndexKeyhashOverrideplus truncated terms).
Type of Change
AI Disclosure
builds, tests and measurements on the EC2 host; reviewed the RUM metadata
and exclusion-constraint cases. A human reviewed and tested the result.
Checklist
git commit -s)Disclosure
This work was created with an even blend of human and AI contributions. AI was
used to make content edits, such as changes to scope, information, and ideas.
AI was used to make new content, such as text, images, analysis, and ideas. AI
was prompted for its contributions, or AI assistance was enabled. AI-generated
content was reviewed and approved. The following model(s) or application(s)
were used: claude-opus-5-5 (via the pi coding agent). (AIA HAb CeNc Hin R claude-opus-5-5 v1.0)