[engine][gateway] Non-idempotent writes are applied twice when the reply is lost after commit · Issue #767 · documentdb/documentdb · GitHub
Skip to content

[engine][gateway] Non-idempotent writes are applied twice when the reply is lost after commit #767

Open

Description

@guanzhousongmicrosoft

The gateway retries a write when it loses the connection to PostgreSQL. If the commit already landed before the connection dropped, the retry applies the write again. Non-idempotent updates end up applied twice, and inserts report a DuplicateKeyError for a write that actually succeeded. This happens with retryWrites=true and with retryWrites=false.

Tested on release/1.0-RC2-v2 @ 40f86c0, documentdb-local PG18 image built from that commit, pymongo.

Observed (one client call each; the reply was lost after commit):

Operation Result
update_one({_id:1}, {$inc:{n:1}}) returns nModified: 1, stored n is 2
find_one_and_update with $inc returns pre-image n: 1, stored n is 2
upsert with $inc stored n is 2
update_many with $inc every document incremented twice
insert_one / insert_many client gets DuplicateKeyError, but the document was inserted

It also happens without any artificial setup. In a fault-injection run with a counter workload, docker stop / SIGTERM to the gateway and pg_ctl stop -m immediate each left the counter at acknowledged + 1, with zero client errors.

Repro (deterministic)

  1. Set synchronous_standby_names to a standby that doesn't exist, so COMMIT waits in SyncRep after the local commit.
  2. Issue update_one({_id:1}, {$inc:{n:1}}) through the gateway.
  3. pg_terminate_backend() the backend waiting in SyncRep, then reset synchronous_standby_names.
  4. The gateway logs Retrying request (attempt 1): ... connection closed, re-runs the update, and n is 2.

Why

  • The gateway always passes NULL as the transaction id: documentdb_api.insert/update/delete($1, $2, $3, NULL) and find_and_modify(..., NULL) (pg_documentdb_gw/documentdb_gateway_core/src/postgres/query_catalog.rs:639-660). As a result the engine's retryable-write dedupe (documentdb_data.retryable_writes) is never used. It stayed empty after thousands of driver retryable writes, and sending the same lsid + txnNumber twice applied the write twice.
  • The gateway retries non-transactional writes on connectivity errors (postgres/conn_mgmt/query_dispatch.rs, short and long retry tiers). The driver's own retry adds a second layer.

Expected
Each write is applied once. Either pass lsid/txnNumber through so the engine's dedupe works, or stop retrying a write whose commit outcome is unknown and surface a retryable error so the driver can retry with the same transaction id.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions