spanner: sporadic "Failed to initialize transaction due to invalid mutation key" on mutation-only transactions after upgrading 8.6.0 -> 8.10.0 · Issue #9153 · googleapis/google-cloud-node · GitHub
Skip to content

spanner: sporadic "Failed to initialize transaction due to invalid mutation key" on mutation-only transactions after upgrading 8.6.0 -> 8.10.0 #9153

Open

Description

@diego-dakszewicz

Environment

  • @google-cloud/spanner: 8.10.0 (google-gax 5.0.6, grpc-gcp as wired by the library)
  • Node.js 24 on GKE (Linux)
  • Multiplexed sessions: library defaults (enabled, including read/write)

Workload
High-volume mutation-only read-write transactions, always the same shape:

await database.runTransactionAsync(async transaction => {
    transaction.upsert('Events', mutations);
    return await transaction.commit();
});

Keys are INT64 (Spanner.int over stringified values), well-formed. Thousands of these commits per day against the same table.

Behavior

  • On 8.6.0 (in production from March through August, multiplexed sessions enabled, mutation-key selection already present): zero errors.
  • Within hours of deploying 8.10.0: sporadic 3 INVALID_ARGUMENT: Failed to initialize transaction due to invalid mutation key at a rate of roughly 6 per day out of thousands of identical transactions. The exact same mutation shape succeeds thousands of times a day; the failing mutations carry valid keys (verified from logs).

Analysis
We diffed build/src between 8.6.0 and 8.10.0:

  • The mutation-key selection heuristic (_mutationKey and its selection rules in transaction.js) is identical between both versions.
  • The new code in the transaction path is the per-transaction gRPC channel affinity for multiplexed sessions: _affinityKey = mux-affinity-... in the Snapshot constructor, affinity injected per request, unbind: true injected on commit/rollback/end.

Given the sporadic, race-like profile under concurrency, we suspect the BeginTransaction/Commit pair is occasionally affected by the new affinity routing.

Workaround
Making the affinity branch unreachable (transaction.js:181, if (session.metadata && session.metadata.multiplexed)) restores the 8.6.0 transaction behavior while keeping multiplexed sessions on. We are running this as a pnpm patch and validating that the error rate drops to zero.

Happy to provide timestamps, request IDs, or run an instrumented build if that helps.

Activity

  1. alkatrivedi commented on Aug 17, 2026

    @alkatrivedi
    Contributor

    Hi @diego-dakszewicz, thanks for raising the bug. We will look into it shortly.

  2. self-assigned this
    on Sep 5, 2026
  3. alkatrivedi commented on Oct 6, 2026

    @alkatrivedi
    Contributor

    Hi @diego-dakszewicz, apologies for the delay in getting back to you. To help us investigate further, could you please share the request IDs and timestamps for the failed requests? With those details, we can check the logs on our side.

  4. diego-dakszewicz commented on Oct 6, 2026

    @diego-dakszewicz
    Author

    Hi @alkatrivedi, here are request IDs and timestamps. Two notes first:

    1. The workaround in my report did not hold. I verified that the running build has the affinity branch disabled (transaction.js:181 unreachable, @google-cloud/spanner 8.10.0 otherwise unmodified) and the error keeps happening: 1 to 11 occurrences per day on every one of the last 20 days (2026-09-17 to 2026-10-06). So please disregard the channel-affinity hypothesis; the cause is something else that changed between 8.6.0 and 8.10.0, or it is server side.

    2. We were not logging x-goog-spanner-request-id until today. First samples, all "3 INVALID_ARGUMENT: Failed to initialize transaction due to invalid mutation key" on a mutation-only read-write transaction (runTransactionAsync + upsert + commit) carrying a single upsert of one row with INT64 keys:

    2026-10-06T18:50:53.679Z  1.8b51c8b2.1.1.115295.1
    2026-10-06T19:04:01.219Z  1.8b51c8b2.1.1.119744.1
    2026-10-06T19:28:20.420Z  1.6d68c529.1.1.34918.1
    2026-10-06T20:31:22.383Z  1.6d68c529.1.1.111736.1
    

    I can post more IDs as they come in if that helps.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

api: spannerIssues related to the Spanner API.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions