You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
spanner: sporadic "Failed to initialize transaction due to invalid mutation key" on mutation-only transactions after upgrading 8.6.0 -> 8.10.0 #9153
Keys are INT64 (Spanner.int over stringified values), well-formed. Thousands of these commits per day against the same table.
Behavior
On 8.6.0 (in production from March through August, multiplexed sessions enabled, mutation-key selection already present): zero errors.
Within hours of deploying 8.10.0: sporadic 3 INVALID_ARGUMENT: Failed to initialize transaction due to invalid mutation key at a rate of roughly 6 per day out of thousands of identical transactions. The exact same mutation shape succeeds thousands of times a day; the failing mutations carry valid keys (verified from logs).
Analysis
We diffed build/src between 8.6.0 and 8.10.0:
The mutation-key selection heuristic (_mutationKey and its selection rules in transaction.js) is identical between both versions.
The new code in the transaction path is the per-transaction gRPC channel affinity for multiplexed sessions: _affinityKey = mux-affinity-... in the Snapshot constructor, affinity injected per request, unbind: true injected on commit/rollback/end.
Given the sporadic, race-like profile under concurrency, we suspect the BeginTransaction/Commit pair is occasionally affected by the new affinity routing.
Workaround
Making the affinity branch unreachable (transaction.js:181, if (session.metadata && session.metadata.multiplexed)) restores the 8.6.0 transaction behavior while keeping multiplexed sessions on. We are running this as a pnpm patch and validating that the error rate drops to zero.
Happy to provide timestamps, request IDs, or run an instrumented build if that helps.
Hi @diego-dakszewicz, apologies for the delay in getting back to you. To help us investigate further, could you please share the request IDs and timestamps for the failed requests? With those details, we can check the logs on our side.
Hi @alkatrivedi, here are request IDs and timestamps. Two notes first:
The workaround in my report did not hold. I verified that the running build has the affinity branch disabled (transaction.js:181 unreachable, @google-cloud/spanner 8.10.0 otherwise unmodified) and the error keeps happening: 1 to 11 occurrences per day on every one of the last 20 days (2026-09-17 to 2026-10-06). So please disregard the channel-affinity hypothesis; the cause is something else that changed between 8.6.0 and 8.10.0, or it is server side.
We were not logging x-goog-spanner-request-id until today. First samples, all "3 INVALID_ARGUMENT: Failed to initialize transaction due to invalid mutation key" on a mutation-only read-write transaction (runTransactionAsync + upsert + commit) carrying a single upsert of one row with INT64 keys:
Activity
alkatrivedi commentedon Aug 17, 2026
Hi @diego-dakszewicz, thanks for raising the bug. We will look into it shortly.
alkatrivedi commentedon Oct 6, 2026
Hi @diego-dakszewicz, apologies for the delay in getting back to you. To help us investigate further, could you please share the request IDs and timestamps for the failed requests? With those details, we can check the logs on our side.
diego-dakszewicz commentedon Oct 6, 2026
Hi @alkatrivedi, here are request IDs and timestamps. Two notes first:
The workaround in my report did not hold. I verified that the running build has the affinity branch disabled (transaction.js:181 unreachable, @google-cloud/spanner 8.10.0 otherwise unmodified) and the error keeps happening: 1 to 11 occurrences per day on every one of the last 20 days (2026-09-17 to 2026-10-06). So please disregard the channel-affinity hypothesis; the cause is something else that changed between 8.6.0 and 8.10.0, or it is server side.
We were not logging x-goog-spanner-request-id until today. First samples, all "3 INVALID_ARGUMENT: Failed to initialize transaction due to invalid mutation key" on a mutation-only read-write transaction (runTransactionAsync + upsert + commit) carrying a single upsert of one row with INT64 keys:
I can post more IDs as they come in if that helps.