xds: Fix intermittent failure in RingHashLoadBalancerTest.allSubchannelsInTransientFailure by kannanjgithub · Pull Request #13130 · grpc/grpc-java · GitHub
Skip to content

xds: Fix intermittent failure in RingHashLoadBalancerTest.allSubchannelsInTransientFailure - #13130

Open
kannanjgithub wants to merge 1 commit into
grpc:masterfrom
kannanjgithub:ringhash_lb_test_npe_fix
Open

kannanjgithub wants to merge 1 commit into
grpc:masterfrom
kannanjgithub:ringhash_lb_test_npe_fix

Conversation

@kannanjgithub

Copy link
Copy Markdown
Contributor

Deliver TRANSIENT_FAILURE in deterministic server order using subchannelList instead of iterating over subchannels.values() (HashMap), and expect 2 connections.

Why this became flaky with the new PickFirst LB:
Old PickFirst unconditionally invoked subchannel.requestConnection() on each call without checking connection state, so triggering proactive connections twice always produced 2 connection requests regardless of iteration order (either [A, B] or [A, A]).

New PickFirst deduplicates connection requests if a subchannel is already in CONNECTING state. When iterating over a HashMap, if the same subchannel was triggered twice, New PickFirst deduplicated it to 1 connection, leading to the assumption in #13094 that New PickFirst would always yield 1 connection. However, if two distinct subchannels were triggered (e.g. server1 then server2), New PickFirst does not deduplicate and yields 2 connections, causing verifyConnection(1) to fail intermittently.

Fix this by iterating deterministically in server order, which reliably triggers two distinct subchannels (server1 and server2) in both old and new PickFirst.

Failure test run in fusion2/d9446b33-dc0e-4dee-b901-ea9b3b57f3c6

…elsInTransientFailure

Deliver TRANSIENT_FAILURE in deterministic server order using subchannelList
instead of iterating over subchannels.values() (HashMap), and expect 2
connections.

Why this became flaky with the new PickFirst LB:
Old PickFirst unconditionally invoked subchannel.requestConnection() on each
call without checking connection state, so triggering proactive connections
twice always produced 2 connection requests regardless of iteration order
(either [A, B] or [A, A]).

New PickFirst deduplicates connection requests if a subchannel is already in
CONNECTING state. When iterating over a HashMap, if the same subchannel was
triggered twice, New PickFirst deduplicated it to 1 connection, leading to the
assumption in grpc#13094 that New PickFirst would always yield 1 connection.
However, if two distinct subchannels were triggered (e.g. server1 then server2),
New PickFirst does not deduplicate and yields 2 connections, causing
verifyConnection(1) to fail intermittently.

Fix this by iterating deterministically in server order, which reliably triggers
two distinct subchannels (server1 and server2) in both old and new PickFirst.

TAG=agy
CONV=d59ec8a0-3bb7-41f8-bb11-76e843a2dada

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant