{{ message }}
Repository navigation
fix(zrpc): skip not-ready endpoints in kube resolver - #5786
Open
SulimanAbdulrazzaq wants to merge 2 commits into
Open
SulimanAbdulrazzaq wants to merge 2 commits into
SulimanAbdulrazzaq wants to merge 2 commits into
Conversation
kevwan
force-pushed
the
fix/kube-resolver-not-ready
branch
from
September 25, 2026 15:37
f3d35e6 to
e5d3a90
Compare
The kube resolver moved from the Endpoints API to EndpointSlices in zeromicro#4987. Endpoints only listed ready addresses in subsets[].addresses, but an EndpointSlice lists every endpoint and reports readiness in its conditions, so terminating and not-ready pods were handed to gRPC as targets. During a rolling update clients kept dialing pods that were shutting down and got connection refused. Only add endpoints whose Ready condition is true or unset, which is the set Endpoints used to publish. Fixes zeromicro#5647
kevwan
force-pushed
the
fix/kube-resolver-not-ready
branch
from
September 25, 2026 16:04
70f4d32 to
daecb0f
Compare
kevwan
approved these changes
Sep 25, 2026
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #5647
The kube resolver moved from the Endpoints API to EndpointSlices in #4987. Endpoints only listed ready addresses in
subsets[].addresses(not-ready ones went tonotReadyAddresses), but an EndpointSlice lists every endpoint and reports readiness inconditions. Since the migration,EventHandleradds every address, so terminating and not-ready pods are handed to gRPC as targets. During a rolling update clients keep dialing pods that are shutting down and getconnection refused, as described in the issue.This change skips endpoints whose
Readycondition isfalseinOnAddandUpdate. A nilReadyis treated as ready, as the EndpointSlice API asks consumers to do. Kubernetes reports terminating endpoints as not ready, so they are dropped too, which gives the same address set the Endpoints API used to publish.OnDeleteis unchanged: it removes the slice's addresses whether or not they were ready.Tests:
TestAddSkipsNotReadyEndpointsandTestUpdateSkipsNotReadyEndpointscover a ready endpoint, one with noReadycondition, a not-ready one and a terminating one. Both fail without the fix (the not-ready and terminating addresses are included) and pass with it.go test -race ./zrpc/resolver/...passes.