KAFKA-21321: Let the refiner see a member that joins in the same heartbeat by suzhiking · Pull Request #23899 · apache/kafka · GitHub
Skip to content

KAFKA-21321: Let the refiner see a member that joins in the same heartbeat - #23899

Open
suzhiking wants to merge 1 commit into
apache:trunkfrom
suzhiking:KAFKA-21321-refiner-joining-member
Open

suzhiking wants to merge 1 commit into
apache:trunkfrom
suzhiking:KAFKA-21321-refiner-joining-member

Conversation

@suzhiking

@suzhiking suzhiking commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

In a streams group heartbeat, the coordinator refines the target
assignment into an intermediate assignment with warm-up tasks. The
target assignment it refines already reflects the membership change of
the heartbeat: maybeUpdateStreamsTargetAssignment applies that change
through an UpdatedMembersAndTargetAssignmentView, which adds a joining
member and relabels a replaced static member to its new member ID. The
refiner, however, was handed group.members(), which does not contain
that member yet, because the records of the heartbeat are only replayed
after the handler returns. The refiner therefore treated the member as
one that had left the group.

The worst effect is on a static member that rejoins under a new member
ID, as it does on every restart. The refiner kept the member's stateful
active tasks on the old member ID, which the replacement removes, so the
rejoined member was not given them and no member ran them until the next
refinement step. In addition, a new member did not get a warm-up task
for a stateful task that the target assignment moves to it, and a new
member that is the target standby holder of a migrating task was not
held back, so the task ran one standby more than configured while the
migration lasted.

maybeUpdateStreamsTargetAssignment now returns the view's members
together with the target assignment, and the refiner is handed those
members whenever it refines that target assignment. The other call site,
which refines the current target assignment of a Stable group when the
heartbeat changes nothing, keeps using group.members(): a heartbeat that
adds or replaces a member never takes that path, because a new member
bumps the group epoch and a replaced static member has left with epoch
-2, so the group is not Stable.

See https://issues.apache.org/jira/browse/KAFKA-21321. The tests cover
two static member rejoins with the real refiner and warm-up tasks
enabled: one from a new process, where the target assignment is
recomputed, and one from the same process with no intermediate
assignment cached for the epoch, as after a coordinator failover. Both
fail without this change, because the rejoined member is given no active
task.

Reviewers: Sean Quah squah@confluent.io

…tbeat

In a streams group heartbeat, the coordinator refines the target
assignment into an intermediate assignment with warm-up tasks. The
target assignment it refines already reflects the membership change of
the heartbeat: maybeUpdateStreamsTargetAssignment applies that change
through an UpdatedMembersAndTargetAssignmentView, which adds a joining
member and relabels a replaced static member to its new member ID. The
refiner, however, was handed group.members(), which does not contain
that member yet, because the records of the heartbeat are only replayed
after the handler returns. The refiner therefore treated the member as
one that had left the group.

The worst effect is on a static member that rejoins under a new member
ID, as it does on every restart. The refiner kept the member's stateful
active tasks on the old member ID, which the replacement removes, so the
rejoined member was not given them and no member ran them until the
next refinement step. In addition, a new member did not get a warm-up
task for a stateful task that the target assignment moves to it, and a
new member that is the target standby holder of a migrating task was
not held back, so the task ran one standby more than configured while
the migration lasted.

maybeUpdateStreamsTargetAssignment now returns the view's members
together with the target assignment, and the refiner is handed those
members whenever it refines that target assignment. The other call
site, which refines the current target assignment of a Stable group
when the heartbeat changes nothing, keeps using group.members(): a
heartbeat that adds or replaces a member never takes that path, because
a new member bumps the group epoch and a replaced static member has
left with epoch -2, so the group is not Stable.

See https://issues.apache.org/jira/browse/KAFKA-21321. The tests cover
two static member rejoins with the real refiner and warm-up tasks
enabled: one from a new process, where the target assignment is
recomputed, and one from the same process with no intermediate
assignment cached for the epoch, as after a coordinator failover. Both
fail without this change, because the rejoined member is given no
active task.
@github-actions github-actions Bot added triage PRs from the community group-coordinator labels Oct 21, 2026
Comment on lines +319 to +332

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What would the PR look like if we lifted UpdatedMembersAndTargetAssignmentView up into streamsGroupHeartbeat instead?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

group-coordinator triage PRs from the community

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants