{{ message }}
Repository navigation
Conversation
notifySubscribers dropped an update whenever a parent's subscriber channel was full. A dropped terminal update means the parent's SDK informer never learns the child finished, and since it never re-lists, the parent waits forever. Raising watchBufferSize only moves the threshold: a 546-child parent still hung at 1000. Each subscription now parks undelivered updates in a map keyed by child action name, with a pump goroutine feeding the caller's channel. A newer update for a still-parked child replaces the older one, so memory is bounded by the number of children and nothing is dropped. The channel returned by Subscribe is unchanged, so the service and mocks are too. Closes flyteorg#8104 Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Dylan Jenkins <dylan.jenkins@ga.gov.au>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tracking issue
Closes #8104
Why are the changes needed?
ActionsClient.notifySubscribersdrops an update when a parent's subscriber channel is full (subscriber channel full, dropping update for parent action). If that update is a child's terminal phase, the parent's SDK informer never learns the child finished. It doesn't re-list, so the parent waits forever and the run has to be aborted. Reconnecting wouldn't help either: the snapshot comes from TaskAction CRs, and finished children's CRs are usually gone by then.Raising
actions.watchBufferSizeonly moves the threshold. We run it at 1000 and still had a parent hang with all 546 of its children SUCCEEDED, under backend load (etcd throttling) that slowed the stream.What changes were proposed in this pull request?
notifySubscribersnever blocks, so a slow stream can't stall the watch workers.Subscribe/Unsubscribekeep their signatures, soActionsServiceand the mocks are unchanged.Unsubscribestops the pump, and the pump closes the channel, so a send can't race the close.watchBufferSizestill sizes the subscriber channel ahead of the parked map, and still sizes the worker channels as before.How was this patch tested?
actions/k8s/client_subscribe_test.go: 1000 children through a 10-slot subscriber all arrive; a parked update is replaced by a newer one for the same child while other children keep their order;Unsubscribecloses the channel. The run-scoping test now waits for async delivery.go test -race ./actions/..., plus-count=50on the subscribe tests.go vet ./actions/...,gofmt -l actions/andgo build ./...are clean.Labels
Check all the applicable boxes
🤖 Generated with Claude Code