Repository navigation
Spread poller scaling suggestions evenly across a fleet - #12636
veeral-patel wants to merge 1 commit into
Conversation
Scaling suggestions ride on dispatched tasks, and under FIFO matching a worker's
share of dispatched tasks equals its share of pollers. Growth is therefore
proportional to current size in both directions, the equilibrium is degenerate,
and any imbalance is preserved rather than corrected. Measured on three
identical workers: 74/12/14 percent of activity pollers, a different winner
each run.
Workers now report their poller target on each poll. pollerShareTracker keeps
the latest report per worker instance for a physical queue -- instance rather
than identity, since a fleet commonly shares one configured identity -- and
compares a worker's poller count against the fleet mean. applyFairShare then
withholds a suggestion that would push a worker further from its share: no
scale-up for a worker above the mean, no scale-down for one below it.
It only ever withholds, never substitutes or redirects. A withheld +1 is
dropped rather than handed to an under-share peer, so this changes which
workers grow and also makes the fleet grow somewhat less; it cannot repair a
gap that already exists.
Because the suggestion is counted before it is withheld, scale_up and
scale_down count suggestions proposed, and hold{over_share,under_share} the
subset denied.
Workers on SDKs that do not report, and queues with fewer than two reporting
workers, are unaffected. Gated on matching.pollerScalingFairnessBand, which
defaults to 0 (disabled); 1.2 is the suggested enabled value.
Shutdown now evicts a worker from the tracker, so a departed worker stops
inflating the fleet mean. That makes removePollerFromHistory run for requests
carrying only a worker instance key, which reaches a debug log the lightweight
CancelOutstandingWorkerPolls fixtures had never exercised; they now set a
logger.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| } | ||
|
|
||
| own := float64(self.target) | ||
| fleet := float64(totalTarget) / float64(live) |
There was a problem hiding this comment.
Heterogeneous or capped workers cap the whole fleet. The mean counts workers that can't grow, so a low target anchors it. Example with band 1.2: worker a is at its configured MaximumNumberOfPollers = 2 (or is just a smaller box), and worker b has target x. Then b is shareOver once x > 1.2·(2+x)/2, which means x > 3. So b stops at about 3 pollers however large the backlog gets. Meanwhile a reads shareUnder and keeps getting +1s it can't act on. The same thing happens with N workers: one 2-poller worker plus two others caps those two at about 4.
The comparison assumes identical, uncapped workers. That matches the benchmark, but mixed instance sizes and per-worker poller maxima are common in real fleets. Some options:
- have the SDK report its max/ceiling and leave saturated workers out of the mean;
- only withhold scale-up when some under-share peer could actually take the suggestion;
- at minimum, say in the dynamic config description that the band assumes homogeneous workers.
There was a problem hiding this comment.
applyFairShare has no test. The new tests only cover pollerShareTracker.classify, so these pieces are unverified:
- the direction logic: withhold
+1when over and-1when under, and pass through+1when under and-1when over; - the
band <= 1and nilpollMetadatashort-circuits; - the
hold{over_share,under_share}metric emission.
A small table test on a physicalTaskQueueManagerImpl with a seeded tracker and a stubbed config would cover them. The MakePollerScalingDecision tests that already exist in physical_task_queue_manager_test.go could be extended to do it.
This branch has not been deployed
1 participant
Problem
Poller scaling suggestions ride on dispatched tasks, and under FIFO matching a worker's share of dispatched tasks equals its share of pollers. Growth is therefore proportional to current size in both directions, the equilibrium is degenerate, and any imbalance between otherwise identical workers is preserved rather than corrected.
Measured on three identically configured workers: 74/12/14 percent of activity pollers, a different winner each run.
Change
Workers report their poller target on each poll (temporalio/sdk-go#2799, proto in temporalio/api#887).
pollerShareTrackerkeeps the latest report per worker instance for a physical queue and compares a worker's poller count against the fleet mean.applyFairSharethen withholds a suggestion that would push a worker further from its share: no scale-up for a worker above the mean, no scale-down for one below it.Gated on
matching.pollerScalingFairnessBand, default0(disabled);1.2is the suggested enabled value. Values at or below 1 disable it.Design notes
workerInstanceKey, notpollerIdentity. A fleet commonly shares one configured identity, and fairness has to tell its members apart. This is whyRemovePollergains the instance key.+1is dropped rather than handed to an under-share peer, so this changes which workers grow and also makes the fleet grow somewhat less. It cannot repair a gap that already exists.classifycovers crashed pods where eviction never runs.scale_up/scale_downcount suggestions proposed andhold{over_share,under_share}the subset denied.One incidental test change: evicting on shutdown makes
removePollerFromHistoryrun for requests carrying only a worker instance key, which reaches a debug log the lightweightCancelOutstandingWorkerPollsfixtures had never exercised. They now set a logger.Testing
go build ./...,go vet, and the fullservice/matchingsuite pass. New unit tests cover over/under/fair classification at band 1.2, single-reporter fail-open, the disable switch, eviction on shutdown, and the TTL sweep.Benchmarked on a worker-join scenario (two workers reach steady state, a third joins). With the band disabled the newcomer never recovers — 2.7 to 6.1 percent of the fleet against an even 33.3 across five runs. With the band at 1.2 it reaches a fair share. The
over_sharecounter is non-zero only when the band is enabled and exactly zero otherwise, so the mechanism is attributable rather than coincidental.Note
make proto-codegenis required after the api change: the generated matching client rebuilds the poll request field by field, and silently drops a new field until regenerated.