General: lastActiveTime is rewritten on every poll, one status write per active ScaledObject per pollingInterval · Issue #8256 · kedacore/keda · GitHub
You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
General: lastActiveTime is rewritten on every poll, one status write per active ScaledObject per pollingInterval #8256
For an active ScaledObject whose target already has replicas, the executor sets status.lastActiveTime = now on every poll
(pkg/scaling/executor/scale_scaledobjects.go, active/no-scale branch). handleResult skips the status patch only when nothing changed, and the
timestamp always changes, so every active ScaledObject costs one status PATCH
per pollingInterval: N / pollingInterval writes/s cluster-wide.
Impact at scale
Load test on EKS (Prometheus triggers, pod-backed Deployments, pollingInterval 10 s):
20k ScaledObjects: ~1,860 ScaledObject status writes/s measured at the
apiserver (apiserver_storage_events_received_total{resource="scaledobjects"}),
vs ~0/s for HPAs.
The apiserver's ScaledObject watch cache ran ~700k resource versions behind
the HPA/pod caches; consistent lists of ScaledObjects failed with Too large resource version.
apiserver_terminated_watchers_total{resource="scaledobjects"} = 2,188
(slow watchers dropped), forcing the operator to re-list.
New ScaledObjects then waited minutes for their HPA: HPA creation p99 up to
379 s, and at 20k on a large control plane, 0 of 500 new ScaledObjects got an
HPA in 10 minutes, while the HPA controller itself was healthy.
Why it's avoidable
lastActiveTime is only read by scaleToZeroOrIdle, for the cooldown before
scaling to zero / idleReplicaCount. A ScaledObject with minReplicaCount > 0 and no idleReplicaCount never reads it.
Proposed change
Don't record it when the ScaledObject can't scale to zero/idle.
Otherwise refresh it at most every cooldownPeriod / 2 (scale-in can then
start at most that much before the full cooldown). Activation from zero/idle
still records it every time.
Implementation + unit tests ready on a fork branch
(ggarb/keda:fix-lastactivetime-writes). Happy to open the PR if this is
approved and assigned to me. Open to a different throttle (e.g. a fixed
interval, or skip-only) if maintainers prefer.
Activity