General: lastActiveTime is rewritten on every poll, one status write per active ScaledObject per pollingInterval · Issue #8256 · kedacore/keda · GitHub
Skip to content

General: lastActiveTime is rewritten on every poll, one status write per active ScaledObject per pollingInterval #8256

Open

Description

@ggarb

Report

For an active ScaledObject whose target already has replicas, the executor sets
status.lastActiveTime = now on every poll
(pkg/scaling/executor/scale_scaledobjects.go, active/no-scale branch).
handleResult skips the status patch only when nothing changed, and the
timestamp always changes, so every active ScaledObject costs one status PATCH
per pollingInterval: N / pollingInterval writes/s cluster-wide.

Impact at scale

Load test on EKS (Prometheus triggers, pod-backed Deployments, pollingInterval 10 s):

  • 20k ScaledObjects: ~1,860 ScaledObject status writes/s measured at the
    apiserver (apiserver_storage_events_received_total{resource="scaledobjects"}),
    vs ~0/s for HPAs.
  • The apiserver's ScaledObject watch cache ran ~700k resource versions behind
    the HPA/pod caches; consistent lists of ScaledObjects failed with
    Too large resource version.
  • apiserver_terminated_watchers_total{resource="scaledobjects"} = 2,188
    (slow watchers dropped), forcing the operator to re-list.
  • New ScaledObjects then waited minutes for their HPA: HPA creation p99 up to
    379 s, and at 20k on a large control plane, 0 of 500 new ScaledObjects got an
    HPA in 10 minutes, while the HPA controller itself was healthy.

Why it's avoidable

lastActiveTime is only read by scaleToZeroOrIdle, for the cooldown before
scaling to zero / idleReplicaCount. A ScaledObject with
minReplicaCount > 0 and no idleReplicaCount never reads it.

Proposed change

  • Don't record it when the ScaledObject can't scale to zero/idle.
  • Otherwise refresh it at most every cooldownPeriod / 2 (scale-in can then
    start at most that much before the full cooldown). Activation from zero/idle
    still records it every time.

Implementation + unit tests ready on a fork branch
(ggarb/keda:fix-lastactivetime-writes). Happy to open the PR if this is
approved and assigned to me. Open to a different throttle (e.g. a fixed
interval, or skip-only) if maintainers prefer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions