Add healthy heartbeating activities to throughput_stress by lilydoar · Pull Request #519 · temporalio/omes · GitHub
Skip to content

Add healthy heartbeating activities to throughput_stress - #519

Open
lilydoar wants to merge 2 commits into
lilydoar/nexus-signal-request-idfrom
lilydoar/healthy-heartbeat-activities
Open

lilydoar wants to merge 2 commits into
lilydoar/nexus-signal-request-idfrom
lilydoar/healthy-heartbeat-activities

Conversation

@lilydoar

Copy link
Copy Markdown
Contributor

What changed

New throughput_stress options add healthy heartbeating remote activities to each internal iteration:

Option Default Meaning
healthy-heartbeat-activities 0 (off) Activities per internal iteration
healthy-heartbeat-duration 30s How long each activity runs while heartbeating
healthy-heartbeat-timeout 10s Heartbeat timeout of each activity

Each activity uses the Go worker's existing Heartbeat action with no failed attempts and a heartbeat interval of a quarter of the timeout. It has a single attempt, so a heartbeat timeout fails the workflow and the run.

Why

The only heartbeats throughput_stress produces come from its retry scenario, where the first attempt deliberately times out. Load runs exercise the heartbeat failure path but not activities that stay alive by heartbeating, which is the path that has to keep working under load, faults and restarts.

Details worth reviewing

  • Interval support for the Heartbeat action exists only in the Go worker, so the help text marks the option Go-only, like the other Go-only options.
  • The Go SDK sends at most one heartbeat RPC per 80% of the heartbeat timeout, so each activity sends about 1 + duration / (0.8 × timeout) RPCs.
  • TestThroughputStressHealthyHeartbeatKeepsActivityAlive runs an activity longer than its heartbeat timeout against a real worker; with heartbeating disabled it fails with a heartbeat timeout.

@lilydoar
lilydoar requested review from a team as code owners October 20, 2026 19:04
@lilydoar
lilydoar force-pushed the lilydoar/healthy-heartbeat-activities branch from e4b7a42 to 3d50d6b Compare October 20, 2026 19:23
@lilydoar
lilydoar changed the base branch from main to lilydoar/nexus-signal-request-id October 20, 2026 19:23
@lilydoar
lilydoar added this pull request to stack #520 October 20, 2026 19:23
throughput_stress only heartbeats in its retry scenario, where the first
attempt deliberately times out. Nothing exercises activities that stay alive
by heartbeating, so load runs cover the heartbeat failure path but not
sustained successful heartbeats.

healthy-heartbeat-activities adds that many remote activities to each internal
iteration. Each heartbeats for healthy-heartbeat-duration with a
healthy-heartbeat-timeout heartbeat timeout and a single attempt, so a
heartbeat timeout fails the run. Off by default. Uses the Go worker's existing
heartbeat activity, so no worker change is needed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant