for and try blocks compile to child workflows — fan-in read/lock amplification on hot parents at scale

@mrsimonemms

Summary

Each for iteration and each try block currently compiles into its own child workflow rather than executing inline in the parent. At scale this creates significant fan-in amplification: a single hot parent workflow spawns many children, and each child completion forces the parent to reload its mutable state, concentrating reads and write-lock contention on the shard(s) holding the hot parents.

Confirmed with @mrsimonemms that the child-per-block split is a DSL-parsing convenience, not a load-bearing design requirement — which is why we're raising this: the behavior looks safe to change, and it would remove a real production bottleneck.

Observed impact (production, Temporal)

A single hot parent with two for loops and a try block spawns ~7 children per execution. Under burst load we measured, in the affected namespace:

resource_exhausted was ~100% cause=BusyWorkflow (per-execution lock contention), peaking ~74/s — while RpsLimit was ≈0 and PersistenceLimit never fired. So this is not frontend RPS or persistence throttling; it's parent-execution contention driven by the child fan-in.
Each child completion triggers a parent GetWorkflowExecution state reload; under burst these reads concentrated on the shards holding the hot parents, running 1–4s (p95 ~4.3s), with ~5–6% returning serviceerror_Unavailable.
History-side, the reads timed out specifically on buffered-events / signals for the hot parents, and RecordChildExecutionCompleted / ScheduleWorkflowTask were the dominant BusyWorkflow operations.

The for-loop children dominate the amplification (a try-block child is comparatively minor).

Request

Explore executing for iterations and try blocks inline in the parent (or a batched form) instead of one child workflow per block, at least as an opt-in, since the child-per-block model amplifies reads and write-lock contention on the parent.

Open questions for maintainers:

Is inline execution feasible for for/try without breaking the DSL parsing model that motivated children?
If children are needed for isolation in some cases, could iterations be batched (N iterations per child) to cut fan-out?
@mrsimonemms noted removing the child wrapper should be possible but might expose a different underlying issue — worth scoping that as part of this.

Context

Discussed with @mrsimonemms in Slack; filing here for tracking per that conversation.

tl;dr I need to have a think about the best way to implement this.

The inline refactor avoids the performance cost of creating large numbers of child workflows, but it also removes an important execution boundary.

Previously, a structural task had its own workflow history and could Continue-As-New independently. With fully inline execution, the entire structural task must currently fit within one workflow run. That is a behavioural regression for large loops or deeply nested structures.

A possible compromise is to create one child workflow for the outermost structural task, then execute all nested structural tasks inline within it. For example:

root workflow
└── for child workflow
    └── try inline
        └── fork inline

This preserves an independent history and Continue-As-New boundary while avoiding a child workflow for every loop iteration, fork branch or nested structure.

The remaining design question is therefore not simply child workflows versus inline execution, but the appropriate granularity of the workflow boundary.

@mrsimonemms

@mrsimonemms Thanks for digging into this and for the compromise, the outer-boundary idea is a solid way to get Continue-As-New safety back without paying a child workflow per iteration.

One thing worth flagging against the original report though: promoting the outermost structural task to a child workflow partially reopens the fan-in contention this issue was filed over.

Given that, would it make sense to make the child-workflow wrapper opt-in per task rather than automatic for every outermost structural task? That way:

The default stays fully inline, zero child-workflow fan-in, which is what actually fixed the contention we reported.
Authors who know a specific loop will run long enough to risk history limits can explicitly opt that one into its own child workflow, paying the isolation cost only where it's needed.

Curious to hear your thoughts on this approach

Short answer is I don't know at this stage. I'm happy to accept proposals on this, but I need to dedicate some time to this issue.

Anyone out there who has ideas, please add to this thread.

Will update from my side when I've had a bright idea. It feels like there's a solution, but I think we need to go back to first principles on this.

@mrsimonemms

Hi @mrsimonemms, thanks for looking into this. We’re relying on this as part of our migration to Temporal for a few workflows, so I wanted to check if you have a sense of whether this is something you expect to get back to soon, or if you’d prefer us to treat it as blocked for now and plan around it? 🙏

@shlomit-branch

Hi @shlomit-branch, treat this as blocked for the time being. From the way it looks, its mutually-exclusive with #469

@mrsimonemms

Hi @mrsimonemms, since #550 already exists and reconciles the try/catch path-threading with #469 , would it help if I picked up finishing that PR? Specifically extending the same inline approach to for, which #550 doesn't cover yet. Let me know if that's the right place to focus, or if the mutual-exclusivity concern with #469 is really about something else that needs scoping first.

Yes, very happy for you to pick up #550.

@mrsimonemms

Hi @mrsimonemms, I’m going to pick this up this week since optimizing these workflows has become more urgent for us.

Hopefully the implementation is simpler than expected. Let me know if you have any concerns or design preferences before I get started.

@ShlomitSibony

Great, thanks @ShlomitSibony. I don't have any thoughts at the moment - I recall it being more complex than I expected at the time, but I think I overcomplicated it in my mind.

@mrsimonemms

Hi @mrsimonemms, I've picked up #550 and extended it to for, in a single PR. Before I open it (branch), I'd like your view on one design question.

The question

Moving for iterations and try/catch bodies inline changes the commands a workflow issues. A workflow that is already running, whose history contains the old child workflows, would fail replay with a non-determinism error after the upgrade. The Codex review on #550 flagged this as High.

To avoid it I added a workflow.GetVersion gate:

New executions record a version marker and run inline.
Histories without the marker keep starting the old child workflows.
Those legacy child workflows stay registered on the worker.

I checked this with replay tests against real histories captured from upstream/main and from the new code. Without the gate, replaying the old history fails with a TMPRL1100 non-determinism error.

What the workaround costs

Three extra files plus the legacy branches in for.go and try.go.
Each for and try body is built twice at startup.
One extra marker event per execution.

Is it worth keeping?

Keep it Nobody's running workflows break on upgrade. The legacy path can stay, or be removed in a later release once users have moved past the older versions.
Drop it and document that users must let workflows that started before the upgrade finish, or Continue-As-New, before upgrading. This is simpler code, but it is a breaking upgrade for anyone with long-running workflows.

Also worth flagging

During a rolling deploy or a rollback, an older worker has no inline path, so I expect it would fail to replay a history started by a newer worker. I haven't tested that, but I'll note it in the PR.

@ShlomitSibony

Thanks @ShlomitSibony. Historically, I've treated these types of changes as breaking changes. Zigflow is versioned and I don't think we should be in the business of versioning workflows AND the application. As long as we communicate it as a breaking change (and create a new minor version - perfectly valid in v0.x.x), I think that's enough.

I'd drop it, but I'm open to other opinions from contributors/maintainers.

/cc @zigflow/maintainers

@ShlomitSibony

I'd drop it too. Versioning the application and every workflow on top of it is a lot of permanent code for something Temporal already handles: the upgrade notes can point users at Worker Versioning, which pins each running workflow to the worker build it started on. That also covers the rolling-deploy and rollback case @ShlomitSibony flagged, since older executions stay on older workers. For anyone not using it, the notes should say plainly: let workflows started before the upgrade finish (or Continue-As-New) first.

Two related points:

For consistency, fix: resolve activity inputs in the workflow before scheduling #615 added a GetVersion gate (zigflow.resolve-activity-inputs). If the policy is "breaking change, not workflow versioning", I'm happy to remove that gate in the same release so we have one rule. I can open that PR once this lands.
With for fully inline, a large loop's whole history now lives in one run. It might be worth a note in the docs about history limits, or guidance to Continue-As-New for very large iterations, so the old child-workflow boundary isn't missed silently.

@YJ-928

Thanks @YJ-928. I agree about removing the GetVersion gate in #615 (and any others).

I think I probably need to write a policy on potential breaking changes so all contributors and users have a single source of truth. The tl;dr is:

in < v1.0.0: breaking changes can be made in minor versions and will be specified in the release docs
in >= v1.0.0: breaking changes can only be made in major versions

It's one of the reasons why v1 has taken a lot longer to deploy than anticipated, because I wanted to make sure we get all these wrinkles out

@mrsimonemms

Thanks @mrsimonemms, that policy is clear and gives contributors one rule to follow.

I'll open one PR that removes both GetVersion gates, zigflow.resolve-activity-inputs (from #615) and zigflow.per-task-activity-aliases, marked as a breaking change (! in the title), with a note for the release docs.

Would you like it to land as soon as it's reviewed, or wait so it ships in the same release as the inline for/try change? If it helps, I can also draft the breaking-change policy as a docs PR for you to edit.

Would you like it to land as soon as it's reviewed

I don't mind when the PR is merged, but I think it makes sense waiting until both are in main before doing a new release to avoid having two breaking changes

Happy for input on the policy document. I don't think it'll need to be especially long, as it basically follows the semantic release document. Its main purpose is so we all know where we stand as users and contributors

@YJ-928

Thanks for the input

guidance to Continue-As-New for very large iterations

@YJ-928 I didn't write "Continue-As-New" as loop advice because a loop can't trigger it from the DSL.

The PR is up for review: #624

Waiting on your gate removal PR

@ShlomitSibony

Thanks @ShlomitSibony. The gate removal is up as #626. It's independent of #624: there's no file overlap, and I merged both locally with build, tests, race and lint all passing, so they can land in either order. I've added the ai-review label to #624 and I'm reviewing it now.

refactor!: remove the GetVersion gates for activity inputs and activi…

Sunbelt Computer Software

PL/B Language Development and Support

Navigation Menu

Repository navigation

`for` and `try` blocks compile to child workflows — fan-in read/lock amplification on hot parents at scale #525

Summary

Observed impact (production, Temporal)

Request

Context

The question

What the workaround costs

Is it worth keeping?

Also worth flagging

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Sunbelt Computer Software

PL/B Language Development and Support

for and try blocks compile to child workflows — fan-in read/lock amplification on hot parents at scale #525

Description

Summary

Observed impact (production, Temporal)

Request

Context

Activity

mrsimonemms commented on Jul 28, 2026

ShlomitSibony commented on Jul 28, 2026

mrsimonemms commented on Jul 28, 2026

shlomit-branch commented on Aug 21, 2026

mrsimonemms commented on Sep 4, 2026

ShlomitSibony commented on Sep 8, 2026

mrsimonemms commented on Sep 8, 2026

ShlomitSibony commented on Oct 5, 2026

mrsimonemms commented on Oct 6, 2026

ShlomitSibony commented on Oct 9, 2026

The question

What the workaround costs

Is it worth keeping?

Also worth flagging

mrsimonemms commented on Oct 10, 2026

YJ-928 commented on Oct 11, 2026

mrsimonemms commented on Oct 11, 2026

YJ-928 commented on Oct 11, 2026

mrsimonemms commented on Oct 11, 2026

ShlomitSibony commented on Oct 11, 2026

YJ-928 commented on Oct 12, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`for` and `try` blocks compile to child workflows — fan-in read/lock amplification on hot parents at scale #525