Activity inputs should be expression-evaluated before scheduling activities · Issue #462 · zigflow/zigflow · GitHub
Skip to content

Activity inputs should be expression-evaluated before scheduling activities #462

Closed
#615
Closed
Activity inputs should be expression-evaluated before scheduling activities#462
#615
Labels
bugSomething isn't workinggoPull requests that update go codegood first issueGood for newcomershelp wantedExtra attention is needednever-stalev1Required to put Zigflow into GA

Description

@mrsimonemms
mrsimonemms
opened on Jun 1, 2026

Summary

Activity-backed tasks currently receive unresolved expression strings in their activity input payloads.

For example, this workflow:

do:
  - fork:
      fork:
        compete: false
        branches:
          - grpc:
              call: grpc
              with:
                proto:
                  endpoint: file:///go/app/examples/external-calls/grpc/basic/proto/basic/v1/basic.proto
                service:
                  name: providers.v1.BasicService
                  host: grpc
                  port: 3000
                method: Command1
                arguments:
                  input: ${ $env.GRPC_INPUT }

results in the gRPC activity receiving:

{
  "arguments": {
    "input": "${ $env.GRPC_INPUT }"
  }
}

The expression is then evaluated by the activity implementation rather than by the workflow before the activity is scheduled.

Why this is a problem

This creates a few rough edges:

  1. Activity inputs in Temporal history are harder to inspect

    The recorded activity input contains the unresolved expression rather than the concrete value passed to the external system.

    That makes debugging harder because it is not obvious what value the activity actually used.

  2. Expression validation may miss activity input expressions

    If expressions inside with payloads are not collected and validated before execution, then invalid, unsupported or non-deterministic expressions may only fail at runtime.

  3. Expression evaluation responsibility is split

    Some expression handling happens in the workflow layer, but activity-backed calls may also perform their own interpolation.

    This makes behaviour harder to reason about and risks inconsistent handling across activity types such as http, grpc, call and run.

  4. Potential determinism gap

    In principle, a non-idempotent expression could be passed into an activity payload and evaluated outside the normal deterministic validation path.

    This may be unlikely in common usage, but it weakens the boundary Zigflow is trying to enforce.

Expected behaviour

Before scheduling any activity-backed task, Zigflow should evaluate expressions in the task's with payload using the workflow expression context.

The activity should receive a fully resolved input payload.

For example, if:

arguments:
  input: ${ $env.GRPC_INPUT }

and GRPC_INPUT=hello, the activity input should contain:

{
  "arguments": {
    "input": "hello"
  }
}

not:

{
  "arguments": {
    "input": "${ $env.GRPC_INPUT }"
  }
}

Scope

This likely applies to most or all activity-backed task types, including:

  • http
  • grpc
  • call
  • run
  • any other task type that schedules a Temporal activity using a with payload

Suggested implementation

Move expression evaluation for activity input payloads into the workflow/task execution layer, before the activity is scheduled.

Activity implementations should receive concrete resolved input and should not need to perform generic Zigflow expression interpolation themselves.

The implementation should also ensure expressions inside activity with payloads are included in expression collection and determinism validation.

Acceptance criteria

  • Expressions inside activity with payloads are collected during validation.
  • Invalid expressions inside activity with payloads fail zigflow validate.
  • Non-deterministic expressions inside activity with payloads are rejected where appropriate.
  • Activity inputs recorded in Temporal history contain resolved values, not unresolved ${ ... } strings.
  • Activity implementations no longer need to perform generic expression interpolation on their own input payloads.
  • Existing examples using environment-backed activity inputs continue to work.
  • Tests cover at least one grpc and one http activity input expression.

Activity

  1. mrsimonemms
    added
    bugSomething isn't working
    help wantedExtra attention is needed
    good first issueGood for newcomers
    never-stale
    goPull requests that update go code
    v1Required to put Zigflow into GA
    on Jun 1, 2026

  2. mrsimonemms commented on Jun 16, 2026

    @mrsimonemms
    mrsimonemms
    on Jun 16, 2026
    CollaboratorAuthor

    I've got a working implementation for this, but I'm not entirely convinced about the long-term behaviour around ${ $data.activity.* }.

    The difficulty is that activity metadata only exists once an activity has actually been dispatched. Values such as ${ $data.activity.attempt } therefore cannot be resolved by the workflow before scheduling in the same way that ${ $env.* }, ${ $input.* } or ordinary ${ $data.* } expressions can.

    The current implementation preserves expressions that reference $data.activity.* and evaluates them inside the activity after activity metadata has been added to the state. This appears to work correctly, but it introduces a split evaluation model and I'm not yet comfortable that we've fully defined the boundary of what should be resolved before scheduling versus what should be preserved for activity-time evaluation.

    Leaving this open for now while I think through that contract a bit more.

  3. YJ-928 commented on Oct 5, 2026

    @YJ-928
    YJ-928
    on Oct 5, 2026
    Member

    Hi @mrsimonemms — I'd be happy to help get this one over the line if it's useful to you.

    On the $data.activity.* concern from #483: one way to keep it predictable would be to resolve, before scheduling, every activity input that doesn't touch activity state, and leave any input that does exactly as it is today, evaluated inside the activity. Deciding which inputs touch activity state from the parsed expression (gojq's parser) rather than a string prefix would also cover the forms the review flagged (${ $data | .activity.attempt }, ${ $data as $d | $d.activity.attempt }). Erring on the side of "references $data and an activity field anywhere → leave it to the activity" means a false positive only keeps today's behaviour, never resolves something too early.

    If that matches what you had in mind, I can rebase #483 onto current main, add tests for those jq forms, and update the docs — or open a fresh PR referencing #462, whichever you prefer. If you'd rather settle the semantics differently (e.g. not supporting $data.activity.* in activity inputs at all), happy to follow that instead.

  4. mrsimonemms commented on Oct 5, 2026

    @mrsimonemms
    mrsimonemms
    on Oct 5, 2026
    CollaboratorAuthor

    Yes please. I'll let you decide whether to improve #483 or create a new one (probably the best option). One of the reasons I abandoned this was that there weren't much in the way of unit/e2e tests at the time and that's been improved greatly now.

    Please shout if you want help/discuss thing

  5. YJ-928 commented on Oct 6, 2026

    @YJ-928
    YJ-928
    on Oct 6, 2026
    Member

    Thanks @mrsimonemms — I'll open a fresh PR referencing #462, building on the approach from #483: inputs that don't touch activity state are resolved before scheduling, anything referencing $data plus an activity field is left for the activity as today (detected from the parsed expression, so the forms from the #483 review are covered). It'll come with unit tests for the detection and the resolution, and an e2e test. I'll shout if anything needs discussing before then.

  6. YJ-928 commented on Oct 6, 2026

    @YJ-928
    YJ-928
    on Oct 6, 2026
    Member

    Following up after digging into the code — two findings worth aligning on before I open the PR:

    1. Validation is already covered on main. zigflow validate already rejects an invalid expression (${ $env.X | }) and a non-deterministic one (${ uuid }) inside grpc and http with payloads, so the PR can focus on resolving inputs before scheduling.

    2. Resolving workflow-side while the activity keeps its own pass evaluates data twice. With Evaluate expressions before executing activities #483's approach, a workflow input comment = "${ $env.API_TOKEN }" used as ${ $input.comment } resolves workflow-side to that literal string, then the activity's TraverseAndEvaluateObj evaluates it again and sends the token's value to the endpoint (I reproduced this against the Evaluate expressions before executing activities #483 branch). Today's single activity-side pass doesn't have this problem.

    My proposal: resolve the with payload workflow-side except expressions that reference activity state, and re-emit any resolved value that itself looks like ${ … } as a jq string literal (${ "…" }), so the activity's existing pass returns it unchanged. That keeps the single-evaluation guarantee without changing activity signatures (so mixed-version workers and in-flight runs stay safe), and only that rare case shows the quoted form in history. Scope would be http and grpc; call: activity already resolves its arguments, and run stays activity-side as #483 also left it.

    Does that match how you'd like it handled, or would you prefer a different approach to the double evaluation (for example, the activity only evaluating the expressions the workflow preserved)?

  7. mrsimonemms commented on Oct 7, 2026

    @mrsimonemms
    mrsimonemms
    on Oct 7, 2026
    CollaboratorAuthor

    My honest take is that I'm not convinced my original pass was correct and I think this should be looked at with fresh eyes (which I'm thrilled that you're doing). I'm happy with the validation (I think there's also an option to validate at runtime) and I'd prefer to only evaluate data once.

    Please approach this as a new PR and don't assume that anything I'd already done is anywhere close to correct.

  8. YJ-928 commented on Oct 8, 2026

    @YJ-928
    YJ-928
    on Oct 8, 2026
    Member

    Thanks, I took it from scratch: #615.

    Data is now evaluated once. The workflow resolves each activity's inputs before scheduling it. The activity evaluates only the expressions it is explicitly handed: those that read $data.activity, which cannot exist before an attempt runs, because one scheduled input serves every retry. There's no second pass over resolved data and no quoting.

    It covers http, grpc and run (shell, script, container). Open histories stay on today's behaviour through a version marker; I checked that by swapping the worker in the middle of a running execution. The two small behaviour changes are listed in the PR.

    If you'd rather not support $data.activity in activity inputs at all, the deferral can go and the activities stop evaluating entirely. Happy to change it either way.

  9. YJ-928
    added 2 commits that reference this issue on Oct 8, 2026

    fix: resolve activity inputs in the workflow before scheduling

    b4d7734

    fix: resolve activity inputs in the workflow before scheduling

    14d03a2
  10. mrsimonemms
    added a commit that references this issue on Oct 8, 2026

    fix: resolve activity inputs in the workflow before scheduling

    7247604
  11. mrsimonemms
    closed this as completedin #615on Oct 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinggoPull requests that update go codegood first issueGood for newcomershelp wantedExtra attention is needednever-stalev1Required to put Zigflow into GA

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions