fix: enforce terminal state transitions and serialize cancel check-then-act by ez-lbz · Pull Request #453 · a2aproject/a2a-dotnet · GitHub
Skip to content

fix: enforce terminal state transitions and serialize cancel check-then-act - #453

Open
ez-lbz wants to merge 8 commits into
a2aproject:mainfrom
ez-lbz:fix/state-machine-cancel-race
Open

ez-lbz wants to merge 8 commits into
a2aproject:mainfrom
ez-lbz:fix/state-machine-cancel-race

Conversation

@ez-lbz

@ez-lbz ez-lbz commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

What changed

1. Terminal state transitions are enforced in TaskProjection

Problem: TaskProjection.ApplyStatus unconditionally overwrote current.Status, and ApplyMessage unconditionally appended to history. A task in a terminal state (COMPLETED/CANCELED/FAILED/REJECTED) could be overwritten back to SUBMITTED or accept further messages via direct state assignment, whenever events bypassed the request-level terminal checks (e.g. a misbehaving agent handler emitting events after a terminal status).

Fix (src/A2A/Server/TaskProjection.cs):

  • Added GuardNotTerminal, called from both ApplyStatus and ApplyMessage.
  • Terminal tasks now throw A2AException with A2AErrorCode.UnsupportedOperation (-32004, the same code A2AServer uses for terminal-state rejections) instead of being mutated.
  • Artifact updates on terminal tasks remain allowed (the projection only guards status/message mutations).

2. CancelTaskAsync check-then-act is serialized under the per-task lock

Problem: CancelTaskAsync read the task state and checked terminal status with no lock between the check and the state transition. Two concurrent cancel requests could both pass the check and both run the cancel handler (TOCTOU), and cancel could race with message/send mutations.

Fix (src/A2A/Server/A2AServer.cs):

  • The entire cancel operation — terminal check, background-CTS signalling, cancel-handler event application, and final state read — now runs under _notifier.AcquireTaskLockAsync(request.Id), the same per-task lock ApplyEventAsync and SubscribeToTaskAsync use.
  • The loser of a concurrent-cancel race re-reads the now-terminal task and fails with TaskNotCancelable instead of double-cancelling.
  • ApplyEventAsync was refactored into a lock-acquiring wrapper plus ApplyEventUnderLockAsync (the locked body), so the cancel path can apply events without re-acquiring the non-reentrant per-task semaphore (which would deadlock).

Testing

  • dotnet test tests/A2A.UnitTests --framework net8.0 — 434 passed, 0 failed (baseline 420, +14 new regression tests: status-overwrite and message-append rejection for all four terminal states, same-terminal-state re-emission rejection, artifact-update tolerance on terminal tasks, and a two-concurrent-cancel test asserting exactly one succeeds while the other gets TaskNotCancelable).
  • dotnet test tests/A2A.AspNetCore.UnitTests --framework net8.0 — 88 passed, 0 failed (unchanged).
  • Behavior change: any agent handler that emits a status update or message after a task reaches a terminal state now gets an UnsupportedOperation error (previously silently applied). Well-behaved handlers are unaffected; cancel still transitions WORKING → CANCELED as before.

@ez-lbz
ez-lbz force-pushed the fix/state-machine-cancel-race branch from 1c26cc0 to 3c8fd2c Compare August 11, 2026 13:37

This comment was marked as resolved.

@chopmob-cloud

Copy link
Copy Markdown
Contributor

@tracyboehrer

Copy link
Copy Markdown
Collaborator

We have rights to resolve the conflicts and resolve the noted issues. Depending on if ez-lbz would rather run with it instead.

@ez-lbz

ez-lbz commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

Thanks — please go ahead, the PR is yours to run with. maintainer_can_modify is enabled on fix/state-machine-cancel-race (head 3ad7e3f), so you can push to the branch directly or lift the commits into a new one, whichever is easier; I will not push to it from here. Two notes for the handoff. Gap 1 — the full-task replacement path in TaskProjection.Apply returning streamEvent.Task before GuardNotTerminal is reached — is squarely inside this PR's scope and is what I would have fixed next. Gaps 2 and 3 are older than this PR, as @chopmob-cloud noted: _taskLocks.TryRemove in ChannelEventNotifier.RemoveChannel runs whenever the last subscriber disconnects, and the background CTS registration in SendMessageAsync sits outside the notifier lock that CancelTaskAsync now holds, so both may be cleaner as their own change rather than folded in here. Happy to review once you have a branch, or to answer anything about the original intent.

tracyboehrer and others added 2 commits October 8, 2026 08:12
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

This comment was marked as resolved.

tracyboehrer and others added 2 commits October 8, 2026 10:08
@tracyboehrer

Copy link
Copy Markdown
Collaborator

@ez-lbz @chopmob-cloud — could you both please consider the following detailed race-condition review before this is merged?

I would not merge the current revision yet. I found one critical race introduced by the latest cleanup design, three additional high-confidence concurrency bugs, and several secondary issues.

# Severity Location Finding Confidence
1 Critical A2AServer.cs:207-208, 229-246, 707-735 Streaming reservation can be released before the final event is persisted 10/10
2 High A2AServer.cs:224-244, 258-316 Client cancellation can permanently discard an already-dequeued stream event 10/10
3 High ChannelEventNotifier.cs:42-70 Removing the last subscriber can orphan a concurrently created subscriber 9/10
4 High A2AServer.cs:75-126, 376-409, 647-678 Blocking sends have no operation reservation and cannot be canceled by CancelTask 9/10
5 High TaskProjection.cs:60-65, 113-159 Artifact updates can modify a terminal task 10/10
6 Medium A2AServer.cs:306-315 An old streaming drain can remove tracking for a newer drain 8/10
7 Medium ChannelEventNotifier.cs:12-13, ITaskStore.cs:17-25 Correctness is only process-local; multiple server instances can overwrite each other 10/10

1. Streaming ownership is released before persistence completes

ReleaseBackgroundCancellationWhenCompletedAsync waits for eventQueue.Completion and agentTask, but channel completion only means the writer completed and the final item was dequeued. It does not mean the stream consumer finished ApplyEventAsync.

A valid interleaving is:

  1. The handler writes its final InputRequired or Completed event and finishes.
  2. The stream enumerator dequeues that event.
  3. eventQueue.Completion completes.
  4. The release task acquires the task lock and removes _backgroundCancellations[taskId].
  5. A continuation reserves the task and reads the old state.
  6. The original stream then persists its final event.

This permits a new handler to start before the preceding operation's final state is persisted. The new GivenStreamingHandlerFinishesAtInputRequired... test cannot detect this because it invokes the continuation only after MoveNextAsync has returned, which is after ApplyEventAsync.

The durable fix is a server-owned event pump that drains the agent queue, persists each event, forwards the persisted event to the client-facing channel, and releases the reservation only after that pump completes.

2. Stream cancellation loses the current event

The streaming loop dequeues a response and then calls ApplyEventAsync using the HTTP/client cancellation token. If the client disconnects while persistence is waiting for the task lock or store, the response has already left the queue. The finally drain processes only the remaining items, so the dequeued response is permanently lost.

That conflicts with reconnect/resubscription semantics because GetTask and a later subscription cannot recover an event generated by the handler. Once an event has been dequeued, its persistence should use a server-lifetime token rather than the client connection token. The server-owned persistence pump above would address this as well.

3. Concurrent subscriber creation can be orphaned

RemoveChannel removes the dictionary entry while holding only lock (set). A failing interleaving is:

  1. Subscriber A starts removing the last channel and locks the subscriber set.
  2. Subscriber B obtains that same set from _subscribers, then waits for its lock.
  3. A removes _subscribers[taskId] and unlocks.
  4. B adds its channel to the now-orphaned set.
  5. Notify looks in _subscribers, finds nothing, and drops subsequent events.

B can then wait forever because its channel is neither notified nor completed. The existing concurrency test checks only for exceptions/deadlocks, not delivery. Creation and last-channel removal need common per-task serialization or reference-counted subscriber-set retirement.

4. Blocking sends are outside the operation-reservation model

Only return-immediately and streaming requests reserve an operation. Default blocking sends do not reserve the task, use only the request token, are absent from _backgroundCancellations, and therefore cannot be stopped by CancelTaskAsync.

Consequences include:

  • With AutoAppendHistory=false, cancellation can persist a terminal state after context resolution but before the stale terminal check and handler start.
  • Blocking sends can overlap other blocking, streaming, or return-immediately operations for the same task.
  • CancelTask can store Canceled while the original handler continues external work because its execution token is unrelated to the per-task cancellation source.

Every task-associated send should use the same operation lease. Reservation, terminal recheck, and history append should occur atomically, and blocking handlers should receive a linked caller/per-task cancellation token.

5. Terminal tasks still accept artifact mutations

ApplyStatus, ApplyMessage, and full-task replacement call GuardNotTerminal; ApplyArtifact does not. A handler can emit Completed followed by a TaskArtifactUpdateEvent, and the artifact will be persisted onto the completed task.

This conflicts with terminal-task immutability and the rule that no further stream updates follow a terminal state. Artifact application should use the same terminal guard, with regression tests for all terminal states.

6. Old streaming drains can remove newer tracking

Streaming cleanup releases the reservation and then calls _backgroundTasks.TryRemove(taskId, out _). In that window, a new operation can reserve the task and add its own drain under the same key. The old drain can then remove the new tracking entry, allowing DisposeAsync to stop tracking active work.

Removal needs to be identity/ownership-safe rather than key-only. A single operation object holding the CTS, handler task, persistence task, and drain completion would simplify this ownership considerably.

7. Locking does not extend across server instances

All atomicity depends on a process-local ChannelEventNotifier, while ITaskStore exposes unconditional read and save operations with no version/CAS contract. Two server instances sharing one durable store can both read Working, independently derive updates, and overwrite one another.

If the SDK supports only one server instance per task partition, that restriction needs documentation. Otherwise the store contract needs optimistic concurrency or conditional writes.

A2A specification comparison

The current implementation can violate the latest lifecycle expectations because generated events may be lost after streaming disconnection, a continuation can begin before the preceding final state is durable, terminal tasks can receive artifact updates, cancellation does not stop blocking execution, and subscriber reconnection can silently lose live events.

Relevant references:

@tracyboehrer

Copy link
Copy Markdown
Collaborator

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants