Problem
initiate_restart (binaries/coordinator/src/lib.rs:4643) sends StopDataflow
to the daemon(s) and registers a PendingRestart under the dataflow's UUID,
but does not remove the entry from running_dataflows — it stays present
until DataflowFinishedOnDaemon fires.
The ControlRequest::Stop / StopByName handlers (binaries/coordinator/src/lib.rs:891-996)
only take the "already stopped" fast path when running_dataflows no longer
contains the UUID. Since a pending restart leaves the entry in place, a
concurrent Stop falls through to stop_dataflow() again, succeeds (the
daemon's StopDataflow handler is idempotent), and queues the caller's
reply_sender onto dataflow.stop_reply_senders.
When DataflowFinishedOnDaemon finally arrives, the coordinator resolves the
pending restart first (spawns a brand-new dataflow under a new UUID,
replaces the running-dataflows entry, tells the restart caller
DataflowRestarted), and only afterward resolves stop_reply_senders with
Ok(DataflowStopped) — unconditionally, with no check that a restart just
superseded the stop. Nothing in the Stop/StopByName handlers checks
pending_restarts before proceeding.
Reproduction (100% reliable, 3/3 runs)
dora start dataflow.yml --detach # -> uuid A
dora restart A &
dora stop A &
Both commands report success (exit 0). But:
dora list
UUID Name Status Nodes
B driving-mongrel Running 1 <- new incarnation still running!
A driving-mongrel Finished 0
dora stop A gave no indication the dataflow was still alive under a new
UUID. Reproduced identically across 3 independent coordinator/daemon
instances on different ports.
Why it matters
Breaks the CLI's stop contract silently. A script/automation that stops a
dataflow for maintenance can be left with a live orphaned incarnation and no
error surfaced anywhere — a correctness/trust bug, not just a UX nit.
Not a duplicate of
Proposed fix
In Stop/StopByName, check pending_restarts.contains_key(&dataflow_uuid)
before falling through to stop_dataflow(): either reject the stop with a
clear error, or cancel the pending restart (erroring its reply_sender) and
proceed with the stop. The single-threaded coordinator event loop makes this
an atomic decision within one turn.
QA tier
New integration/e2e test alongside daemon-reconnect-e2e.rs, driving a real
coordinator + daemon + node, firing Restart and Stop concurrently on the
same UUID, asserting the stop's semantics are honored (or the restart is
rejected) instead of silently losing.
Problem
initiate_restart(binaries/coordinator/src/lib.rs:4643) sendsStopDataflowto the daemon(s) and registers a
PendingRestartunder the dataflow's UUID,but does not remove the entry from
running_dataflows— it stays presentuntil
DataflowFinishedOnDaemonfires.The
ControlRequest::Stop/StopByNamehandlers (binaries/coordinator/src/lib.rs:891-996)only take the "already stopped" fast path when
running_dataflowsno longercontains the UUID. Since a pending restart leaves the entry in place, a
concurrent
Stopfalls through tostop_dataflow()again, succeeds (thedaemon's
StopDataflowhandler is idempotent), and queues the caller'sreply_senderontodataflow.stop_reply_senders.When
DataflowFinishedOnDaemonfinally arrives, the coordinator resolves thepending restart first (spawns a brand-new dataflow under a new UUID,
replaces the running-dataflows entry, tells the restart caller
DataflowRestarted), and only afterward resolvesstop_reply_senderswithOk(DataflowStopped)— unconditionally, with no check that a restart justsuperseded the stop. Nothing in the
Stop/StopByNamehandlers checkspending_restartsbefore proceeding.Reproduction (100% reliable, 3/3 runs)
Both commands report success (exit 0). But:
dora stop Agave no indication the dataflow was still alive under a newUUID. Reproduced identically across 3 independent coordinator/daemon
instances on different ports.
Why it matters
Breaks the CLI's stop contract silently. A script/automation that stops a
dataflow for maintenance can be left with a live orphaned incarnation and no
error surfaced anywhere — a correctness/trust bug, not just a UX nit.
Not a duplicate of
SpawnedNodeResult{restart:true}leaks the node inrunning_nodes→ dataflow never finishes (stop-vs-restart race and respawn-failure both hit it) #2936 — daemon-level node restart-loop leak (single-daemon, node lifecycle,not this
Restartcontrol-request path)dataflow.start()on an already-stopping dataflow (barrier-completed-by-node-death start isn't gated onstop_sent) #3053 —dataflow.start()on an already-stopping dataflow via the startupbarrier — different trigger mechanism
PendingRestarton daemon death mid-restart,already fixed; doesn't cover concurrent-
Stop-vs-Restartdaemon-reconnect-e2e.rs,fault-tolerance-e2e.rs,node-lifecycle-e2e.rs,ws_control_tests.rs) exercisesRestartracingStopon the same UUID.Proposed fix
In
Stop/StopByName, checkpending_restarts.contains_key(&dataflow_uuid)before falling through to
stop_dataflow(): either reject the stop with aclear error, or cancel the pending restart (erroring its
reply_sender) andproceed with the stop. The single-threaded coordinator event loop makes this
an atomic decision within one turn.
QA tier
New integration/e2e test alongside
daemon-reconnect-e2e.rs, driving a realcoordinator + daemon + node, firing
RestartandStopconcurrently on thesame UUID, asserting the stop's semantics are honored (or the restart is
rejected) instead of silently losing.