Coordinator: Stop/StopByName racing a concurrent Restart on the same dataflow silently loses — dora stop reports success but a new incarnation keeps running · Issue #3113 · dora-rs/dora · GitHub
Skip to content

Coordinator: Stop/StopByName racing a concurrent Restart on the same dataflow silently loses — dora stop reports success but a new incarnation keeps running #3113

Description

@GuTS805

Problem

initiate_restart (binaries/coordinator/src/lib.rs:4643) sends StopDataflow
to the daemon(s) and registers a PendingRestart under the dataflow's UUID,
but does not remove the entry from running_dataflows — it stays present
until DataflowFinishedOnDaemon fires.

The ControlRequest::Stop / StopByName handlers (binaries/coordinator/src/lib.rs:891-996)
only take the "already stopped" fast path when running_dataflows no longer
contains the UUID. Since a pending restart leaves the entry in place, a
concurrent Stop falls through to stop_dataflow() again, succeeds (the
daemon's StopDataflow handler is idempotent), and queues the caller's
reply_sender onto dataflow.stop_reply_senders.

When DataflowFinishedOnDaemon finally arrives, the coordinator resolves the
pending restart first (spawns a brand-new dataflow under a new UUID,
replaces the running-dataflows entry, tells the restart caller
DataflowRestarted), and only afterward resolves stop_reply_senders with
Ok(DataflowStopped) — unconditionally, with no check that a restart just
superseded the stop. Nothing in the Stop/StopByName handlers checks
pending_restarts before proceeding.

Reproduction (100% reliable, 3/3 runs)

dora start dataflow.yml --detach     # -> uuid A
dora restart A &
dora stop    A &

Both commands report success (exit 0). But:

dora list
UUID    Name             Status    Nodes
B       driving-mongrel  Running   1     <- new incarnation still running!
A       driving-mongrel  Finished  0

dora stop A gave no indication the dataflow was still alive under a new
UUID. Reproduced identically across 3 independent coordinator/daemon
instances on different ports.

Why it matters

Breaks the CLI's stop contract silently. A script/automation that stops a
dataflow for maintenance can be left with a live orphaned incarnation and no
error surfaced anywhere — a correctness/trust bug, not just a UX nit.

Not a duplicate of

Proposed fix

In Stop/StopByName, check pending_restarts.contains_key(&dataflow_uuid)
before falling through to stop_dataflow(): either reject the stop with a
clear error, or cancel the pending restart (erroring its reply_sender) and
proceed with the stop. The single-threaded coordinator event loop makes this
an atomic decision within one turn.

QA tier

New integration/e2e test alongside daemon-reconnect-e2e.rs, driving a real
coordinator + daemon + node, firing Restart and Stop concurrently on the
same UUID, asserting the stop's semantics are honored (or the restart is
rejected) instead of silently losing.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions