PeriodicScheduler re-arms itself after stop(), so measurements can outlive tracker.stop() · Issue #1308 · mlco2/codecarbon · GitHub
Skip to content

PeriodicScheduler re-arms itself after stop(), so measurements can outlive tracker.stop() #1308

Open

Description

@davidberenstein1957

Problem

PeriodicScheduler can keep running (and can run its payload) after stop() has returned. In the tracker this means _measure_power_and_energy may execute after EmissionsTracker.stop() has already written the final row and exited the output handlers — producing an extra measurement, a post-run live_out push (Prometheus/API handlers), or an exception logged after "Done!".

Reproduction

import threading, time
from codecarbon.external.scheduler import PeriodicScheduler

calls = []
s = PeriodicScheduler(0.05, lambda: calls.append(1))
gate, orig = threading.Event(), s.start
# widen the (real, but narrow) window between the timer firing and _run reaching start()
s.start = lambda from_run=False: (from_run and gate.wait(2), orig(from_run=from_run))[1]
orig(from_run=False)
time.sleep(0.2)   # timer fired, _run is blocked before start()
s.stop()
gate.set()
time.sleep(0.4)
print(s._stopped, len(calls))   # -> False, and the count keeps climbing forever

Root cause

codecarbon/external/scheduler.py:26-42. _run() calls start(from_run=True), and the from_run flag deliberately bypasses the _stopped check, so a fired timer always arms its successor:

if from_run or self._stopped:
    self._stopped = False
    self._timer = Timer(self.interval, self._run)

If stop() lands between the timer firing and _run acquiring the lock, stop() cancels an already-fired (uncancellable) timer and flips _stopped = True; _run then sets it back to False and arms a timer nobody holds a reference to — the scheduler becomes unstoppable. Separately, stop() reads _stopped outside the lock (scheduler.py:48), and _run never re-checks the flag before invoking the payload, so the payload runs to completion after stop() returned. EmissionsTracker.stop() sets self._scheduler = None immediately after (emissions_tracker.py:909-914), so that late payload mutates tracker state whose handlers have already been exit()ed (emissions_tracker.py:937-938).

Expected vs actual

Expected: after stop() returns, no further payload invocations and no re-arming.
Actual: the reschedule path is not serialized against the stop path over _stopped, so both a stray timer and a late payload invocation can survive stop().

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions