Repository navigation
Conversation
Adds an IdleShutdown consumer bootstep. When worker_max_idle is set and the worker has had no running, reserved or scheduled tasks for that many seconds, it does a warm shutdown with exit code 0 so a process manager can start a fresh worker. Closes celery#9031
MehrazRumman
left a comment
There was a problem hiding this comment.
Checked at 6584f36. t/unit/worker/test_consumer.py passes (133 passed). state.should_stop = EX_OK (0) works because maybe_shutdown() checks is not None. The CI workflows for this commit are still waiting for maintainer approval.
A busy worker can be shut down as idle
check() only samples active_requests / reserved_requests / scheduled_requests once a second. A task that is received and finished between two checks never resets last_busy. So a worker running a steady stream of short tasks looks idle to the step. I drove the step with the real worker.state functions, using one short task between each pair of checks:
shut down at t+10s after completing 10 tasks (one per second) # worker_max_idle = 10
The worker was never idle, and it still exited. Tracking completed work as well as the current snapshot fixes this. state.all_total_count[0] is incremented in task_accepted():
def check(self):
count = state.all_total_count[0]
if (count != self._last_count or state.active_requests
or state.reserved_requests or state.scheduled_requests):
self._last_count = count
self.last_busy = monotonic()
elif monotonic() - self.last_busy >= self.max_idle:
...Initialise _last_count in start(), and add a test with a task that starts and finishes between two checks, which is the case the current tests miss.
Smaller points
- The neighbouring
worker_max_tasks_per_childandworker_max_memory_per_childsettings also have--max-tasks-per-child/--max-memory-per-childflags oncelery worker. A matching--max-idlewould be consistent, and it is handy in a container command line. docs/userguide/configuration.rst: the line after the example contains only spaces.- Once the busy-detection fix is in, the docs could say that the timer resets on any task activity, not only when tasks are present at a check.
A task that started and finished between two checks did not reset the idle timer, so a worker running a steady stream of short tasks could be shut down. Also compare state.all_total_count with the previous check, add a test for this case, and update the docs.
@myayush did you fixed them as well? also I commented on the original issue |

Closes #9031
Adds a
worker_max_idlesetting (seconds, off by default). If the worker has no running, reserved or scheduled tasks for that long, it does a normal warm shutdown with exit code 0, so a process manager (systemd, Kubernetes) can start a fresh one. This lets idle workers give memory back.It's a small consumer bootstep (
celery/worker/consumer/idle.py) that checks once a second on the worker's timer, so it works with every pool.Added unit tests in
t/unit/worker/test_consumer.pyand a docs entry in the configuration guide.