feat(worker): add worker_max_idle to shut down idle workers by myayush · Pull Request #10871 · celery/celery · GitHub
Skip to content

feat(worker): add worker_max_idle to shut down idle workers - #10871

Open
myayush wants to merge 6 commits into
celery:mainfrom
myayush:worker-max-idle-9031
Open

myayush wants to merge 6 commits into
celery:mainfrom
myayush:worker-max-idle-9031

Conversation

@myayush

@myayush myayush commented Oct 16, 2026

Copy link
Copy Markdown

Closes #9031

Adds a worker_max_idle setting (seconds, off by default). If the worker has no running, reserved or scheduled tasks for that long, it does a normal warm shutdown with exit code 0, so a process manager (systemd, Kubernetes) can start a fresh one. This lets idle workers give memory back.

It's a small consumer bootstep (celery/worker/consumer/idle.py) that checks once a second on the worker's timer, so it works with every pool.

Added unit tests in t/unit/worker/test_consumer.py and a docs entry in the configuration guide.

Adds an IdleShutdown consumer bootstep. When worker_max_idle is set and the worker has had no running, reserved or scheduled tasks for that many seconds, it does a warm shutdown with exit code 0 so a process manager can start a fresh worker.

Closes celery#9031

@MehrazRumman MehrazRumman left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked at 6584f36. t/unit/worker/test_consumer.py passes (133 passed). state.should_stop = EX_OK (0) works because maybe_shutdown() checks is not None. The CI workflows for this commit are still waiting for maintainer approval.

A busy worker can be shut down as idle

check() only samples active_requests / reserved_requests / scheduled_requests once a second. A task that is received and finished between two checks never resets last_busy. So a worker running a steady stream of short tasks looks idle to the step. I drove the step with the real worker.state functions, using one short task between each pair of checks:

shut down at t+10s after completing 10 tasks (one per second)   # worker_max_idle = 10

The worker was never idle, and it still exited. Tracking completed work as well as the current snapshot fixes this. state.all_total_count[0] is incremented in task_accepted():

def check(self):
    count = state.all_total_count[0]
    if (count != self._last_count or state.active_requests
            or state.reserved_requests or state.scheduled_requests):
        self._last_count = count
        self.last_busy = monotonic()
    elif monotonic() - self.last_busy >= self.max_idle:
        ...

Initialise _last_count in start(), and add a test with a task that starts and finishes between two checks, which is the case the current tests miss.

Smaller points

  • The neighbouring worker_max_tasks_per_child and worker_max_memory_per_child settings also have --max-tasks-per-child / --max-memory-per-child flags on celery worker. A matching --max-idle would be consistent, and it is handy in a container command line.
  • docs/userguide/configuration.rst: the line after the example contains only spaces.
  • Once the busy-detection fix is in, the docs could say that the timer resets on any task activity, not only when tasks are present at a check.

auvipy and others added 2 commits October 17, 2026 17:19
A task that started and finished between two checks did not reset the idle timer, so a worker running a steady stream of short tasks could be shut down. Also compare state.all_total_count with the previous check, add a test for this case, and update the docs.
@myayush

myayush commented Oct 17, 2026

Copy link
Copy Markdown
Author

@auvipy
auvipy self-requested a review October 17, 2026 14:18
@auvipy

auvipy commented Oct 17, 2026 •

Copy link
Copy Markdown
Member

Smaller points

  • The neighbouring worker_max_tasks_per_child and worker_max_memory_per_child settings also have --max-tasks-per-child / --max-memory-per-child flags on celery worker. A matching --max-idle would be consistent, and it is handy in a container command line.
  • docs/userguide/configuration.rst: the line after the example contains only spaces.
  • Once the busy-detection fix is in, the docs could say that the timer resets on any task activity, not only when tasks are present at a check.

@myayush did you fixed them as well? also I commented on the original issue

@myayush

myayush commented Oct 17, 2026

Copy link
Copy Markdown
Author

@auvipy auvipy added this to the 5.8.0 milestone Oct 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Restart worker after idle period

3 participants