Repository navigation
Restart worker after idle period #9031
Description
I'll work around this, so this is just a suggestion, to help the AI-comrads, as these often have long running tasks and hw/memory requirements.
- I have checked the issues list
for similar or identical feature requests. - I have checked the pull requests list
for existing proposed implementations of this feature. - I have checked the commit log
to find out if the same feature was already implemented in the
main branch. - I have included all related issues and possible duplicate issues
in this issue (If there are none, check this box anyway).
Related Issues and Possible Duplicates
- None
Related Issues
- None
Possible Duplicates
- None
Brief Summary
To leverage loading (AI/LLM-)models in (GPU-)memory and having memory available, an option similar to worker_max_tasks_per_child that kills a worker after an idle-time to free memory would be helpful. This naturally implements caching and loading if needed.
Design
Architectural Considerations
I couldn't quit grasp the celery comsumer-loop, but I'm going to mimic this behavior by spinning off a thread that checks idle-time and releases memory when too idle.
However, my best guess is that this is the consumer loop. My next guess is that it would have some sort of condition-variable to wait for jobs. I'd suggest adding a timeout there to sometimes check worker_max_idle.
Proposed Behavior
After a worker has been idle for at least worker_max_idle, the worker is either killed or restarted.
Proposed UI/UX
worker_max_idle=120
Activity
thedrow commentedon Sep 5, 2024
Contributions are welcome!
Nusnus commentedon Sep 5, 2024
@prhbrt
I’m not sure about the restarted, but the killed can make sense.
A host system can instantiate a new worker if the previous one was killed to “restart” the consumer
Would you like this value to say, “Kill the worker in 120 seconds since the last running task finished (and no other task is currently running)“?
Lastly,
Can you better define the problem you’re trying to solve with this?
yingw787 commentedon Aug 3, 2025
I just found this issue, but it's applicable to me:
--max-memory-per-child=200000(or 200MB) and that does help, but it'd be nice to get back to a relaxed memory state.myayush commentedon Oct 15, 2026
I'd like to work on this. Proposal: a
worker_max_idlesetting (seconds, off by default). If the worker has no running, reserved or scheduled tasks for that long, it shuts down cleanly (exit code 0), and a process manager like systemd or Kubernetes can start a new one. I'll open a PR with tests and docs shortly.auvipy commentedon Oct 17, 2026
@myayush did you check this?
myayush commentedon Oct 17, 2026
@auvipy Yes:
worker_max_idleseconds, it stops cleanly. Then systemd or Kubernetes can start a new worker.