Repository navigation
Update scheduler locking to capture ownership and detect expiring locks - #2411
terencehonles wants to merge 5 commits into
Conversation
There was a problem hiding this comment.
🧹 Nitpick comments (1)
rq/scheduler.py (1)
161-164: 💤 Low valueIterate over
successful_locksinstead of_queue_names.The loop updates locks for all queue names, but only
successful_locksare actually owned. Whilexx=Trueprevents creating keys that don't exist, this issues unnecessary Redis commands for queues where the lock was lost.♻️ Suggested fix
- for name in self._queue_names: + for name in successful_locks: self.connection.set( self.get_locking_key(name), f'{self._ppid}:{self._pid}:{self._token}', ex=self.interval + 60, xx=True )🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rq/scheduler.py` around lines 161 - 164, The loop currently iterates self._queue_names and issues Redis SETs for every queue even if the lock wasn't acquired; change the loop to iterate over successful_locks (the collection of queues we actually own) and call self.connection.set(self.get_locking_key(name), f'{self._ppid}:{self._pid}:{self._token}', ex=self.interval + 60, xx=True) only for those names so you avoid unnecessary Redis commands for lost locks while keeping the same expiration and xx=True behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@rq/scheduler.py`:
- Around line 161-164: The loop currently iterates self._queue_names and issues
Redis SETs for every queue even if the lock wasn't acquired; change the loop to
iterate over successful_locks (the collection of queues we actually own) and
call self.connection.set(self.get_locking_key(name),
f'{self._ppid}:{self._pid}:{self._token}', ex=self.interval + 60, xx=True) only
for those names so you avoid unnecessary Redis commands for lost locks while
keeping the same expiration and xx=True behavior.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 0b85301b-5126-4265-87a7-f258ae655c4e
📒 Files selected for processing (4)
rq/queue.pyrq/scheduler.pyrq/worker/base.pytests/test_scheduler.py
| # PID is only set after forking, so fall back to the parent's PID if the child has not forked yet | ||
| return int(ppid if not pid else pid) |
There was a problem hiding this comment.
I noticed that os.getpid() is going to be different between the first call to acquire_locks which is done by the parent, and subsequent calls which will be done by the child. I have captured both PIDs in the value stored in Redis, and I assume this behavior here is fine because it captures what was previously happening during start-up.
I don't believe anyone was using the PID here to specifically signal the scheduler's process, but if they do it too early they will signal the worker and not the scheduler.
d176f4d to
02e6da2
Compare
02e6da2 to
4c0bbb1
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
rq/scheduler.py (1)
162-165: 💤 Low valueConsider iterating over
successful_locksinstead of_queue_names.The loop updates locks for all queue names, but only
successful_lockscontain locks this scheduler owns. While SET withxx=Trueharmlessly fails for unowned locks, iterating oversuccessful_lockswould be more efficient and clearer in intent.♻️ Suggested change
- for name in self._queue_names: + for name in successful_locks: self.connection.set( self.get_locking_key(name), f'{self._ppid}:{self._pid}:{self._token}', ex=self.interval + 60, xx=True )🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rq/scheduler.py` around lines 162 - 165, The loop currently iterates self._queue_names and calls self.connection.set(...) for each name even though only locks in successful_locks are actually owned; change the loop to iterate over successful_locks instead of self._queue_names so you only refresh owned locks. Locate the block using successful_locks, replace the iterator from self._queue_names to successful_locks, keeping the call to self.connection.set(self.get_locking_key(name), f'{self._ppid}:{self._pid}:{self._token}', ex=self.interval + 60, xx=True) so ttl logic (interval) and token composition remain unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@rq/scheduler.py`:
- Line 273: The while loop uses an undefined module-level name Status which will
raise NameError; update the reference in the loop condition to use the class
attribute (self.Status.STOPPED) or the module-level enum
(SchedulerStatus.STOPPED) to match how Status is defined on RQScheduler (see
RQScheduler and Status), e.g., change "while self.status != Status.STOPPED:" to
reference the correct symbol self.Status.STOPPED (or SchedulerStatus.STOPPED) so
the name resolves at runtime.
---
Nitpick comments:
In `@rq/scheduler.py`:
- Around line 162-165: The loop currently iterates self._queue_names and calls
self.connection.set(...) for each name even though only locks in
successful_locks are actually owned; change the loop to iterate over
successful_locks instead of self._queue_names so you only refresh owned locks.
Locate the block using successful_locks, replace the iterator from
self._queue_names to successful_locks, keeping the call to
self.connection.set(self.get_locking_key(name),
f'{self._ppid}:{self._pid}:{self._token}', ex=self.interval + 60, xx=True) so
ttl logic (interval) and token composition remain unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 670d3c2e-2663-44fa-a2e1-ff234803857a
📒 Files selected for processing (4)
rq/queue.pyrq/scheduler.pyrq/worker/base.pytests/test_scheduler.py
There was a problem hiding this comment.
🧹 Nitpick comments (1)
rq/scheduler.py (1)
162-165: 💤 Low valueIterate over
successful_locksinstead of_queue_nameswhen updating locks.The loop updates locks for all queue names, but only
successful_lockscontain verified locks. TheXXflag makes the spurious writes no-ops, but iterating oversuccessful_locksis cleaner and avoids unnecessary Redis calls.♻️ Suggested improvement
- for name in self._queue_names: + for name in successful_locks: self.connection.set( self.get_locking_key(name), f'{self._ppid}:{self._pid}:{self._token}', ex=self.interval + 60, xx=True )🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rq/scheduler.py` around lines 162 - 165, The loop currently iterates over self._queue_names when calling self.connection.set(..., xx=True) to refresh locks; change it to iterate over successful_locks (the collection of verified locks) instead so only held locks are updated. Specifically, replace usages of self._queue_names in that loop with successful_locks while still using self.get_locking_key(name) and the same value f'{self._ppid}:{self._pid}:{self._token}' and expiry self.interval + 60; keep the xx=True behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@rq/scheduler.py`:
- Around line 162-165: The loop currently iterates over self._queue_names when
calling self.connection.set(..., xx=True) to refresh locks; change it to iterate
over successful_locks (the collection of verified locks) instead so only held
locks are updated. Specifically, replace usages of self._queue_names in that
loop with successful_locks while still using self.get_locking_key(name) and the
same value f'{self._ppid}:{self._pid}:{self._token}' and expiry self.interval +
60; keep the xx=True behavior.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: ab9879dd-c198-42f4-af6b-9b1a54e3c5da
📒 Files selected for processing (4)
rq/queue.pyrq/scheduler.pyrq/worker/base.pytests/test_scheduler.py
…ks before shutting down
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #2411 +/- ##
==========================================
+ Coverage 93.61% 93.69% +0.07%
==========================================
Files 28 38 +10
Lines 3760 4950 +1190
==========================================
+ Hits 3520 4638 +1118
- Misses 240 312 +72 ☔ View full report in Codecov by Sentry. |
…ster lua scripts, add typing to scripts
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (2)
rq/scripts.py (2)
23-31: ⚖️ Poor tradeoffConsider using weak references for connection keys to prevent memory leaks.
The cache uses
Redisconnection objects as dictionary keys. If connections are recreated over the application lifetime, stale entries accumulate since there's no cleanup mechanism.For long-running processes that create many connections, consider using
weakref.WeakKeyDictionaryor restructuring the cache to use a hashable connection identifier.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rq/scripts.py` around lines 23 - 31, The _registered_scripts cache currently uses Redis connection objects as keys (see _registered_scripts and get_registered_script) which can leak when connections are recreated; replace the plain dict with a weakref.WeakKeyDictionary keyed by the Redis connection (or alternatively key by a stable connection identifier such as connection.client_id or a connection-specific string) so entries are removed when connections are garbage-collected, and update get_registered_script to use that WeakKeyDictionary and still call connection.register_script(source) when missing.
34-52: 💤 Low valueGuard against empty keys list to avoid unnecessary Redis roundtrip.
The wrapper calls the Lua script even when
keysis empty, which is a no-op but still incurs network overhead.🛡️ Proposed guard
def delete_scheduler_locks(connection: Redis, token: str, keys: Sequence[str]) -> int: + if not keys: + return 0 return get_registered_script(connection, DELETE_SCHEDULER_LOCKS)(keys=keys, args=[token])🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rq/scripts.py` around lines 34 - 52, The delete_scheduler_locks wrapper currently calls the Lua script even when keys is empty; add a guard in the delete_scheduler_locks function to return 0 immediately if keys is empty (e.g., if not keys: return 0) to avoid the unnecessary Redis roundtrip, otherwise proceed to call get_registered_script(connection, DELETE_SCHEDULER_LOCKS)(keys=keys, args=[token]).
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@rq/scripts.py`:
- Around line 1-21: Remove the unused Any import from the top-level imports and
reorder the import block to follow PEP8/ruff ordering (stdlib, third-party,
local) so imports like calendar, logging, time, datetime... come first, then
Redis, then local modules (DuplicateJobError, blue, green); keep the
TYPE_CHECKING block for runtime-only imports (Sequence, Script, Job) and run
ruff check --fix rq/scripts.py to auto-sort/fix remaining lint issues and ensure
logger = logging.getLogger('rq.scripts') remains after imports.
---
Nitpick comments:
In `@rq/scripts.py`:
- Around line 23-31: The _registered_scripts cache currently uses Redis
connection objects as keys (see _registered_scripts and get_registered_script)
which can leak when connections are recreated; replace the plain dict with a
weakref.WeakKeyDictionary keyed by the Redis connection (or alternatively key by
a stable connection identifier such as connection.client_id or a
connection-specific string) so entries are removed when connections are
garbage-collected, and update get_registered_script to use that
WeakKeyDictionary and still call connection.register_script(source) when
missing.
- Around line 34-52: The delete_scheduler_locks wrapper currently calls the Lua
script even when keys is empty; add a guard in the delete_scheduler_locks
function to return 0 immediately if keys is empty (e.g., if not keys: return 0)
to avoid the unnecessary Redis roundtrip, otherwise proceed to call
get_registered_script(connection, DELETE_SCHEDULER_LOCKS)(keys=keys,
args=[token]).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 5110c44a-3bef-44ec-bebb-9ebc91291e07
📒 Files selected for processing (2)
rq/scheduler.pyrq/scripts.py
7c5f068 to
30a300d
Compare
2d64ce8 to
376dd41
Compare
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
rq/scripts.py (1)
23-31: 💤 Low valueScript cache may retain references to closed connections.
Using
connectionas part of the cache key prevents garbage collection of connection objects even after they're closed. In long-running processes that dynamically create/destroy connections, this could cause a slow memory leak.For typical RQ usage with stable connections this is acceptable, but consider using a
WeakValueDictionaryor connection ID-based key if dynamic connections become a concern.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rq/scripts.py` around lines 23 - 31, The cache currently uses the tuple (source, connection) in _registered_scripts which holds strong refs to Redis connection objects; replace this with a weak-reference keyed structure so closed connections can be GC'd: import weakref and change _registered_scripts to a weakref.WeakKeyDictionary mapping connection -> dict[source, Script], then update get_registered_script to look up the per-connection dict (creating it if missing) and register/cache the Script under the source key rather than embedding the connection in the tuple; keep function names _registered_scripts and get_registered_script and continue to call connection.register_script(source) as currently done.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@rq/scripts.py`:
- Around line 35-48: The Lua snippet in SCHEDULER_DELETE_LOCKS uses string.find
with a pattern built via string.format, which treats ARGV[1] as a Lua pattern
and can mis-match when the token contains metacharacters; change the ownership
check to an exact suffix comparison by reading the value with redis.call('GET',
key) and using string.sub to compare the tail against ARGV[1] (and only delete
when equal), preserving the existing semantics of iterating over KEYS and
returning count — replace the pattern/string.find logic with a strict string.sub
equality check that handles empty or nil GET results safely.
- Around line 62-69: The Lua script's lock matching uses string.find with a
pattern (local pattern = string.format(':%s$', ARGV[1])) which can produce false
positives; change it to compute the exact suffix (local suffix = ':' .. ARGV[1])
and for each key read the value into a local (local val = redis.call('GET', key)
or ''), then test exact match with string.sub(val, -string.len(suffix)) ==
suffix before incrementing count and calling redis.call('SET', key, ARGV[2],
'ex', ARGV[3]); update the block containing local pattern, count, the loop over
KEYS and the redis.call('SET') to use suffix and string.sub for exact-match
checking.
---
Nitpick comments:
In `@rq/scripts.py`:
- Around line 23-31: The cache currently uses the tuple (source, connection) in
_registered_scripts which holds strong refs to Redis connection objects; replace
this with a weak-reference keyed structure so closed connections can be GC'd:
import weakref and change _registered_scripts to a weakref.WeakKeyDictionary
mapping connection -> dict[source, Script], then update get_registered_script to
look up the per-connection dict (creating it if missing) and register/cache the
Script under the source key rather than embedding the connection in the tuple;
keep function names _registered_scripts and get_registered_script and continue
to call connection.register_script(source) as currently done.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 498cb106-19aa-4eca-ad6b-74e263a3d031
📒 Files selected for processing (2)
rq/scheduler.pyrq/scripts.py
🚧 Files skipped from review as they are similar to previous changes (1)
- rq/scheduler.py
|
If the objective is to check whether the lock is still owned by the current scheduler, wouldn't atomically checking the PID inside the locking key before renewing the lock in a single atomic lua script be sufficient? |
The PID will collide for us since we run in a container and they'll be the only thing we're starting up. In the general case it's just asking for a collision error, which may be unlikely, but will surface as a hard to reproduce bug. |
In that case, let's use a random scheduler.name instead? It's more consistent with how workers are designed. We should also have a I think |
|
Sure that works, I just picked the secrets token because it was an easy way to get random data. I'm not sure if I arrived at that before looking at the implementation more closely, but while a uuid would lose some randomness it's definitely sufficient! |
I was starting to make this change, and I realized we're probably both talking about the In that case, is there anything wrong with the scheduler maintaining its own token? |
I was indeed referring to RQScheduler, which does not currently have a name and a separate hash to store it's metadata. This is why I mentioned this should first be implemented in a different PR since the churn will be quite high. Having |
|
FYI, I'm working on |

This updates the RQ scheduler to store a start-up token in Redis, and check the token upon start-up and heartbeat refreshes to ensure the lock is still owned by the current scheduler.
This also changes the worker's burst mode to not start the scheduler when running the maintenance tasks.
fixes: #2338
Summary by CodeRabbit