Update scheduler locking to capture ownership and detect expiring locks by terencehonles · Pull Request #2411 · rq/rq · GitHub
Skip to content

Update scheduler locking to capture ownership and detect expiring locks - #2411

Open
terencehonles wants to merge 5 commits into
rq:masterfrom
terencehonles:add-scheduler-lock-ownership
Open

terencehonles wants to merge 5 commits into
rq:masterfrom
terencehonles:add-scheduler-lock-ownership

Conversation

@terencehonles

@terencehonles terencehonles commented May 26, 2026 •

Copy link
Copy Markdown
Contributor

This updates the RQ scheduler to store a start-up token in Redis, and check the token upon start-up and heartbeat refreshes to ensure the lock is still owned by the current scheduler.

This also changes the worker's burst mode to not start the scheduler when running the maintenance tasks.

fixes: #2338

Summary by CodeRabbit

  • New Features
    • Optional "burst" scheduler mode, improved scheduler PID reporting and more accurate lock ownership resolution.
  • Bug Fixes
    • Robust lock handling, validation and recovery to avoid lost or misattributed locks; fixes for races during acquisition, heartbeat and shutdown; scheduler auto-start behavior refined.
  • Refactor
    • Centralized script registration/caching and targeted scheduler-lock operations; unique enqueue/schedule paths consolidated and annotated.
  • Tests
    • Added tests for lock acquisition races and heartbeat ownership changes.

Review Change Stack

@coderabbitai

coderabbitai Bot commented May 26, 2026 •

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
rq/scheduler.py (1)

161-164: 💤 Low value

Iterate over successful_locks instead of _queue_names.

The loop updates locks for all queue names, but only successful_locks are actually owned. While xx=True prevents creating keys that don't exist, this issues unnecessary Redis commands for queues where the lock was lost.

♻️ Suggested fix
-        for name in self._queue_names:
+        for name in successful_locks:
             self.connection.set(
                 self.get_locking_key(name), f'{self._ppid}:{self._pid}:{self._token}', ex=self.interval + 60, xx=True
             )
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@rq/scheduler.py` around lines 161 - 164, The loop currently iterates
self._queue_names and issues Redis SETs for every queue even if the lock wasn't
acquired; change the loop to iterate over successful_locks (the collection of
queues we actually own) and call self.connection.set(self.get_locking_key(name),
f'{self._ppid}:{self._pid}:{self._token}', ex=self.interval + 60, xx=True) only
for those names so you avoid unnecessary Redis commands for lost locks while
keeping the same expiration and xx=True behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@rq/scheduler.py`:
- Around line 161-164: The loop currently iterates self._queue_names and issues
Redis SETs for every queue even if the lock wasn't acquired; change the loop to
iterate over successful_locks (the collection of queues we actually own) and
call self.connection.set(self.get_locking_key(name),
f'{self._ppid}:{self._pid}:{self._token}', ex=self.interval + 60, xx=True) only
for those names so you avoid unnecessary Redis commands for lost locks while
keeping the same expiration and xx=True behavior.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 0b85301b-5126-4265-87a7-f258ae655c4e

📥 Commits

Reviewing files that changed from the base of the PR and between 4ee8428 and d176f4d.

📒 Files selected for processing (4)
  • rq/queue.py
  • rq/scheduler.py
  • rq/worker/base.py
  • tests/test_scheduler.py

Comment thread rq/queue.py
Comment on lines +279 to +280
# PID is only set after forking, so fall back to the parent's PID if the child has not forked yet
return int(ppid if not pid else pid)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I noticed that os.getpid() is going to be different between the first call to acquire_locks which is done by the parent, and subsequent calls which will be done by the child. I have captured both PIDs in the value stored in Redis, and I assume this behavior here is fine because it captures what was previously happening during start-up.

I don't believe anyone was using the PID here to specifically signal the scheduler's process, but if they do it too early they will signal the worker and not the scheduler.

@terencehonles
terencehonles force-pushed the add-scheduler-lock-ownership branch from d176f4d to 02e6da2 Compare May 26, 2026 12:07
@terencehonles
terencehonles force-pushed the add-scheduler-lock-ownership branch from 02e6da2 to 4c0bbb1 Compare May 26, 2026 12:10

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
rq/scheduler.py (1)

162-165: 💤 Low value

Consider iterating over successful_locks instead of _queue_names.

The loop updates locks for all queue names, but only successful_locks contain locks this scheduler owns. While SET with xx=True harmlessly fails for unowned locks, iterating over successful_locks would be more efficient and clearer in intent.

♻️ Suggested change
-        for name in self._queue_names:
+        for name in successful_locks:
             self.connection.set(
                 self.get_locking_key(name), f'{self._ppid}:{self._pid}:{self._token}', ex=self.interval + 60, xx=True
             )
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@rq/scheduler.py` around lines 162 - 165, The loop currently iterates
self._queue_names and calls self.connection.set(...) for each name even though
only locks in successful_locks are actually owned; change the loop to iterate
over successful_locks instead of self._queue_names so you only refresh owned
locks. Locate the block using successful_locks, replace the iterator from
self._queue_names to successful_locks, keeping the call to
self.connection.set(self.get_locking_key(name),
f'{self._ppid}:{self._pid}:{self._token}', ex=self.interval + 60, xx=True) so
ttl logic (interval) and token composition remain unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@rq/scheduler.py`:
- Line 273: The while loop uses an undefined module-level name Status which will
raise NameError; update the reference in the loop condition to use the class
attribute (self.Status.STOPPED) or the module-level enum
(SchedulerStatus.STOPPED) to match how Status is defined on RQScheduler (see
RQScheduler and Status), e.g., change "while self.status != Status.STOPPED:" to
reference the correct symbol self.Status.STOPPED (or SchedulerStatus.STOPPED) so
the name resolves at runtime.

---

Nitpick comments:
In `@rq/scheduler.py`:
- Around line 162-165: The loop currently iterates self._queue_names and calls
self.connection.set(...) for each name even though only locks in
successful_locks are actually owned; change the loop to iterate over
successful_locks instead of self._queue_names so you only refresh owned locks.
Locate the block using successful_locks, replace the iterator from
self._queue_names to successful_locks, keeping the call to
self.connection.set(self.get_locking_key(name),
f'{self._ppid}:{self._pid}:{self._token}', ex=self.interval + 60, xx=True) so
ttl logic (interval) and token composition remain unchanged.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 670d3c2e-2663-44fa-a2e1-ff234803857a

📥 Commits

Reviewing files that changed from the base of the PR and between d176f4d and 02e6da2.

📒 Files selected for processing (4)
  • rq/queue.py
  • rq/scheduler.py
  • rq/worker/base.py
  • tests/test_scheduler.py

Comment thread rq/scheduler.py Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
rq/scheduler.py (1)

162-165: 💤 Low value

Iterate over successful_locks instead of _queue_names when updating locks.

The loop updates locks for all queue names, but only successful_locks contain verified locks. The XX flag makes the spurious writes no-ops, but iterating over successful_locks is cleaner and avoids unnecessary Redis calls.

♻️ Suggested improvement
-        for name in self._queue_names:
+        for name in successful_locks:
             self.connection.set(
                 self.get_locking_key(name), f'{self._ppid}:{self._pid}:{self._token}', ex=self.interval + 60, xx=True
             )
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@rq/scheduler.py` around lines 162 - 165, The loop currently iterates over
self._queue_names when calling self.connection.set(..., xx=True) to refresh
locks; change it to iterate over successful_locks (the collection of verified
locks) instead so only held locks are updated. Specifically, replace usages of
self._queue_names in that loop with successful_locks while still using
self.get_locking_key(name) and the same value
f'{self._ppid}:{self._pid}:{self._token}' and expiry self.interval + 60; keep
the xx=True behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@rq/scheduler.py`:
- Around line 162-165: The loop currently iterates over self._queue_names when
calling self.connection.set(..., xx=True) to refresh locks; change it to iterate
over successful_locks (the collection of verified locks) instead so only held
locks are updated. Specifically, replace usages of self._queue_names in that
loop with successful_locks while still using self.get_locking_key(name) and the
same value f'{self._ppid}:{self._pid}:{self._token}' and expiry self.interval +
60; keep the xx=True behavior.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: ab9879dd-c198-42f4-af6b-9b1a54e3c5da

📥 Commits

Reviewing files that changed from the base of the PR and between 02e6da2 and 4c0bbb1.

📒 Files selected for processing (4)
  • rq/queue.py
  • rq/scheduler.py
  • rq/worker/base.py
  • tests/test_scheduler.py

@codecov

codecov Bot commented May 27, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 83.87097% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.69%. Comparing base (2de9491) to head (376dd41).
⚠️ Report is 268 commits behind head on master.

Files with missing lines Patch % Lines
rq/scheduler.py 80.64% 12 Missing ⚠️
rq/queue.py 71.42% 2 Missing ⚠️
rq/scripts.py 95.45% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2411      +/-   ##
==========================================
+ Coverage   93.61%   93.69%   +0.07%     
==========================================
  Files          28       38      +10     
  Lines        3760     4950    +1190     
==========================================
+ Hits         3520     4638    +1118     
- Misses        240      312      +72     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (2)
rq/scripts.py (2)

23-31: ⚖️ Poor tradeoff

Consider using weak references for connection keys to prevent memory leaks.

The cache uses Redis connection objects as dictionary keys. If connections are recreated over the application lifetime, stale entries accumulate since there's no cleanup mechanism.

For long-running processes that create many connections, consider using weakref.WeakKeyDictionary or restructuring the cache to use a hashable connection identifier.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@rq/scripts.py` around lines 23 - 31, The _registered_scripts cache currently
uses Redis connection objects as keys (see _registered_scripts and
get_registered_script) which can leak when connections are recreated; replace
the plain dict with a weakref.WeakKeyDictionary keyed by the Redis connection
(or alternatively key by a stable connection identifier such as
connection.client_id or a connection-specific string) so entries are removed
when connections are garbage-collected, and update get_registered_script to use
that WeakKeyDictionary and still call connection.register_script(source) when
missing.

34-52: 💤 Low value

Guard against empty keys list to avoid unnecessary Redis roundtrip.

The wrapper calls the Lua script even when keys is empty, which is a no-op but still incurs network overhead.

🛡️ Proposed guard
 def delete_scheduler_locks(connection: Redis, token: str, keys: Sequence[str]) -> int:
+    if not keys:
+        return 0
     return get_registered_script(connection, DELETE_SCHEDULER_LOCKS)(keys=keys, args=[token])
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@rq/scripts.py` around lines 34 - 52, The delete_scheduler_locks wrapper
currently calls the Lua script even when keys is empty; add a guard in the
delete_scheduler_locks function to return 0 immediately if keys is empty (e.g.,
if not keys: return 0) to avoid the unnecessary Redis roundtrip, otherwise
proceed to call get_registered_script(connection,
DELETE_SCHEDULER_LOCKS)(keys=keys, args=[token]).
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@rq/scripts.py`:
- Around line 1-21: Remove the unused Any import from the top-level imports and
reorder the import block to follow PEP8/ruff ordering (stdlib, third-party,
local) so imports like calendar, logging, time, datetime... come first, then
Redis, then local modules (DuplicateJobError, blue, green); keep the
TYPE_CHECKING block for runtime-only imports (Sequence, Script, Job) and run
ruff check --fix rq/scripts.py to auto-sort/fix remaining lint issues and ensure
logger = logging.getLogger('rq.scripts') remains after imports.

---

Nitpick comments:
In `@rq/scripts.py`:
- Around line 23-31: The _registered_scripts cache currently uses Redis
connection objects as keys (see _registered_scripts and get_registered_script)
which can leak when connections are recreated; replace the plain dict with a
weakref.WeakKeyDictionary keyed by the Redis connection (or alternatively key by
a stable connection identifier such as connection.client_id or a
connection-specific string) so entries are removed when connections are
garbage-collected, and update get_registered_script to use that
WeakKeyDictionary and still call connection.register_script(source) when
missing.
- Around line 34-52: The delete_scheduler_locks wrapper currently calls the Lua
script even when keys is empty; add a guard in the delete_scheduler_locks
function to return 0 immediately if keys is empty (e.g., if not keys: return 0)
to avoid the unnecessary Redis roundtrip, otherwise proceed to call
get_registered_script(connection, DELETE_SCHEDULER_LOCKS)(keys=keys,
args=[token]).
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 5110c44a-3bef-44ec-bebb-9ebc91291e07

📥 Commits

Reviewing files that changed from the base of the PR and between 4c0bbb1 and 7c5f068.

📒 Files selected for processing (2)
  • rq/scheduler.py
  • rq/scripts.py

Comment thread rq/scheduler.py Outdated
Comment thread rq/scripts.py
@terencehonles
terencehonles force-pushed the add-scheduler-lock-ownership branch from 7c5f068 to 30a300d Compare May 27, 2026 11:28
@terencehonles
terencehonles force-pushed the add-scheduler-lock-ownership branch from 2d64ce8 to 376dd41 Compare May 27, 2026 12:11

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (1)
rq/scripts.py (1)

23-31: 💤 Low value

Script cache may retain references to closed connections.

Using connection as part of the cache key prevents garbage collection of connection objects even after they're closed. In long-running processes that dynamically create/destroy connections, this could cause a slow memory leak.

For typical RQ usage with stable connections this is acceptable, but consider using a WeakValueDictionary or connection ID-based key if dynamic connections become a concern.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@rq/scripts.py` around lines 23 - 31, The cache currently uses the tuple
(source, connection) in _registered_scripts which holds strong refs to Redis
connection objects; replace this with a weak-reference keyed structure so closed
connections can be GC'd: import weakref and change _registered_scripts to a
weakref.WeakKeyDictionary mapping connection -> dict[source, Script], then
update get_registered_script to look up the per-connection dict (creating it if
missing) and register/cache the Script under the source key rather than
embedding the connection in the tuple; keep function names _registered_scripts
and get_registered_script and continue to call
connection.register_script(source) as currently done.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@rq/scripts.py`:
- Around line 35-48: The Lua snippet in SCHEDULER_DELETE_LOCKS uses string.find
with a pattern built via string.format, which treats ARGV[1] as a Lua pattern
and can mis-match when the token contains metacharacters; change the ownership
check to an exact suffix comparison by reading the value with redis.call('GET',
key) and using string.sub to compare the tail against ARGV[1] (and only delete
when equal), preserving the existing semantics of iterating over KEYS and
returning count — replace the pattern/string.find logic with a strict string.sub
equality check that handles empty or nil GET results safely.
- Around line 62-69: The Lua script's lock matching uses string.find with a
pattern (local pattern = string.format(':%s$', ARGV[1])) which can produce false
positives; change it to compute the exact suffix (local suffix = ':' .. ARGV[1])
and for each key read the value into a local (local val = redis.call('GET', key)
or ''), then test exact match with string.sub(val, -string.len(suffix)) ==
suffix before incrementing count and calling redis.call('SET', key, ARGV[2],
'ex', ARGV[3]); update the block containing local pattern, count, the loop over
KEYS and the redis.call('SET') to use suffix and string.sub for exact-match
checking.

---

Nitpick comments:
In `@rq/scripts.py`:
- Around line 23-31: The cache currently uses the tuple (source, connection) in
_registered_scripts which holds strong refs to Redis connection objects; replace
this with a weak-reference keyed structure so closed connections can be GC'd:
import weakref and change _registered_scripts to a weakref.WeakKeyDictionary
mapping connection -> dict[source, Script], then update get_registered_script to
look up the per-connection dict (creating it if missing) and register/cache the
Script under the source key rather than embedding the connection in the tuple;
keep function names _registered_scripts and get_registered_script and continue
to call connection.register_script(source) as currently done.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 498cb106-19aa-4eca-ad6b-74e263a3d031

📥 Commits

Reviewing files that changed from the base of the PR and between 7c5f068 and 376dd41.

📒 Files selected for processing (2)
  • rq/scheduler.py
  • rq/scripts.py
🚧 Files skipped from review as they are similar to previous changes (1)
  • rq/scheduler.py

Comment thread rq/scripts.py
Comment thread rq/scripts.py Outdated
@selwin

selwin commented May 28, 2026

Copy link
Copy Markdown
Collaborator

If the objective is to check whether the lock is still owned by the current scheduler, wouldn't atomically checking the PID inside the locking key before renewing the lock in a single atomic lua script be sufficient?

@terencehonles

terencehonles commented May 28, 2026 •

Copy link
Copy Markdown
Contributor Author

If the objective is to check whether the lock is still owned by the current scheduler, wouldn't atomically checking the PID inside the locking key before renewing the lock in a single atomic lua script be sufficient?

The PID will collide for us since we run in a container and they'll be the only thing we're starting up. In the general case it's just asking for a collision error, which may be unlikely, but will surface as a hard to reproduce bug.

@selwin

selwin commented May 29, 2026

Copy link
Copy Markdown
Collaborator

The PID will collide for us since we run in a container and they'll be the only thing we're starting up.

In that case, let's use a random scheduler.name instead? It's more consistent with how workers are designed. We should also have a scheduler.register_birth() to store more scheduler related data such as date created, last heartbeat etc.

I think scheduler.register_birth() would have to be a separate PR though. Changing locking mechanism is scary enough for me, I don't want to cram too many things into a single PR.

@terencehonles

terencehonles commented May 29, 2026 •

Copy link
Copy Markdown
Contributor Author

Sure that works, I just picked the secrets token because it was an easy way to get random data. I'm not sure if I arrived at that before looking at the implementation more closely, but while a uuid would lose some randomness it's definitely sufficient!

@terencehonles

terencehonles commented May 29, 2026 •

Copy link
Copy Markdown
Contributor Author

Sure that works, I just picked the secrets token because it was an easy way to get random data. I'm not sure if I arrived at that before looking at the implementation more closely, but while a uuid would lose some randomness it's definitely sufficient!

I was starting to make this change, and I realized we're probably both talking about the CronScheduler which is different. The Scheduler doesn't have a name. I was also probably thinking of WorkerPool which auto-names its workers, because the CronScheduler only has a partially random name and only if not provided (unrelated and not important).

In that case, is there anything wrong with the scheduler maintaining its own token?

@selwin

selwin commented May 30, 2026

Copy link
Copy Markdown
Collaborator

I was starting to make this change, and I realized we're probably both talking about the CronScheduler which is different. The Scheduler doesn't have a name.

I was indeed referring to RQScheduler, which does not currently have a name and a separate hash to store it's metadata. This is why I mentioned this should first be implemented in a different PR since the churn will be quite high.

Having name and a separate key will bring it's design close to RQ's Worker and CronScheduler. I'll work on this sometime this week.

@selwin

selwin commented Jun 6, 2026

Copy link
Copy Markdown
Collaborator

FYI, I'm working on scheduler.name here #2414

@selwin

selwin commented Jun 12, 2026

Copy link
Copy Markdown
Collaborator

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RQ scheduler can sometimes duplicate jobs

2 participants