iframe-proxy

Mantisus · 2026-10-15T21:58:46Z

Description

RequestLoader and RequestManager replace is_empty() and is_finished() with a single check_readiness(). It returns one of four statuses, exported from crawlee.request_loaders: ReadyStatus, WaitingStatus, StalledStatus or FinishedStatus. WaitingStatus.ready_at says when a source expects to have work again, and ThrottlingRequestManager sets it from its cooldowns. StalledStatus.reason says why a source can't make progress. Storage clients keep is_empty() and is_finished(). The upgrade guide covers the migration.

ThrottlingRequestManager.get_stall_reason() is removed. The crawler reads a stall from the status, so it now also detects a stalled domain when the ThrottlingRequestManager is wrapped in a RequestManagerTandem.
RequestManagerTandem moves a request from its loader only when the loader is ready, so a SitemapRequestLoader that's still loading sitemaps no longer holds up the requests already in the manager.
RequestLoader.fetch_next_request() no longer has to wait for a request. A None means nothing is pending right now, not that the loader is finished. SitemapRequestLoader.fetch_next_request() returns None once its sitemaps are loaded, instead of waiting until every fetched request is handled.

Issues

Part of: Replace is_empty/is_finished with check_readiness() and add pacing signals to RequestManager #2296

✍️ Drafted by Claude Code

vdusek · 2026-10-16T07:25:58Z

+```python
+from crawlee.request_loaders import FinishedStatus, ReadyStatus
+
+# v1
+has_requests = not await rq.is_empty()
+done = await rq.is_finished()
+
+# v2
+has_requests = isinstance(await rq.check_readiness(), ReadyStatus)
+done = isinstance(await rq.check_readiness(), FinishedStatus)
+```


As I mentioned above, this is much less convenient than before. You now need an additional import and an isinstance() check.

Maybe we could introduce helper methods for checking statuses, so determining whether an RQ is in a particular status would be a single method call?

ok, then let's use literal comparison at least, so users aren't pushed into an extra import

@janbuchar Have you considered keeping is_finished as a simple helper that wraps

isinstance(await rq.check_readiness(), FinishedStatus)

?

We could add similar helpers for the other statuses where they make sense.

I mean, the new API is a bit worse in terms of ergonomics (maybe with AI agents it doesn't matter that much?).

ok, then let's use literal comparison at least, so users aren't pushed into an extra import

Implemented in 883a68c

vdusek

one potential optimization note

vdusek · 2026-10-21T07:01:16Z


        request_manager = await self.get_request_manager()
-        return not await request_manager.is_empty()
+        return (await request_manager.check_readiness()).status == 'ready'


Note: The pool calls __is_finished_function and then __is_task_ready_function on every pass, and both now run check_readiness(). On a queue that is empty but not finished, each call makes two backend calls (is_empty() + is_finished()), so a pass costs 4 calls where it used to cost 2. That covers the end of a crawl, waiting on in-progress requests, and throttler cooldowns. On the platform these are API calls, and ThrottlingRequestManager repeats them for every domain queue. Could the status from __is_finished_function be reused here, or is that too tied to the call order in the pool?

✍️ Drafted by Claude Code

It will be resolved in #2332

vdusek · 2026-10-21T07:19:35Z

+    if first.status == 'waiting':
+        if second.status != 'waiting' or second.ready_at is None:
+            return first
+        if first.ready_at is not None and first.ready_at <= second.ready_at:
+            return first
+        return second


Question: a waiting status with ready_at=None means requests are in progress or being added, so work can show up any moment. Joining it with ready_at=T returns T, which promises nothing before T. Nothing reads ready_at yet, but it's public. Shouldn't None win here? I see test_join_keeps_known_ready_at pins the current behavior, so maybe I'm missing the intent.

✍️ Drafted by Claude Code

It's intentional, and JS joins them the same way ("the earlier of the two known wake-up times, unknown only if neither source announced one"). ready_at isn't a promise that nothing shows up before it. It says when a wake-up is worth scheduling, on top of the regular polling AutoscaledPool does anyway. I added a comment to the join and clarified the ready_at docstring.

vdusek · 2026-10-21T07:19:35Z

+        stalled = [
+            domain
+            for domain, readiness in zip(candidates, candidate_statuses, strict=True)
+            if readiness.status == 'ready'
+        ]
+
+        if stalled:


Note: a stall wins here even if a candidate's inner queue is waiting only because a background add_requests batch is still running, so an idle crawler can raise PersistentRateLimitError while new requests are arriving. The old get_stall_reason did the same, so this isn't a regression and is fine as a follow-up.

✍️ Drafted by Claude Code

Good catch. I fixed it here, since this PR is what sets the order between the statuses.

github-actions Bot added the t-tooling Issues with this label are in the ownership of the tooling team. label Oct 15, 2026

Mantisus self-assigned this Oct 15, 2026

Mantisus marked this pull request as ready for review October 15, 2026 21:59

vdusek reviewed Oct 16, 2026

View reviewed changes

vdusek force-pushed the v2 branch from 792e5a6 to 08747f7 Compare October 16, 2026 07:43

Mantisus force-pushed the check-readiness branch 2 times, most recently from 3629140 to a30518d Compare October 18, 2026 17:01

vdusek requested review from Pijukatel and vdusek October 19, 2026 11:00

vdusek reviewed Oct 21, 2026

View reviewed changes

Mantisus force-pushed the check-readiness branch from caac18c to bbe9b52 Compare October 21, 2026 17:08

Mantisus added 4 commits October 21, 2026 18:12

replace is_empty and is_finished with check_readiness

d80c60c

add status literal in statuses

6899836

fixes

c4e5eb5

update upgrade guide

09c2557

Mantisus force-pushed the check-readiness branch from bbe9b52 to 09c2557 Compare October 21, 2026 18:44

Mantisus requested a review from vdusek October 21, 2026 18:45

Sunbelt Computer Software

PL/B Language Development and Support

Conversation

Mantisus commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Issues

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vdusek left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Mantisus commented Oct 15, 2026 •

edited

Loading

vdusek left a comment •

edited

Loading