Activity

1. vdusek

   added

   bugSomething isn't working.Something isn't working.

   t-toolingIssues with this label are in the ownership of the tooling team.Issues with this label are in the ownership of the tooling team.

   on Sep 30, 2025
2. janbuchar commented on Sep 30, 2025

   

   janbuchar

   on Sep 30, 2025

   Collaborator

   More actions

   At the moment, the AutoscaledPool only cares about 429s returned by the Apify API. We never tried to prevent clobbering the target website.
3. Pijukatel commented on Oct 3, 2025

   

   Pijukatel

   on Oct 3, 2025

   Last edited by Pijukatel

   Collaborator

   More actions

   Imagine a general scenario where there are links to various different URLs in the request queue. You get 429 from one of those URLs, but the other unrelated URLs are not affected. Slowing down the whole crawler in general does not seem good in such a case. The crawler should probably implement some additional backoff logic on a different level without the autoscaled pool even noticing it.

   That would require tracking 429s on groups of URLs and acting(back-off logic, fixed larger interval between requests on that specific site,...) on that group, without affecting different unrelated groups of URLs.
4. Pijukatel commented on Oct 3, 2025

   

   Pijukatel

   on Oct 3, 2025

   Last edited by Pijukatel

   Collaborator

   More actions

   I would suggest to also not implementing this in BasicCrawler, but instead creating a dedicated component that would be only used by BasicCrawler and group related functionality there.

   now we have in __run_task_function:

   1 - get request
   2 - possibly looks at robots.txt
   3 - start processing the request, ...
   

   I would extract the step 2 and together with this new functionality, delegate it to RequestAnalyzer (placeholder name).
   What is the responsibility of RequestAnalyzer?

   • If desired, decide how the request should be handled based on robots.txt (optional, which can be turned on/off)
   • Keep track of intentionally time-blocked/ slowed down/ delayed requests. So, for example, something like this would be tracked internally:

   group1=UrlGroup(match="crawlee.dev")
   group1.next_allowed_request_at = datetime(...)
   group1.already_retrived_unhandled_requests = {https://crawlee.dev/a,  https://crawlee.dev/b,  https://crawlee.dev/c}
   group2=UrlGroup(match="example.com")
   group2.next_allowed_request_at = datetime(...)
   group2.already_retrived_un_handled_requests = {https://example.com/abc}
   

   • RequestAnalyzer would implement fetch_next_request that would pick from one of the previously delayed requests if available and ready to go and some sort of add_request

   And the new __run_task_function would be something like this:

   0 - Check RequestAnalyzer if it has some previously delayed unhandled request that is ready to be handled now
   1 - get new request if no previously fetched request is ready from RequestAnalyzer
   2 - Use RequestAnalyzer only on new request to see if it belongs to one of the groups that should be delayed, or not handled at all
   3 - start processing the request, if RequestAnalyzer is OK with it.
   
5. MrAliHasan commented on Feb 20, 2026

   

   MrAliHasan

   on Feb 20, 2026

   Last edited by MrAliHasan

   Contributor

   More actions

   Hi @vdusek @janbuchar @Pijukatel — I've implemented a fix for this.

   After a deep dive into the codebase, I fully agree with @Pijukatel's analysis: this should be handled at the request level, not the autoscaled pool level. A global concurrency reduction would punish unrelated domains that aren't rate-limited.

   What I built:

   A new RequestThrottler component (standalone module at src/crawlee/_request_throttler.py) that:

   • Tracks 429s per domain — example.com getting rate-limited doesn't affect other-site.com
   • Exponential backoff — 2s → 4s → 8s → ... → 60s (capped), per domain
   • Respects Retry-After headers — parses both integer seconds and HTTP-date formats
   • Reclaims throttled requests — if a domain is rate-limited, the request goes back to the queue; other domains continue normally
   • Resets on success — consecutive 429 count resets when a request to that domain succeeds

   The AutoscaledPool is completely untouched — the throttling happens transparently in BasicCrawler.__run_task_function before the request is processed.

   Integration points:

   • _basic_crawler.py — throttle check before processing, 429 recording via _raise_for_session_blocked_status_code, success reset on RequestState.DONE
   • _abstract_http_crawler.py / _playwright_crawler.py — pass URL + Retry-After header to the detection method

   Tests: 13 new unit tests passing (domain independence, exponential backoff, max delay cap, Retry-After priority, success reset, etc.) + all 8 existing autoscaling tests pass with zero regressions.

   I see this as a focused first step toward the full RequestAnalyzer that @Pijukatel outlined (with robots.txt integration, URL group management, etc.) — keeping scope tight for reviewability.

   Happy to open a PR if you'd like to review. Let me know if you'd prefer any changes to the approach.
6. MrAliHasan

   added 2 commits that reference this issue on Feb 20, 2026

   fix: add per-domain RequestThrottler for 429 backoff (apify#1437)

   ...

   64f7247

   refactor: replace RequestThrottler with ThrottlingRequestManager

   ...

   62ab3b8
7. AhsanSheraz commented on Mar 24, 2026

   

   AhsanSheraz

   on Mar 24, 2026

   Last edited by AhsanSheraz

   More actions

   Hey @vdusek
   You can also try pyresilience. That is exactly the kind of overload loop pyresilience is trying to prevent with backoff, jitter, circuit breaking, and shared bulkhead/rate-limit controls. This library have all the 7 resilience patterns, wit 1 decorator.

   Pypi: https://pypi.org/project/pyresilience/
   Docs: https://pyresilience.readthedocs.io

   👀React with 👀1Reacted by Jan Buchar
8. vdusek

   closed this as completedin #1762on May 6, 2026
9. vdusek

   added a commit that references this issue on May 6, 2026

   feat: Add opt-in per-domain request throttling for HTTP 429 backoff (#…

   ...

   This issue will close once commit c17f4d5 is merged into the 'master' branch.

   c17f4d5