{{ message }}
Repository navigation
Add support for crawl-delay robots.txt directive #1396
Description
Summary
RobotsTxtFile.get_crawl_delay() already parses the Crawl-delay directive, but the value is never used to throttle requests. When respect_robots_txt_file=True, the crawler should automatically enforce per-domain delays specified in robots.txt.
Why
- Respect server capacity limits and avoid IP bans / HTTP 429 responses.
- Better compatibility with Cloudflare Bot Management and AI Crawl Control — polite crawlers are less likely to be blocked.
- Cost control with Cloudflare Pay Per Crawl — respecting rate limits avoids unnecessary charges.
- Automatic per-domain throttling at scale without manual configuration.
References
- Crawl-delay directive (Wikipedia)
- Existing code:
RobotsTxtFile.get_crawl_delay()
Activity
janbuchar commentedon Dec 4, 2025
This is tricky due to the nature of the Apify request queue which we need to support. It is not possible to set a "don't process this before " attribute there, so you cannot check the URL of a freshly dequeued request and re-insert it at the start of the queue with this set up. Also, you cannot consume requests from the queue selectively by domain.
An approach that I already suggested for the JS version is to make a separate non-default unnamed queue for each domain where we want to allow a delay. Then we can implement the "cooldown period" on the request queue client so that it only yields a request once every N seconds. Finally, we'll wrap these request queue clients with one that will check all of them for requests ready for processing and redirect new requests to appropriate queues.
vdusek commentedon Feb 17, 2026
Moving to the icebox for now, as the Cloudflare Pay Per Crawl is still in private beta, let's see how it evolves.
refactor: replace RequestThrottler with ThrottlingRequestManager