{{ message }}
Repository navigation
documentation - how to restart failed crawl jobs #1033
Description
It would be great if there was more documentation on how to deal with large crawl jobs that fail. For example, in my current crawl job, the job failed because the proxy I was using failed (e.g., ran out of credits). Now the request has reached its maximum retry count, and won't be processed again. What's the correct way to update the request so that it'll get picked up for processing again?
Activity
Mantisus commentedon Feb 27, 2025
I just want to make sure I understand your case correctly.
You are interested in a solution for the case. How can I use
CRAWLEE_PURGE_ON_START=0to restore the queue so that links that were not processed because they ran out of repetition limits are taken back to work?yqiang commentedon Feb 27, 2025
Right - perhaps it's just me but this seems like it would be a common failure mode.
Mantisus commentedon Feb 28, 2025
I don't think we have simple mechanisms for this.
However, we should probably expand the queue interaction capabilities to give users the ability to handle the case you described. And yes, we definitely need more examples in the documentation :)
I would handle the case you described during runtime using
crawler.error_handler. When any errors occur, theerror_handlerdecides whether the request should return to the queue or be processed differently. If many proxy-related errors are registered, then the crawler stops. After that, you can restart it with the flagCRAWLEE_PURGE_ON_START=0and a new proxy pool, so that it continues working.janbuchar commentedon Mar 4, 2025
You can also use the
BasicCrawler.failed_request_handlerwhich gets invoked with after the maximum number of retries for a request has been reached and re-enqueue the failed request or store it somewhere so that you can restart it later.yqiang commentedon Mar 4, 2025
Great, I'll give this a try next time I run into this issue.