{{ message }}
Repository navigation
Handle unprocessed requests in batch_add_requests #456
Description
in _request_queue_client.py (async and sync client) there is a batch_add_requests method. This method is calling Apify API which can return some unprocessed requests in rare cases. This can happen for example due to rate limiting when there is high load. Currently it seems that those unprocessed requests are not handled in any way and will be just ignored.
Maybe the method should at least retry to add unprocessed requests in some final batch? Maybe with some backoff?
(Some context apify/crawlee-python#1155 (comment))
Activity
B4nan commentedon Apr 16, 2025
IIRC we deal with this in crawlee, at least in the JS version.
Pijukatel commentedon Apr 16, 2025
I do not think it is done in Python version. There we assume it just works and we only print the response. This response can contain info about unprocessed requests, but we do not handle that there.
@janbuchar I might have missed some handling on some other level, so please correct me if I am wrong.
feat: Handle unprocessed requests in `add_requests_batched` (#1159)
feat: Handle unprocessed requests in `add_requests_batched` (apify#1159)
feat: Handle unprocessed requests in `add_requests_batched` (apify#1159)