{{ message }}
Repository navigation
Running different requests with different crawlers? #573
Answered
by
janbuchar
honzajavorek
asked this question in
Q&A
Replies: 2 comments 3 replies
|
Well, the best approach to this would be to have separate
We are aware of this shortcoming though, and we'd like to enable running multiple interconnected crawlers in the future. |
2 replies
Answer selected by
honzajavorek
|
Worth noting on this thread. Looks like the team have now implemented it. |
1 reply
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment

Well, the best approach to this would be to have separate
RequestQueueinstances for the separate crawlers and to add requests directly to the queue of the right crawler in your request handlers. There are however some challenges:Request.open()will always resolve to the same unnamed queue. This may or may not be a problem if you're running on Apify. Locally, you'll probably need to purge the named queues manually before each run.await asyncio.gather(crawler_1.run(), crawler_2.run())also won't work right off the bat - I assume that only one of your crawlers will have some start urls a…