Why crawlee stop even if the queue is not empty #3284
-
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 2 replies
-
|
Hello @lawrenceching and thanks for trying out Crawlee! Could you provide us with a minimal executable script that reproduces the issue? Is the log that you're sending complete or did you leave something out? |
Beta Was this translation helpful? Give feedback.
-
|
hi @janbuchar , thanks for your reply. I think I know what's happing. In my case, I have a external system to constantly push URLs to crawlee instance. Can crawlee keep running even the queue is empty? when new URL add, crawlee start to crawl immediately? |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
This is actually very easy - see https://crawlee.dev/js/api/basic-crawler/interface/BasicCrawlerOptions#keepAlive