You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Let crawlers manage only the collaborators they create and extract per-run state #2301
BasicCrawler decides whether to enter and exit a collaborator by whether it is already active (_basic_crawler.py around line 832), not by who created it. An injected Statistics, SessionPool or BrowserPool that is not yet active is entered and torn down by the first crawler that runs, so sharing one between crawlers breaks: the first to finish exits it while the other still uses it. A user-supplied Statistics and SessionPool are also reset on every repeated run() (around line 728). The Playwright crawler always enters a passed browser_pool, and AdaptivePlaywrightCrawler initializes and clears a passed rendering_type_predictor.
Per-run state lives on the crawler itself (_running, _has_finished_before, _last_run_failed, _unexpected_stop and others), and repeated runs depend on resetting it by hand.
JS v4 settled both: a crawler owns, starts, resets and stops only what it built, and each run() gets a fresh run object.
Proposed solution
Track owned vs injected collaborators explicitly. The crawler creates and manages its defaults; injected instances are borrowed, so the caller enters and exits them, and the crawler never resets them.
Apply this to the event manager, statistics, session pool, HTTP client, browser pool, rendering type predictor and request manager.
Extract per-run state into an internal CrawlerRun built on every run().
Acceptance criteria
Two crawlers can share one SessionPool, Statistics or BrowserPool and run concurrently.
Re-running a crawler does not reset injected collaborators.
Upgrade notes describing who enters and exits each injected object.
Activity