Let crawlers manage only the collaborators they create and extract per-run state · Issue #2301 · apify/crawlee-python · GitHub
Skip to content

Let crawlers manage only the collaborators they create and extract per-run state #2301

Open

Description

@vdusek

BasicCrawler decides whether to enter and exit a collaborator by whether it is already active (_basic_crawler.py around line 832), not by who created it. An injected Statistics, SessionPool or BrowserPool that is not yet active is entered and torn down by the first crawler that runs, so sharing one between crawlers breaks: the first to finish exits it while the other still uses it. A user-supplied Statistics and SessionPool are also reset on every repeated run() (around line 728). The Playwright crawler always enters a passed browser_pool, and AdaptivePlaywrightCrawler initializes and clears a passed rendering_type_predictor.

Per-run state lives on the crawler itself (_running, _has_finished_before, _last_run_failed, _unexpected_stop and others), and repeated runs depend on resetting it by hand.

JS v4 settled both: a crawler owns, starts, resets and stops only what it built, and each run() gets a fresh run object.

Proposed solution

  • Track owned vs injected collaborators explicitly. The crawler creates and manages its defaults; injected instances are borrowed, so the caller enters and exits them, and the crawler never resets them.
  • Apply this to the event manager, statistics, session pool, HTTP client, browser pool, rendering type predictor and request manager.
  • Extract per-run state into an internal CrawlerRun built on every run().

Acceptance criteria

  • Two crawlers can share one SessionPool, Statistics or BrowserPool and run concurrently.
  • Re-running a crawler does not reset injected collaborators.
  • Upgrade notes describing who enters and exits each injected object.

JS references

Related: #2133 and #2132 build on this.

This will be a breaking change.

✍️ Drafted by Claude Code

Activity

  1. added this to the 2.0 milestone on Oct 13, 2026
  2. added
    t-toolingIssues with this label are in the ownership of the tooling team.
    debtCode quality improvement or decrease of technical debt.
    on Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    debtCode quality improvement or decrease of technical debt.t-toolingIssues with this label are in the ownership of the tooling team.

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions