{{ message }}
Repository navigation
Redesign browser pool and the whole browsers subpackage #1744
Description
❗ Note: This is a starting point for discussion, not a final decision. The team should review and adjust before any implementation begins.❗
Context
The browser pool and the whole browsers subpackage have several architectural issues and poor abstractions. These make it difficult to extend (e.g., remote browser support in #1743 and maybe even lifecycle hooks in #1741). We've discussed these issues a few times already. With the introduction of #1743, this may be a good opportunity to finally establish a well-structured and robust design for the browser pool. This effort should also be coordinated with the new user pool initiative in #1435.
Reference
- Python:
src/crawlee/browsers/ - JS BrowserPool:
packages/browser-pool/src/browser-pool.ts - JS abstract classes:
packages/browser-pool/src/abstract-classes/ - JS LaunchContext:
packages/browser-pool/src/launch-context.ts - JS fingerprinting hooks:
packages/browser-pool/src/fingerprinting/hooks.ts
This will be a breaking change.
Activity
[-]Redesign browsers subpackage[/-][+]Redesign browser pool and the whole browsers subpackage[/+]BrowserPluginvsBrowserFactory#179 as a duplicate of this issuePijukatel commentedon Mar 5, 2026
Also to consider is a scenario like this: #1776 , where there is a pre-existing persistent browser context created outside of our code.
Possible fix: #1778
vdusek commentedon Mar 5, 2026
cc @janbuchar, @l2ysho, as I believe you may have already been working on this.
janbuchar commentedon Mar 9, 2026
I believe we should first come up with the "ideal" interface of a browser pool. Essentially, the
BrowserCrawlerin JS only callsnewPageandretireBrowserControllerin case of a session error. Then, theBrowserControlleris used mostly for cookie management.The rest of the interface is either unused, or used by downstream users.
Then there is the Controller and Plugin infrastructure, and I'm not sure if I can reliably describe what those do.