Repository navigation
Standardize cookie handling #933
Description
Currently we have 3 main cookie handling mechanisms depending on the HTTP client or browser, and none work correctly.
-
HttpxHttpClient.
This solution is closest to expected. Cookies are stored inSession. However, we usedictwhich loses the cookie-domain relationship. This can cause issues during cross-domain crawling. -
CurlImpersonateHttpClient.
Sessionknows nothing about cookies and all cookies are stored at theAsyncSessionlevel. As a result, if we don't use proxies, all sessions have identical cookies. If we work with proxies, cookies become tied to the proxy. -
Playwright.
Sessionknows nothing about cookies and all cookies are stored at thePlaywrightContextlevel, meaning all sessions working from one context will operate with the same cookies.
Activity
[-]Standardize Cookie Handling[/-][+]Standardize cookie handling[/+]B4nan commentedon Jan 27, 2025
@Mantisus thanks for bringing this up. Let's split it into 3 separate PRs please.
janbuchar commentedon Jan 27, 2025
Regarding 1, this will probably involve changing the
Sessionso that it uses a more sophisticated cookie jar implementation. I'm not sure if this can be made backwards compatible...B4nan commentedon Jan 27, 2025
I would focus on 3 first, that feels like the biggest issue to me. Scraping multiple domains in a single crawler is not a very common use case.
janbuchar commentedon Jan 27, 2025
Agreed, even though 2. is similar in terms of severity (but yeah, playwright is a bit more popular)
Mantisus commentedon Jan 30, 2025
2 of 3. But for multi-domain cookie support, we'd really have to go to something like
http.cookiejar.CookieJarinstead ofdict.feat: add support `use_incognito_pages` for `browser_launch_options` …
fix: fix `CurlImpersonateHttpClient` cookies handler (#946)
refactor!: change `Session` cookies from `dict` to `SessionCookies` w…