You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Store the proxy in Session and drop session and request arguments from ProxyConfiguration #2298
The crawler resolves a proxy for every request with ProxyConfiguration.new_proxy_info(session_id, request, proxy_tier) (_basic_crawler.py_get_proxy_info). Session stickiness lives in ProxyConfiguration._used_proxy_urls, an in-memory dict keyed by session ID, so it is lost on migration even when the session pool is persisted.
JS v4 assigns the proxy once, when a session is created, and stores it in the session.
Proposed solution
Add Session.proxy_info, set by the session pool when it creates a session and persisted in SessionModel. Crawlers read the proxy from the session.
ProxyConfiguration.new_proxy_info() / new_url() take no session ID or request. A restored session passes its previous ProxyInfo back so the configuration can reuse it.
new_url_function takes no arguments; remove ProxyInfo.session_id and _used_proxy_urls.
Warn when a crawler gets both a custom session_pool and a proxy_configuration, since the pool decides the proxy.
The protocol extraction itself is #2137. Removing proxy tiers is a separate follow-up issue.
Activity
refactor!: remove proxy tiers and `Request.forefront` (#2336)