Store the proxy in `Session` and drop session and request arguments from `ProxyConfiguration` · Issue #2298 · apify/crawlee-python · GitHub
Skip to content

Store the proxy in Session and drop session and request arguments from ProxyConfiguration #2298

Open

Description

@vdusek

The crawler resolves a proxy for every request with ProxyConfiguration.new_proxy_info(session_id, request, proxy_tier) (_basic_crawler.py _get_proxy_info). Session stickiness lives in ProxyConfiguration._used_proxy_urls, an in-memory dict keyed by session ID, so it is lost on migration even when the session pool is persisted.

JS v4 assigns the proxy once, when a session is created, and stores it in the session.

Proposed solution

  • Add Session.proxy_info, set by the session pool when it creates a session and persisted in SessionModel. Crawlers read the proxy from the session.
  • ProxyConfiguration.new_proxy_info() / new_url() take no session ID or request. A restored session passes its previous ProxyInfo back so the configuration can reuse it.
  • new_url_function takes no arguments; remove ProxyInfo.session_id and _used_proxy_urls.
  • Warn when a crawler gets both a custom session_pool and a proxy_configuration, since the pool decides the proxy.

The protocol extraction itself is #2137. Removing proxy tiers is a separate follow-up issue.

JS references

This will be a breaking change.

✍️ Drafted by Claude Code

Activity

  1. added this to the 2.0 milestone on Oct 13, 2026
  2. added
    enhancementNew feature or request.
    t-toolingIssues with this label are in the ownership of the tooling team.
    on Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request.t-toolingIssues with this label are in the ownership of the tooling team.

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions