Background

Currently, our approach to HTTP fingerprinting is fragmented across different components. This leads to potential inconsistencies where, for example, HTTP headers might not align with TLS fingerprints or device characteristics, making our scrapers easier to detect. Furthermore, tracking down the code responsible for various parts of the fingerprinting functionality is difficult.

Objective

Create a unified approach to HTTP fingerprinting across all Crawlee components to produce more realistic and consistent scraper behavior. This will be ported to JS crawlee as a part of v4.

Proposed Solution

1. Create a FingerprintProfile data structure that encapsulates:

   • HTTP headers collection
   • Browser type and version (for TLS impersonation)
   • Device characteristics (viewport, screen resolution, etc.)
   • Proxy configuration that aligns with the fingerprint's locale/behavior
   • potentially any other stuff I forgot about or that will be added later on

2. Integrate this structure across Crawlee components:

   • components responsible for fingerprinting should accept a FingerprintProfile instance in the API responsible for handling individual requests
   • HTTP clients should apply appropriate headers and proxy settings
   • Browser Pool should select browsers with matching TLS fingerprints and inject appropriate DOM properties (viewport, locale, ...)
   • the FingerprintProfile should probably be included in the Session objects
   • the way the FingerprintProfile is generated should be configurable, ideally in a way that allows adding custom code

@Pijukatel @vdusek @B4nan