Commit history

1. Commits on May 2, 2025

   • feat: add `on_skipped_request` decorator, to process links skipped according to `robots.txt` rules (#1166)

     Show description for bd16f14
     

     4 peopleauthored
     bd16f14View commit details

     Copy full SHA for bd16f14

     View code at this pointBrowse repository at this point
2. Commits on Apr 29, 2025

   • docs: Add guide for running crawler in web server (#1174)

     Show description for ec2dd15
     Pijukatel

     authored
     ec2dd15View commit details

     Copy full SHA for ec2dd15

     View code at this pointBrowse repository at this point
3. Commits on Apr 24, 2025

   • feat: add `respect_robots_txt_file` option (#1162)

     Show description for c23f365
     

     4 peopleauthored
     c23f365View commit details

     Copy full SHA for c23f365

     View code at this pointBrowse repository at this point

   • docs: add guide about error handling (#1149)

     Show description for 65dc880
     

     Mantisus

     and

     vdusek

     authored
     65dc880View commit details

     Copy full SHA for 65dc880

     View code at this pointBrowse repository at this point

   • docs: add GCP Cloud Run deployment guide (#1167)

     Show description for 3121f44
     Mantisus

     authored
     3121f44View commit details

     Copy full SHA for 3121f44

     View code at this pointBrowse repository at this point
4. Commits on Apr 10, 2025

   • docs: Add to the documentation guide of login with `HttpCrawler` and `PlaywrightCrawler` (#1142)

     Show description for a01291e
     Mantisus

     authored
     a01291eView commit details

     Copy full SHA for a01291e

     View code at this pointBrowse repository at this point

   • docs: add example resume a crawl (#1143)

     Show description for 40edf70
     Mantisus

     authored
     40edf70View commit details

     Copy full SHA for 40edf70

     View code at this pointBrowse repository at this point
5. Commits on Apr 7, 2025

   • feat: Add `ErrorSnapshotter` to `ErrorTracker` (#1125)

     Show description for 9666092
     Pijukatel

     authored
     9666092View commit details

     Copy full SHA for 9666092

     View code at this pointBrowse repository at this point
6. Commits on Apr 4, 2025

   • docs: add deploy guide for google functions (#1078)

     Show description for f457792
     Mantisus

     authored
     f457792View commit details

     Copy full SHA for f457792

     View code at this pointBrowse repository at this point
7. Commits on Apr 3, 2025

   • feat: Add `requests` argument to `EnqueueLinksFunction` (#1024)

     Show description for fc8444c
     Pijukatel

     authored
     fc8444cView commit details

     Copy full SHA for fc8444c

     View code at this pointBrowse repository at this point

   • docs: Add "Run on Apify" button (#1127)

     Show description for e7ebda7
     vdusek

     authored
     e7ebda7View commit details

     Copy full SHA for e7ebda7

     View code at this pointBrowse repository at this point
8. Commits on Mar 31, 2025

   • feat: add Session binding capability via `session_id` in `Request` (#1086)

     Show description for cda7b31
     Mantisus

     authored
     cda7b31View commit details

     Copy full SHA for cda7b31

     View code at this pointBrowse repository at this point
9. Commits on Mar 18, 2025

   • feat: Add `statistics_log_format` parameter to `BasicCrawler` (#1061)

     Show description for 635ae4a
     Mantisus

     authored
     635ae4aView commit details

     Copy full SHA for 635ae4a

     View code at this pointBrowse repository at this point
10. Commits on Mar 7, 2025

    • feat: Use fingerprint generator in `PlaywrightCrawler` by default (#1060)

      Show description for 09cec53
      

      Pijukatel

      and

      vdusek

      authored
      09cec53View commit details

      Copy full SHA for 09cec53

      View code at this pointBrowse repository at this point
11. Commits on Mar 3, 2025

    • refactor!: move `cli` dependencies to optional dependencies (#1011)

      Show description for 4382959
      Mantisus

      authored
      4382959View commit details

      Copy full SHA for 4382959

      View code at this pointBrowse repository at this point

    • chore: upgrade to Ruff v0.9 (#1026)

      Show description for 55e287f
      vdusek

      authored
      55e287fView commit details

      Copy full SHA for 55e287f

      View code at this pointBrowse repository at this point
12. Commits on Feb 28, 2025

    • refactor!: update status code handling (#1028)

      Show description for 6b59471
      Mantisus

      authored
      6b59471View commit details

      Copy full SHA for 6b59471

      View code at this pointBrowse repository at this point
13. Commits on Feb 25, 2025

    • refactor!: Replace enum with literal for `EnqueueStrategy` (#1019)

      Show description for d2481ef
      vdusek

      authored
      d2481efView commit details

      Copy full SHA for d2481ef

      View code at this pointBrowse repository at this point

    • refactor!: change `Session` cookies from `dict` to `SessionCookies` with `CookieJar` (#984)

      Show description for 6523b3a
      Mantisus

      authored
      6523b3aView commit details

      Copy full SHA for 6523b3a

      View code at this pointBrowse repository at this point

    • refactor!: сhange default `incognito context` to `persistent context` for `Playwright` (#985)

      Show description for f01520d
      Mantisus

      authored
      f01520dView commit details

      Copy full SHA for f01520d

      View code at this pointBrowse repository at this point
14. Commits on Feb 19, 2025

    • feat!: Enable additional status codes arguments to PlaywrightCrawler (#959)

      Show description for 87cf446
      Pijukatel

      authored
      87cf446View commit details

      Copy full SHA for 87cf446

      View code at this pointBrowse repository at this point
15. Commits on Feb 18, 2025

    • chore(deps): update curl-impersonate to 0.9 (#986)

      Show description for a3a2b4f
      Mantisus

      authored
      a3a2b4fView commit details

      Copy full SHA for a3a2b4f

      View code at this pointBrowse repository at this point

    • feat: Add adaptive context helpers (#964)

      Show description for e248f17
      

      Pijukatel

      and

      janbuchar

      authored
      e248f17View commit details

      Copy full SHA for e248f17

      View code at this pointBrowse repository at this point
16. Commits on Feb 12, 2025

    • refactor!: remove Base prefix from abstract class names (#980)

      Show description for 8ccb5d4
      vdusek

      authored
      8ccb5d4View commit details

      Copy full SHA for 8ccb5d4

      View code at this pointBrowse repository at this point

    • chore: rm Apify as optional dependecy (#982)

      vdusek

      authored
      39a26e2View commit details

      Copy full SHA for 39a26e2

      View code at this pointBrowse repository at this point

    • refactor!: remove unused config properties (#978)

      Show description for 4b7fe29
      vdusek

      authored
      4b7fe29View commit details

      Copy full SHA for 4b7fe29

      View code at this pointBrowse repository at this point

    • docs: Add anti-blocking guide (#961)

      Show description for 207c983
      Pijukatel

      authored
      207c983View commit details

      Copy full SHA for 207c983

      View code at this pointBrowse repository at this point
17. Commits on Feb 11, 2025

    • docs: Set line length to docs related code to 90 (#973)

      Show description for a2e8496
      Pijukatel

      authored
      a2e8496View commit details

      Copy full SHA for a2e8496

      View code at this pointBrowse repository at this point
18. Commits on Feb 7, 2025

    • feat: Add AdaptivePlaywrightCrawler (#872)

      Show description for 5ba70b6
      

      Pijukatel

      and

      janbuchar

      authored
      5ba70b6View commit details

      Copy full SHA for 5ba70b6

      View code at this pointBrowse repository at this point
19. Commits on Feb 5, 2025

    • feat: Integrate browserforge fingerprints (#829)

      Show description for 2b156b4
      

      3 peopleauthored
      2b156b4View commit details

      Copy full SHA for 2b156b4

      View code at this pointBrowse repository at this point
20. Commits on Jan 30, 2025

    • docs: update setting up and running in cloud (#936)

      Show description for 2f9dae4
      vdusek

      authored
      2f9dae4View commit details

      Copy full SHA for 2f9dae4

      View code at this pointBrowse repository at this point
21. Commits on Jan 27, 2025

    • feat: add `transform_request_function` for `enqueue_links` (#923)

      Show description for 6b15957
      

      3 peopleauthored
      6b15957View commit details

      Copy full SHA for 6b15957

      View code at this pointBrowse repository at this point
22. Commits on Jan 23, 2025

    • feat: add `block_requests` helper for `PlaywrightCrawler` (#919)

      Show description for 1030459
      

      Mantisus

      and

      vdusek

      authored
      1030459View commit details

      Copy full SHA for 1030459

      View code at this pointBrowse repository at this point
23. Commits on Jan 22, 2025

    • feat: Add keep_alive flag to `crawler.__init__` (#921)

      Show description for 7a82d0c
      Pijukatel

      authored
      7a82d0cView commit details

      Copy full SHA for 7a82d0c

      View code at this pointBrowse repository at this point
24. Commits on Jan 20, 2025

    • docs: fix typo (#920)

      Show description for 5a7158e
      Mantisus

      authored
      5a7158eView commit details

      Copy full SHA for 5a7158e

      View code at this pointBrowse repository at this point