…apify#1136)

### Description
- Add e2e tests for playwright templates
- Add e2e tests for pip-based templates
- Fix template issue with pip template that has additional dependencies
- Separate all e2e tests in CI to run in it's own runner to allow
selective re-triggers (e2e tests work on real network and it will always
have some flakiness due to that)
- Reduce `max_requests_per_crawl` to 10 in all templates to be aligned
with doc examples and to make the tests faster.

### Issues

- Closes: apify#1109

### Description

- Added `ErrorSnapshotter` that can take page snapshot (screenshot or
html) on each first encountered unique error.
- Added documentation describing how to use it.

### Issues

- Closes: apify#151

### Testing

- Added unit tests.
- Example `PlaywrightCrawler` based actor run with `ErrorSnapshotter`:
https://console.apify.com/actors/C0lWh1UCQvgdArp6R/runs/UNuaiRWBDgxiJau0U#storage

### Description

- Add checks to ensure that the selected package manager is installed
and executable.
- Suppress cookie-cutter logs when a failure occurs.

### Issues

- Closes: apify#1138

This PR contains the following updates:

| Update | Change |
|---|---|
| lockFileMaintenance | All locks refreshed |

🔧 This Pull Request updates lock files to use the latest dependency
versions.

---

### Configuration

📅 **Schedule**: Branch creation - "before 4am on monday" (UTC),
Automerge - At any time (no schedule defined).

🚦 **Automerge**: Enabled.

♻ **Rebasing**: Whenever PR is behind base branch, or you tick the
rebase/retry checkbox.

👻 **Immortal**: This PR will be recreated if closed unmerged. Get
[config
help](https://redirect.github.com/renovatebot/renovate/discussions) if
that's undesired.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR was generated by [Mend Renovate](https://mend.io/renovate/).
View the [repository job
log](https://developer.mend.io/github/apify/crawlee-python).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzOS4yMzUuMiIsInVwZGF0ZWRJblZlciI6IjM5LjIzNS4yIiwidGFyZ2V0QnJhbmNoIjoibWFzdGVyIiwibGFiZWxzIjpbXX0=-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

…1137)

### Description

- add example resume a crawl

### Issues

- Closes: apify#901

…`PlaywrightCrawler` (apify#1142)

### Description

- Add to the documentation examples of login with `HttpCrawler` and
`PlaywrightCrawler`

### Issues

- Closes: apify#870

### Description

Update `UnprocessedRequest` to match actual data.
Add test.

### Issues

- Closes: apify#1150

… and the handler is executed for `PlaywrightCrawler` (apify#1163)

### Description

- For `PlaywrightCrawler`, cookies should only be saved to the session
store when the handler is fully executed. This is because the browser
may continue to set cookies while the handler is being executed

### Testing

- Add a test simulating the installation of a cookie in the browser
during the `default_handler` execution process
- Update the `test_isolation_cookies` test

### Description
Adds retry to unprocessed requests in call `add_requests_batched`.
Retry calls recursively `_process_batch`, which initially works on full
request batch and then on batches of unprocessed requests until retry
limit is reached or all requests are processed. Each retry is done after
linearly increasing delay with each attempt.

Unprocessed requests are not counted in `request_queue.get_total_count`
Add test.

### Issues

- Closes: [Handle unprocessed requests in
batch_add_requests](apify/apify-sdk-python#456)

…tation count exceeds maximum (apify#1147)

### Description

- Call `failed_request_handler` for `SessionError` when session rotation
count exceeds maximum

### Description

- Add GCP Cloud Run deployment guide

### Issues

- Closes: apify#707

### Description

- add guide about error handling

---------

Co-authored-by: Vlada Dusek <v.dusek96@gmail.com>

…PlaywrightCrawler` using `APIRequestContext` bound to the browser context (apify#1134)

### Description

- This PR, adds an internal `HttpClient` implementation for
`PlaywrightCrawler` using the `APIRequestContext` from `Playwright`.
This ensures that HTTP requests use the same proxies as the browser
context.

### Issues

- Closes: apify#928

### Description
Update `test_enqueue_links` to use test server.
(The test was still using real page instead of test server and the test
started being blocked by `Cloudflare` when running in CI)

### Description

Make timeout based error logs in request handler more readable by
filtering clutter in stack trace.
Add one line warning with optional most important stack trace line for
failed request handler calls.

### Issues

- Closes: apify#1158

…orTracker` (apify#1181)

### Description

- Fix getting an error message for errors not having message data in
`args`. If the error was raised in a chain, tries to get the previous
error message from `__context__`. Otherwise uses the error class name as
the message.

### Issues

- Closes: apify#1179 

### Testing

- Add new tests

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

### Description

Add guide for running crawler in web server

### Issues

- Closes: apify#1148

- closes apify#809, but I chose to make it a regular class and not a mixin
(the initial attempt was super cumbersome and clashed with existing
public interface of `Statistics` and `SessionPool`).
- will help with apify#99 

## Key Changes
- Introduced a new `RecoverableState` helper class that:
  - Uses Pydantic models for state representation and validation
  - Handles state persistence to a `KeyValueStore`
- Automatically registers/deregisters with event system for state
persistence

- Refactored existing components to use the new abstraction:
- Converted `SessionPool` to use `RecoverableState` for state management
- Converted `Statistics` to use `RecoverableState` for state management
- Converted `KeyValueStore` to use `RecoverableState` for autosaved
values

## Benefits
- More consistent handling of state persistence across the library
- Reduced code duplication for state management logic
- Better separation of concerns between state data and persistence logic

### Description

Temporarily add bounded `certifi` dependency until certification issues
on latest release of `certifi` are fully resolved.
(`certifi` was indirect dependency of `crawlee` that was not version
bounded by any of the direct dependencies).

### Issues

- Related: apify#1182

…cording to `robots.txt` rules (apify#1166)

### Description

- This PR supplements apify#1162 by adding an `on_skipped_request` decorator
to handle references skipped according to `robots.txt` rules

### Issues

- Closes: apify#1160
- Related apify#1162

---------

Co-authored-by: Vlada Dusek <v.dusek96@gmail.com>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Apify Release Bot <noreply@apify.com>

…entation (apify#1192)

### Description

Make it possible to apply `Configuration` to the Playwright crawler.
Added a test.

### Issues

- Closes: apify#1185

…d to it

This PR contains the following updates:

| Update | Change |
|---|---|
| lockFileMaintenance | All locks refreshed |

🔧 This Pull Request updates lock files to use the latest dependency
versions.

---

### Configuration

📅 **Schedule**: Branch creation - "before 4am on monday" (UTC),
Automerge - At any time (no schedule defined).

🚦 **Automerge**: Enabled.

♻ **Rebasing**: Whenever PR is behind base branch, or you tick the
rebase/retry checkbox.

👻 **Immortal**: This PR will be recreated if closed unmerged. Get
[config
help](https://redirect.github.com/renovatebot/renovate/discussions) if
that's undesired.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR was generated by [Mend Renovate](https://mend.io/renovate/).
View the [repository job
log](https://developer.mend.io/github/apify/crawlee-python).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0MC4xMS45IiwidXBkYXRlZEluVmVyIjoiNDAuMTEuOSIsInRhcmdldEJyYW5jaCI6Im1hc3RlciIsImxhYmVscyI6W119-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>