{{ message }}
Repository navigation
Commit c2013a0
authored
feat: support timeouts in the
Closes #2138
## Summary
`SendRequestFunction` (used via `context.send_request()` in request
handlers) only accepted `url`, `method`, `payload` and `headers`, so a
handler could not bound how long an extra HTTP call may take. All four
HTTP client implementations (`HttpxHttpClient`,
`CurlImpersonateHttpClient`, `ImpitHttpClient`, `PlaywrightHttpClient`)
already support a per-request `timeout: timedelta | None` — this PR just
exposes it through the public contract.
## Changes
- `src/crawlee/_types.py`: add `timeout: timedelta | None = None` to the
`SendRequestFunction` protocol + docstring.
- `src/crawlee/crawlers/_basic/_basic_crawler.py`: thread `timeout`
through the `_prepare_send_request_function` closure into
`HttpClient.send_request`.
- `src/crawlee/crawlers/_playwright/_playwright_http_client.py`: **fix**
— `PlaywrightHttpClient` passed `timeout.total_seconds()` to
Playwright's `APIRequestContext.fetch`, which expects **milliseconds**.
Without this fix, any timeout passed via `context.send_request()` on a
Playwright crawler would be ~1000x too short (e.g.
`timedelta(seconds=12)` → 12ms).
- `tests/unit/crawlers/_basic/test_basic_crawler.py`: regression test —
`context.send_request('/slow?delay=2',
timeout=timedelta(milliseconds=100))` raises `asyncio.TimeoutError`;
plus a unit test asserting `timeout` is forwarded to
`HttpClient.send_request`.
- `tests/unit/crawlers/_playwright/test_playwright_http_client.py`:
regression test asserting the seconds→milliseconds conversion.
## Validation
- `pytest tests/unit/crawlers/_basic/test_basic_crawler.py` → 112
passed, 1 skipped
- New tests (forwarding, real timeout, Playwright ms conversion) all
pass
- `ruff check` + `ruff format --check` clean on all touched files
Note: no new dependency; `timeout` uses the existing `timedelta`
convention across the Python HTTP clients (the JS implementation names
it `timeoutMillis`, but the Python clients are timedelta-based).send_request context helper (#2174)1 parent 3332897 commit c2013a0
5 files changed
Lines changed: 87 additions & 2 deletions
File tree
- src/crawlee
- crawlers
- _basic
- _playwright
- tests/unit/crawlers
- _basic
- _playwright
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1321 | 1321 | | |
1322 | 1322 | | |
1323 | 1323 | | |
| 1324 | + | |
1324 | 1325 | | |
1325 | 1326 | | |
1326 | 1327 | | |
| |||
1329 | 1330 | | |
1330 | 1331 | | |
1331 | 1332 | | |
| 1333 | + | |
1332 | 1334 | | |
1333 | 1335 | | |
1334 | 1336 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
99 | 99 | | |
100 | 100 | | |
101 | 101 | | |
102 | | - | |
| 102 | + | |
103 | 103 | | |
104 | 104 | | |
105 | 105 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
| 31 | + | |
31 | 32 | | |
32 | 33 | | |
33 | 34 | | |
| |||
523 | 524 | | |
524 | 525 | | |
525 | 526 | | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
| 542 | + | |
| 543 | + | |
| 544 | + | |
| 545 | + | |
| 546 | + | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
| 550 | + | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
| 554 | + | |
| 555 | + | |
| 556 | + | |
| 557 | + | |
| 558 | + | |
| 559 | + | |
| 560 | + | |
| 561 | + | |
| 562 | + | |
| 563 | + | |
| 564 | + | |
| 565 | + | |
| 566 | + | |
| 567 | + | |
| 568 | + | |
| 569 | + | |
| 570 | + | |
| 571 | + | |
| 572 | + | |
| 573 | + | |
| 574 | + | |
| 575 | + | |
| 576 | + | |
| 577 | + | |
| 578 | + | |
526 | 579 | | |
527 | 580 | | |
528 | 581 | | |
| |||

0 commit comments