Comparing master@{1day}...master · apify/crawlee-python · GitHub
Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: apify/crawlee-python
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: master@{1day}
Choose a base ref
...
head repository: apify/crawlee-python
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: master
Choose a head ref
  • 4 commits
  • 15 files changed
  • 3 contributors

Commits on Oct 26, 2026

  1. fix: ignore a UTF-8 byte order mark in robots.txt (#2351) 

    ### Description

- `RobotsTxtFile.load` decodes the response body with `utf-8`, which
keeps a leading UTF-8 byte order mark (BOM) as `\ufeff`. Protego then
does not recognize the first line (`\ufeffUser-agent: *`) as a
`User-agent` directive, so the rules of the first group are ignored and
`is_allowed` returns `True` for disallowed paths. With
`respect_robots_txt_file=True`, the crawler fetches pages the site
disallows.
- The body is now decoded with `utf-8-sig`, which drops the BOM if
present and is otherwise identical to `utf-8`.
- Adds a regression test that loads a robots.txt with a BOM through the
test server.

### Issues

- Closes: #2350

### Testing

- New test `test_load_robots_txt_with_utf8_bom` in
`tests/unit/_utils/test_robots.py`. It fails on `master` (`assert not
robots.is_allowed(...)` gets `True`) and passes with the fix, for all
HTTP client parametrizations.
- `uv run pytest tests/unit/_utils/test_robots.py`: 20 passed.
- `ruff check`, `ruff format --check`, and `ty check` pass on the
changed files.

### Checklist

- [ ] CI passed
    @RaphaelFakhri
    RaphaelFakhri authored Oct 26, 2026
    Configuration menu
    Copy the full SHA
    3e6ee66 View commit details
    Browse the repository at this point in the history

  2. chore(release): Update changelog and package version [skip ci]

    @apify-service-account
    apify-service-account authored Oct 26, 2026
    Configuration menu
    Copy the full SHA
    0555144 View commit details
    Browse the repository at this point in the history

  3. chore(deps): lock file maintenance (#2353)

    @renovate
    renovate[bot] authored Oct 26, 2026
    Configuration menu
    Copy the full SHA
    449b82b View commit details
    Browse the repository at this point in the history

  4. chore(deps): update dependencies and GitHub Actions (#2354)

    @renovate
    renovate[bot] authored Oct 26, 2026
    Configuration menu
    Copy the full SHA
    cd6f748 View commit details
    Browse the repository at this point in the history
Loading