Reject absolute-form targets without an authority before parsing by asvetlov · Pull Request #13858 · aio-libs/aiohttp · GitHub
Skip to content

Reject absolute-form targets without an authority before parsing - #13858

Merged
asvetlov merged 1 commit into
aio-libs:masterfrom
asvetlov:drop-triple-slash-invalid-url
Sep 29, 2026
Merged

asvetlov merged 1 commit into
aio-libs:masterfrom
asvetlov:drop-triple-slash-invalid-url

Conversation

@asvetlov

Copy link
Copy Markdown
Member

What do these changes do?

Follow-up to #13855 for aio-libs/yarl#1937. With that change, yarl's default WHATWG mode reads a host from http:host/p, http:/host/p and http:///host/p, as browsers do. The Python and Cython request parsers passed an absolute-form request target to URL() and rejected it only if yarl found no host. They now first check that the target has // followed by a non-empty authority, so GET http:///protected and GET http:/protected stay rejected as RFC 9110 requires, whatever yarl version is installed.

The client tests used http:///example.com and http:///path as URLs without a host. They now use http:// where an invalid URL is needed, and http:///example.com is dropped from the invalid URL test data. With yarl#1937 it is http://example.com/, and as a redirect it leads to example.com, as in browsers.

Are there changes in behavior for the user?

No with the released yarl, where such targets were already rejected. With yarl#1937, the server keeps rejecting absolute-form targets without an authority instead of reading a host from them.

Is it a substantial burden for the maintainers to support this?

No, it is one string check shared by both parsers.

Related issue number

Follow-up to #13855; needed by aio-libs/yarl#1937.

Checklist

  • I think the code is well written
  • Unit tests for the changes exist
  • Documentation reflects the changes: N/A, no documented behavior changes
  • If you provide code modification, please add yourself to CONTRIBUTORS.txt: N/A, already listed
  • Add a new news fragment into the CHANGES/ folder

Drafted with Claude Code (Claude Opus 5.5); reviewed by @asvetlov.

Agent run details (optional, for reviewers)

Tests with the Cython extensions built (llhttp generated, make cythonize, build_ext --inplace): pytest tests -n auto, 5246 passed, 22 skipped, 17 xfailed, both with yarl master (f761c37) and with the yarl#1937 branch (8d0f111). Pure-Python run of test_http_parser.py, test_client_functional.py and test_client_session.py with the yarl#1937 branch: 884 passed.
Lint: black, isort, codespell passed. The local pre-commit flake8 hook fails to load flake8-requirements (pkg_resources missing), so flake8 (E, F, W) was run directly and passed. yesqa was skipped for the same reason, because it wrongly strips # noqa: I900.

yarl's WHATWG mode reads a host from "http:host/p" and "http:///host/p", so the request parsers check for "//" and a non-empty authority first, as RFC 9110 requires a host. The invalid URL test data no longer relies on "http:///example.com" having no host.
@pyba-chronographer pyba-chronographer Bot added the bot:chronographer:provided There is a change note present in this PR label Sep 29, 2026
@codecov

codecov Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

@asvetlov
asvetlov marked this pull request as ready for review September 29, 2026 08:43
@asvetlov
asvetlov requested a review from webknjaz as a code owner September 29, 2026 08:43
@asvetlov asvetlov added backport-3.14 Trigger automatic backporting to the 3.14 release branch by Patchback robot backport-3.15 Trigger automatic backporting to the 3.15 release branch by Patchback robot labels Sep 29, 2026
@codspeed

codspeed Bot commented Sep 29, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 97 untouched benchmarks
⏩ 83 skipped benchmarks1


Comparing asvetlov:drop-triple-slash-invalid-url (09201c3) with master (44bbdb8)

Open in CodSpeed

Footnotes

  1. 83 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@greptile-apps

greptile-apps Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

RetriggerConfidence Score: 5/5

[Critical risk] Adds validation to HTTP request target parsing before URL parsing.

No blocking issues identified; safe to merge.

What we checked:

  • Ran general contract validation to examine how the guard handles backslashes and target acceptance, and observed that the guard alone does not reject the backslash; invalid targets returned 400 in both runs and the valid target returned 200. T-Rex
  • Compared two runs and confirmed both exited with code 0; identified a Python/Cython discrepancy for input http://example.com#f (Python accepted, Cython rejected) that was identical on the PR base and head, so it is not a PR-introduced finding. T-Rex

Reviews (1) · Last reviewed commit: "Reject absolute-form targets without an ..."

@asvetlov
asvetlov merged commit ce960dd into aio-libs:master Sep 29, 2026
58 of 60 checks passed
@patchback

patchback Bot commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Backport to 3.15: 💚 backport PR created

✅ Backport PR branch: patchback/backports/3.15/ce960dd17e34f143762c14c32b9e14435ea7ce11/pr-13858

Backported as #13859

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

@patchback

patchback Bot commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

asvetlov added a commit that referenced this pull request Sep 29, 2026
…out an authority before parsing (#13859)

**This is a backport of PR #13858 as merged into master
(ce960dd).**

<!-- Thank you for your contribution! -->

## What do these changes do?

Follow-up to #13855 for aio-libs/yarl#1937. With that change, yarl's
default WHATWG mode reads a host from `http:host/p`, `http:/host/p` and
`http:///host/p`, as browsers do. The Python and Cython request parsers
passed an absolute-form request target to `URL()` and rejected it only
if yarl found no host. They now first check that the target has `//`
followed by a non-empty authority, so `GET http:///protected` and `GET
http:/protected` stay rejected as RFC 9110 requires, whatever yarl
version is installed.

The client tests used `http:///example.com` and `http:///path` as URLs
without a host. They now use `http://` where an invalid URL is needed,
and `http:///example.com` is dropped from the invalid URL test data.
With yarl#1937 it is `http://example.com/`, and as a redirect it leads
to `example.com`, as in browsers.

## Are there changes in behavior for the user?

No with the released yarl, where such targets were already rejected.
With yarl#1937, the server keeps rejecting absolute-form targets without
an authority instead of reading a host from them.

## Is it a substantial burden for the maintainers to support this?

No, it is one string check shared by both parsers.

## Related issue number

Follow-up to #13855; needed by aio-libs/yarl#1937.

## Checklist

- [x] I think the code is well written
- [x] Unit tests for the changes exist
- [ ] Documentation reflects the changes: N/A, no documented behavior
changes
- [ ] If you provide code modification, please add yourself to
`CONTRIBUTORS.txt`: N/A, already listed
- [x] Add a new news fragment into the `CHANGES/` folder

Drafted with Claude Code (Claude Opus 5.5); reviewed by @asvetlov.

<details>
<summary>Agent run details (optional, for reviewers)</summary>

Tests with the Cython extensions built (llhttp generated, `make
cythonize`, `build_ext --inplace`): `pytest tests -n auto`, 5246 passed,
22 skipped, 17 xfailed, both with yarl master (f761c37) and with the
yarl#1937 branch (8d0f111). Pure-Python run of `test_http_parser.py`,
`test_client_functional.py` and `test_client_session.py` with the
yarl#1937 branch: 884 passed.
Lint: black, isort, codespell passed. The local pre-commit flake8 hook
fails to load `flake8-requirements` (`pkg_resources` missing), so flake8
(E, F, W) was run directly and passed. yesqa was skipped for the same
reason, because it wrongly strips `# noqa: I900`.
</details>

Co-authored-by: Andrew Svetlov <andrew.svetlov@gmail.com>
asvetlov added a commit that referenced this pull request Sep 29, 2026
) (#13860)

<!-- Thank you for your contribution! -->

Follow-up to #13855 for aio-libs/yarl#1937. With that change, yarl's
default WHATWG mode reads a host from `http:host/p`, `http:/host/p` and
`http:///host/p`, as browsers do. The Python and Cython request parsers
passed an absolute-form request target to `URL()` and rejected it only
if yarl found no host. They now first check that the target has `//`
followed by a non-empty authority, so `GET http:///protected` and `GET
http:/protected` stay rejected as RFC 9110 requires, whatever yarl
version is installed.

The client tests used `http:///example.com` and `http:///path` as URLs
without a host. They now use `http://` where an invalid URL is needed,
and `http:///example.com` is dropped from the invalid URL test data.
With yarl#1937 it is `http://example.com/`, and as a redirect it leads
to `example.com`, as in browsers.

No with the released yarl, where such targets were already rejected.
With yarl#1937, the server keeps rejecting absolute-form targets without
an authority instead of reading a host from them.

No, it is one string check shared by both parsers.

Follow-up to #13855; needed by aio-libs/yarl#1937.

- [x] I think the code is well written
- [x] Unit tests for the changes exist
- [ ] Documentation reflects the changes: N/A, no documented behavior
changes
- [ ] If you provide code modification, please add yourself to
`CONTRIBUTORS.txt`: N/A, already listed
- [x] Add a new news fragment into the `CHANGES/` folder

Drafted with Claude Code (Claude Opus 5.5); reviewed by @asvetlov.

<details>
<summary>Agent run details (optional, for reviewers)</summary>

Tests with the Cython extensions built (llhttp generated, `make
cythonize`, `build_ext --inplace`): `pytest tests -n auto`, 5246 passed,
22 skipped, 17 xfailed, both with yarl master (f761c37) and with the
yarl#1937 branch (8d0f111). Pure-Python run of `test_http_parser.py`,
`test_client_functional.py` and `test_client_session.py` with the
yarl#1937 branch: 884 passed.
Lint: black, isort, codespell passed. The local pre-commit flake8 hook
fails to load `flake8-requirements` (`pkg_resources` missing), so flake8
(E, F, W) was run directly and passed. yesqa was skipped for the same
reason, because it wrongly strips `# noqa: I900`.
</details>

(cherry picked from commit ce960dd)

<!-- Thank you for your contribution! -->

## What do these changes do?

<!-- Please give a short brief about these changes. -->

## Are there changes in behavior for the user?

<!-- Outline any notable behaviour for the end users. -->

## Is it a substantial burden for the maintainers to support this?

<!--
Stop right there! Pause. Just for a minute... Can you think of anything
obvious that would complicate the ongoing development of this project?

Try to consider if you'd be able to maintain it throughout the next
5 years. Does it seem viable? Tell us your thoughts! We'd very much
love to hear what the consequences of merging this patch might be...

This will help us assess if your change is something we'd want to
entertain early in the review process. Thank you in advance!
-->

## Related issue number

<!-- Will this resolve any open issues? -->
<!-- Remember to prefix with 'Fixes' if it closes an issue (e.g. 'Fixes
#123'). -->

## Checklist

- [ ] I think the code is well written
- [ ] Unit tests for the changes exist
- [ ] Documentation reflects the changes
- [ ] If you provide code modification, please add yourself to
`CONTRIBUTORS.txt`
  * The format is &lt;Name&gt; &lt;Surname&gt;.
  * Please keep alphabetical order, the file is sorted by names.
- [ ] Add a new news fragment into the `CHANGES/` folder
  * name it `<issue_or_pr_num>.<type>.rst` (e.g. `588.bugfix.rst`)
  * if you don't have an issue number, change it to the pull request
    number after creating the PR
    * `.bugfix`: A bug fix for something the maintainers deemed an
      improper undesired behavior that got corrected to match
      pre-agreed expectations.
    * `.feature`: A new behavior, public APIs. That sort of stuff.
    * `.deprecation`: A declaration of future API removals and breaking
      changes in behavior.
    * `.breaking`: When something public is removed in a breaking way.
      Could be deprecated in an earlier release.
    * `.doc`: Notable updates to the documentation structure or build
      process.
    * `.packaging`: Notes for downstreams about unobvious side effects
      and tooling. Changes in the test invocation considerations and
      runtime assumptions.
    * `.contrib`: Stuff that affects the contributor experience. e.g.
      Running tests, building the docs, setting up the development
      environment.
    * `.misc`: Changes that are hard to assign to any of the above
      categories.
  * Make sure to use full sentences with correct case and punctuation,
    for example:
    ```rst
    Fixed issue with non-ascii contents in doctest text files
    -- by :user:`contributor-gh-handle`.
    ```

    Use the past tense or the present tense a non-imperative mood,
    referring to what's changed compared to the last released version
    of this project.
asvetlov added a commit to aio-libs/yarl that referenced this pull request Sep 29, 2026
<!-- Thank you for your contribution! -->

## What do these changes do?

In the default WHATWG mode, `http:example.com/`, `http:/example.com/`
and `http:///example.com/` now parse as `http://example.com/`, as the
WHATWG URL Standard does for the `http`, `https`, `ws`, `wss` and `ftp`
schemes (the "special authority ignore slashes" state). Previously the
text stayed the path of a URL without an authority and printed as
`http:///example.com/`. Userinfo or a port without a host
(`http:@/www.example.com`) is rejected, as for `http://@/x`. `http:`,
`http:/`, `http:?q`, `http://` and `http:///` are kept as they were,
since there is no host to read. An input with `//` is absolute against
any base, as in WHATWG, so `join()` no longer keeps the base host for
`http:///example.com/`.

Against a base with the same scheme, WHATWG resolves the same input as a
relative reference (`http:foo.com` against `http://example.org/foo/bar`
is `http://example.org/foo/foo.com`). `URL.join()` only sees the parsed
reference, so a URL parsed this way keeps its original path in the cache
(and in its pickle state), and `join()` uses it when the schemes match.
RFC 3986 mode is unchanged: it already keeps these URLs without an
authority.

In `tools/conformance/REPORT.md`, 33 rows move from "WHATWG mode differs
from WHATWG, RFC mode follows RFC 3986" to "yarl follows each standard
in its mode" (28 to 61). Four inputs now fail in WHATWG mode without
changing section. Three are `http:\\host\` (a backslash in the authority
is rejected; backslash handling is a separate change). The fourth is
`http:[61:27]/:foo` against an http base, which WHATWG only accepts
relative to the base, while yarl parses the reference alone first. "yarl
differs, RFC 3986 and WHATWG agree" is unchanged.

## Are there changes in behavior for the user?

Yes. In WHATWG mode, `URL("http:example.com/")` has host `example.com`,
and `URL("http:@/x")` raises `ValueError`.
`URL(base).join(URL("http:g"))` for an http base gives the same result
as before.

## Is it a substantial burden for the maintainers to support this?

No. The conversion is one helper called from both parsers, and join()
checks one cache key for references with an authority.

## Related issue number

Follow-up to #1931 and #1936. The "Aiohttp tests" job needs
aio-libs/aiohttp#13855 (merged) and aio-libs/aiohttp#13858, which keeps
aiohttp's server rejecting absolute-form request targets without an
authority.

## Checklist

- [x] I think the code is well written
- [x] Unit tests for the changes exist
- [x] Documentation reflects the changes
- [ ] If you provide code modification, please add yourself to
`CONTRIBUTORS.txt`
- [x] Add a new news fragment into the `CHANGES/` folder

Drafted with Claude Code (Claude Opus 5.5); reviewed by @asvetlov.

<details>
<summary>Agent run details (optional, for reviewers)</summary>

Tests: `pytest tests` 2699 passed, 57 xfailed; new module
`tests/test_url_special_authority.py` fully covered.
Lint: `pre-commit` (ruff, flake8, mypy) passed.
Docs: `make doc-spelling` and `make -C docs doctest` passed.
Conformance: `python tools/conformance/compare.py --check` passes
against this tree.
</details>
@asvetlov
asvetlov deleted the drop-triple-slash-invalid-url branch September 29, 2026 10:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-3.14 Trigger automatic backporting to the 3.14 release branch by Patchback robot backport-3.15 Trigger automatic backporting to the 3.15 release branch by Patchback robot bot:chronographer:provided There is a change note present in this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant