repomatic.http module

Shared HTTP client for the API clients.

request() is the single implementation of the request loop: every datasource shares its timeout, its User-Agent, its truncated-body retry and the one FetchError every failure becomes. Caching policy stays with the callers (each client owns its cache namespace, TTL and serialization), while get_cached_json() shares the caching mechanics for the clients that store verbatim bodies.

Most responses are JSON (get_json()); get_text() serves the datasources that are text rather than an API payload (the astral-sh/setup-uv checksum table repomatic.deps.pins reads, the gitignore.io template repomatic.gitignore fetches), and get_bytes() the ones written back verbatim (a downloaded label-definition file). download() streams a large artifact to disk instead of reading it into memory.

repomatic.http.DEFAULT_TIMEOUT = 10

Socket timeout in seconds for every HTTP fetch repomatic makes.

A stalled connection must fail the operation, not hang it.

repomatic.http.USER_AGENT = 'repomatic/8.0.0.dev0'

User-Agent header sent with every fetch this module makes.

Identifying the client beats urllib’s anonymous default on every count: some registries throttle unidentified agents harder, and an operator reading an upstream access log can tell which release of this tool hit them.

repomatic.http.ERROR_BODY_EXCERPT = 400

Characters of an error response body a FetchError keeps.

Enough for the error object an API answers with, short of a whole HTML error page dumped into a log line.

repomatic.http.RETRYABLE_METHODS = frozenset({'GET', 'HEAD'})

Methods whose request is sent again after a truncated response body.

Both are safe to repeat. A truncated answer to a POST, PATCH or PUT says nothing about whether the server applied the change, so sending it again could apply it twice.

repomatic.http.DOWNLOAD_ATTEMPTS = 3

Attempts at one download() before it gives up.

Runner egress fails in ways that clear on the next connection: a TLS-intercepting proxy, a dropped connection, a short body. A retry after a pause keeps such a fault from ending a whole release job.

repomatic.http.DOWNLOAD_CHUNK_SIZE = 65536

Bytes download() reads from the response, writes and hashes per step.

repomatic.http.DOWNLOAD_TIMEOUT = 30

Socket timeout of a download(), in seconds.

A stall guard, not a transfer budget: urlopen applies it to each blocking socket operation, so a healthy multi-minute download is unaffected while a dead connection fails in seconds. Larger than DEFAULT_TIMEOUT, which is sized for small API responses.

exception repomatic.http.FetchError(message, *, status=None, body='')

Bases: RuntimeError

Raised when an HTTP request could not complete cleanly.

Wraps every failure mode of request(): HTTP 4xx/5xx, network error, timeout, malformed URL, truncated body (after its retries), and the decode errors of the get_* readers. Callers decide whether a failure is fatal (GitHub pagination, where a missing page corrupts the result) or a soft miss (PyPI/npm lookups, logged and treated as “no data”).

status

HTTP status of an error response, or None when no response came.

body

The start of the error response body, empty when there was none.

At most ERROR_BODY_EXCERPT characters, decoded leniently: an API error object fits, a whole error page does not.

class repomatic.http.Response(status, headers, body)

Bases: NamedTuple

A completed HTTP response.

Create new instance of Response(status, headers, body)

status: int

The HTTP status code.

headers: dict[str, str]

Response headers, keyed by lowercase name.

body: bytes

The raw response body.

repomatic.http.error_excerpt(error)

The start of an error response’s body, empty when it cannot be read.

Reading the body is a network read too, and fails the same ways as the request: the refusal still stands without its body. Shared with the clients that call urllib themselves, like the S3 half of repomatic.cloudflare.r2.

Parameters:

error (HTTPError) – The refusal urllib raised.

Return type:

str

Returns:

At most ERROR_BODY_EXCERPT characters of the body, decoded leniently.

repomatic.http.request(url, *, method='GET', data=None, headers=None, timeout=10)

Send one HTTP request and return the response.

A truncated body (IncompleteRead) is transient (a flaky connection or an interfering proxy), so a GET or HEAD request is sent again after each pause in _TRUNCATED_BODY_BACKOFF_SECONDS. Every other failure mode, and a truncated answer to any other method (see RETRYABLE_METHODS), fails straight away.

Parameters:

• url (str) – The URL to request.

• method (str) – The HTTP method.

• data (bytes | None) – The request body, already encoded. A JSON body also needs a Content-Type header in headers.

• headers (Mapping[str, str] | None) – Request headers, merged over the User-Agent default (caller wins on conflict).

• timeout (float) – Socket timeout in seconds. data goes out in one sendall call, which spends the timeout on the whole body: a large upload needs a longer one.

Return type:

Response

Returns:

The status, headers and body of the response.

Raises:

FetchError – On an HTTP error status, carrying the status and the start of the error body. Also on a network error, a timeout, a URL urllib cannot open, or a body still truncated after the last retry.

repomatic.http.download(url, dest, *, timeout=30, progress=None)

Stream url into dest and return the SHA-256 of the body.

The body goes to disk chunk by chunk, so a large binary never sits in memory. A body shorter than its Content-Length fails as truncated. Its digest would otherwise read as a checksum mismatch, which blames the pin or the artifact for a network fault.

A network failure, a truncated body and a server error (5xx) are tried again, up to DOWNLOAD_ATTEMPTS times, after a pause that grows with each attempt. A client error (4xx) fails at once: the same request gets the same answer. A failed attempt leaves no file behind.

Parameters:

• url (str) – The URL to download.

• dest (Path) – Where to write the body.

• timeout (float) – Socket timeout in seconds.

• progress (Callable[[int], AbstractContextManager[Callable[[int], object]]] | None) – Feedback for an interactive terminal. Called with the advertised length (0 when the server sends none) before the first chunk, it returns a context manager. The context manager yields the callable that receives the size of each chunk.

Return type:

str

Returns:

Lowercase hex SHA-256 digest of the body.

Raises:

FetchError – On a client error, or when the last attempt fails.

repomatic.http.get_json(url, *, headers=None, timeout=10)

GET url and parse the body as JSON, retrying on truncation.

Parameters:

• url (str) – The URL to fetch.

• headers (Mapping[str, str] | None) – Extra request headers, merged over the JSON Accept default (caller wins on conflict).

• timeout (float) – Socket timeout in seconds.

Return type:

Any

Returns:

The decoded JSON value.

Raises:

FetchError – On any failure (see the class docstring).

repomatic.http.get_text(url, *, headers=None, timeout=10)

GET url and decode the body as UTF-8 text.

An undecodable body is a failed fetch rather than something to paper over with replacement characters: every caller parses what it reads, and parsing mojibake yields a wrong answer instead of a missing one.

Parameters:

• url (str) – The URL to fetch.

• headers (Mapping[str, str] | None) – Extra request headers, merged over the plain-text Accept default (caller wins on conflict).

• timeout (float) – Socket timeout in seconds.

Return type:

str

Returns:

The decoded response body.

Raises:

FetchError – On any failure (see the class docstring), including a body that is not valid UTF-8.

repomatic.http.get_bytes(url, *, headers=None, timeout=10)

GET url and return its raw body.

For payloads written back verbatim (a downloaded label-definition file), where decoding would only risk corrupting bytes nothing here reads.

Parameters:

• url (str) – The URL to fetch.

• headers (Mapping[str, str] | None) – Extra request headers, merged over the wildcard Accept default (caller wins on conflict).

• timeout (float) – Socket timeout in seconds.

Return type:

bytes

Returns:

The raw response body.

Raises:

FetchError – On any failure (see the class docstring).

repomatic.http.get_json_soft(url, log_label, *, headers=None)

GET url as JSON, logging any failure as a soft miss.

Parameters:

• url (str) – The URL to fetch.

• log_label (str) – Human-readable label for the debug log on failure.

• headers (Mapping[str, str] | None) – Extra request headers, merged over the JSON Accept default (caller wins on conflict).

Return type:

Any | None

Returns:

The decoded JSON value, or None on any failure (HTTP error, network error, timeout, JSON parse error).

repomatic.http.get_cached_json(namespace, key, url, *, ttl, log_label, force_refresh=False)

GET url as JSON through the raw-response cache.

A fresh cached body under namespace/key short-circuits the network; otherwise the response is fetched, cached verbatim (when ttl is positive), and returned parsed. The caller keeps the caching policy: it picks the namespace, the cache key, and the TTL.

Note

force_refresh skips the cache read but keeps the write, which is what separates it from ttl=0: the latter also skips the store, so a caller using it to bypass a stale entry would leave that entry in place for the next reader. A forced refresh replaces it.

Parameters:

• namespace (str) – Cache namespace (like "pypi" or "npm").

• key (str) – Cache key within the namespace, usually the package name.

• url (str) – The URL to fetch on a cache miss.

• ttl (int) – Freshness TTL in seconds; 0 disables caching.

• log_label (str) – Human-readable label for the debug log on failure.

• force_refresh (bool) – Ignore any cached body and re-fetch, then store the fresh response.

Return type:

Any | None

Returns:

The parsed JSON value, or None on any fetch failure.