oxidizer/crates/fetch/README.md at main · microsoft/oxidizer · GitHub
Skip to content

README.md

Latest commit

 

History

History
810 lines (608 loc) · 30.9 KB

README.md

File metadata and controls

810 lines (608 loc) · 30.9 KB
Raw
Fetch Logo

Fetch

crate.io docs.rs MSRV CI Coverage License This crate was developed as part of the Oxidizer project

A fast, safe HTTP client that works across runtimes and transports.

let response = client.get("https://example.com").fetch().await?;
assert!(response.status().is_success());

HttpClient supports different async runtimes and transports. It includes secure defaults, resilience, telemetry, and test utilities.

Why a new HTTP client?

fetch includes the features that services commonly need:

  • Secure TLS validation, retries, circuit breaking, hedging, and OpenTelemetry telemetry.
  • Fake responses through the test-util feature.
  • A request pipeline that supports custom logging, metrics, retries, and other handlers.
  • Memory pooling and zero-copy operations for large responses.

fetch does not require one runtime, I/O implementation, or HTTP transport. A transport handler performs I/O at the end of the request pipeline. Resilience, telemetry, routing, logging, and retries run above that handler.

  • Tokio works with the bundled transport. A custom transport can use another runtime and I/O implementation.
  • A transport implements RequestHandler. It can use the bundled hyper transport, a custom client, or another client such as reqwest.

A library can use fetch while the application selects the runtime and transport. See the custom module and custom::create_builder for an example.

How does it compare to reqwest?

Both fetch and reqwest use hyper by default. fetch also includes resilience and supports custom runtimes and transports. A custom transport can use reqwest while the rest of the fetch pipeline stays unchanged.

Feature fetch reqwest
Runtime support Tokio and custom runtimes Tokio on native targets; browser implementation on wasm32
Custom transport or I/O Built in Not supported
TLS/HTTPS rustls or native-tls rustls or native-tls
Resilience Retries and circuit breaking are enabled by default; hedging is built in and optional Built-in retries; circuit breaking and hedging require external middleware
JSON support Built in Built in
Testing tools Built in Requires custom or external tools
OpenTelemetry metrics Built in Requires custom integration
Structured request logging Built-in tracing events Requires custom middleware
Advanced HTTP client features 1 Not supported yet 2 Available through optional features
Request pipeline Built in Requires custom or external tools
Zero-copy buffers Built in Partial support through Bytes
Linux support Full support Full support

Note: fetch uses methods such as get(), post(), and fetch(), which are similar to reqwest methods.

Getting Started

The bundled transport runs on Tokio. Use a custom transport for another runtime. See the custom module.

use fetch::{HttpClient, HttpError};

#[tokio::main]
async fn main() -> Result<(), HttpError> {
    // Create a client with the bundled Tokio transport.
    let client: HttpClient = HttpClient::new_tokio();

    // Return an error for a `4xx` or `5xx` status, then read the body as text.
    let body: String = client.get("https://example.com").fetch_text_body().await?;

    println!("response: {body}");

    Ok(())
}

Note: Use HttpClient::builder_tokio to add handlers or change timeouts and other configuration.

Making Requests

Use methods such as HttpClient::get and HttpClient::post for common requests. Each method returns a builder for request-specific configuration.

GET Requests

// Send a GET request.
let response: HttpResponse = client.get("https://www.example.com").fetch().await?;

POST Requests

// Send a POST request with a text body.
let response = client
    .post("https://httpbin.org/post")
    .text("the exact body that is sent")
    .fetch()
    .await?;

Other request methods

The client has methods such as HttpClient::put and HttpClient::delete. Use HttpClient::request for another HTTP method:

// Send a PATCH request.
let response = client
    .request(Method::PATCH, "https://api.example.com/items/42")
    .fetch()
    .await?;

A request can include headers, an HTTP version, and a body:

let response = client
    .post("https://api.example.com/upload")
    // Add HTTP headers.
    .header(header::AUTHORIZATION, "Bearer token123")
    .header(header::CONTENT_TYPE, "application/json")
    // Select the HTTP version.
    .version(Version::HTTP_2)
    .text("{\"name\": \"document.pdf\"}")
    .fetch()
    .await?;

These methods return HttpRequestBuilder, which configures and sends the request.

Reuse a base URI

Use HttpClientBuilder::base_uri to set one BaseUri for all requests.

The client base URI overrides a base URI in a request.

let client = builder
    .base_uri(BaseUri::from_static("https://example.com/api/v1/")) // The trailing slash is required.
    .build();

// This request uses `https://example.com/api/v1/foo/bar`.
let response = client.get("/foo/bar").fetch().await?;

Handling Responses

HttpRequestBuilder::fetch returns an HttpResponse with the body, status code, headers, and extensions. HttpResponse is a type alias for Response<HttpBody>.

A response supports these operations:

// Send a GET request.
let mut response: HttpResponse = client.get("https://www.example.com").fetch().await?;

// Return an error if the status is not successful.
response = response.ensure_success()?;

// Read the headers.
println!("Headers: {}", response.headers().len());

// Consume the response and return its body.
let body: HttpBody = response.into_body();

// Read the body as text.
let text: String = body.into_text().await?;

println!("Response body: {}", text);

Specialized Fetch Methods

These methods fetch a response and convert its body:

Each method returns Response<T>. The response retains its status, headers, and extensions.

// Return the response with a text body.
let response = client
    .get("https://api.example.com/users")
    .fetch_text()
    .await?;

// Read response metadata before the body.
println!("Status: {}", response.status());
println!("Content-Type: {:?}", response.headers().get("content-type"));

// Return an error for an unsuccessful status, then extract the body.
let text: String = response.ensure_success()?.into_body();

Body-Only Shortcuts

The _body variants call ensure_success, discard the response metadata, and return the converted body.

// Fetch the response, require a successful status, and return the text body.
let text: String = client
    .get("https://api.example.com/users")
    .fetch_text_body()
    .await?;
println!("body: {text}");

URL Handling

The HTTP client uses templated_uri for URLs. Uri supports dynamic path segments and query parameters through templates.

Templates follow RFC 6570 level 3. Uri and string types also support URLs without typed templates. In that case, the full PathAndQuery string becomes the template.

handlers::Logging records the URL template as url.path.template.

This example creates a Uri with a templated path:

use templated_uri::{BaseUri, EscapedString, PathAndQueryTemplate, Uri, templated};

#[templated(template = "/users/{user_id}/", unredacted)]
#[derive(Clone)]
struct UserPath {
    user_id: EscapedString, // EscapedString escapes the value for use in URIs.
}

let user_path = UserPath {
    user_id: EscapedString::from_static("12345"),
};

client
    .get(
        Uri::default()
            .with_base(BaseUri::from_static("https://api.example.com"))
            .with_path_and_query(user_path),
    )
    .fetch_text()
    .await?;

Classification in URLs

templated_uri uses data_privacy to classify URL paths and queries. The classified attribute lets a PathAndQueryTemplate use classified data_privacy types.

use data_privacy::Sensitive;
use templated_uri::{EscapedString, PathAndQueryTemplate, templated};

#[templated(template = "/{org_id}/user/{user_id}")]
struct UserPath {
    #[unredacted]
    org_id: u32,
    user_id: Sensitive<EscapedString>,
}

JSON Support

The json feature supports these operations:

Owned JSON with fetch_json

Use fetch_json for owned data. It deserializes JSON directly into the target type:

// Define a type with owned data.
#[derive(Serialize, Deserialize)]
struct Person {
    id: u32,
    name: String,
}

let person = Person {
    id: 1,
    name: "Alice Johnson".to_owned(),
};

// Send and receive owned JSON.
let response: Response<Person> = client
    .put("https://api.company.com/people")
    .json(&person)
    .fetch_json::<Person>()
    .await?;

// Read the response metadata.
let response = response.ensure_success()?;

// Extract the deserialized value.
let person: Person = response.into_body();

println!("Person retrieved, name: {}", person.name);

fetch_json_body fetches the response, requires a successful status, and returns the owned value.

Zero-Copy JSON with fetch_json_ref

Pair fetch_json_ref with borrowed string fields such as Cow<'a, str>. Json<T> borrows from the response buffer. Prefer Cow<'a, str> over &'a str. Cow uses an owned value when an escaped JSON string cannot be borrowed.

// Define a type that can borrow string data.
#[derive(Serialize, Deserialize)]
struct Person<'a> {
    id: u32,
    #[serde(borrow)]
    name: Cow<'a, str>,
}

let person = Person {
    id: 1,
    name: "Alice Johnson".into(),
};

// Send and receive zero-copy JSON.
let response: Response<Json<Person>> = client
    .put("https://api.company.com/people")
    .json(&person)
    .fetch_json_ref::<Person>()
    .await?;

// Read the response metadata.
let response = response.ensure_success()?;

// Extract the JSON wrapper.
let mut json_body: Json<Person> = response.into_body();

// Parse the JSON data using zero-copy deserialization.
// The parsed Person borrows string data from the underlying buffer.
let person: Person = json_body.read()?;

println!("Person retrieved, name: {}", person.name);

Borrowed fields reduce heap allocations and copies. The parsed value cannot outlive the response buffer.

Request Pipeline

Each request passes through a sequence of RequestHandler implementations.

A handler can:

  • Modify the request before the next handler receives it.
  • Return a response without calling the next handler, for example to serve a cached response.
  • Process the response from the next handler.
  • Add logging, metrics, or other shared behavior.

The transport handler is the final RequestHandler. It performs I/O and returns a response. fetch supplies the resilience, telemetry, routing, and logging layers above it. The transport can use the bundled hyper implementation, custom runtime and I/O code, or another client such as reqwest. See custom and custom::create_builder for a custom transport.

Built-in Pipeline Types

The client supports three pipeline types:

Standard Pipeline

The standard pipeline is the default. The outermost handler processes the request first and the response last.

See StandardRequestPipeline and HttpClientBuilder::standard_pipeline.

Custom Pipeline

A custom pipeline contains the handlers that its factory returns. See HttpClientBuilder::custom_pipeline.

Minimal Pipeline

The minimal pipeline contains only Dispatch. Dispatch passes each request to the transport.

// Create a client with only the dispatch handler.
let minimal_client = builder.minimal_pipeline().build();

// Wrap the client with custom processing.
let wrapped_client = MyHttpWrapper::new(minimal_client);

Use this pipeline when another component supplies all middleware.

Creating Custom Handlers

See RequestHandler to implement request or response processing.

Testing with the HTTP Client

Enable test-util to return fake responses without network I/O.

A fake client can:

  • Return different HTTP responses.
  • Exercise retry and error handling.
  • Keep tests deterministic by avoiding network calls.
  • Reproduce cases that are difficult to produce with a service.

HttpClient::new_fake creates a client from predefined responses or a FakeHandler:

// Always return a 200 response.
let client = HttpClient::new_fake(StatusCode::OK);

// Return a sequence of responses without bodies.
let client = HttpClient::new_fake(vec![StatusCode::OK, StatusCode::INTERNAL_SERVER_ERROR]);

// Create a fake response.
let response = HttpResponseBuilder::new_fake()
    .status(StatusCode::INTERNAL_SERVER_ERROR)
    .text("fake text")
    .build()
    .expect("always succeeds");

// Always return the same response.
let client = HttpClient::new_fake(response);

// Use a synchronous or asynchronous custom handler.
let fake_handler = FakeHandler::from_fn(|req| {
    println!("Fake handler called for request, url: {}", req.uri());

    HttpResponseBuilder::new_fake()
        .status(StatusCode::INTERNAL_SERVER_ERROR)
        .text("fake text")
        .build()
});

// Create a client that uses the custom handler.
let client = HttpClient::new_fake(fake_handler);

Memory management with BytesView

fetch uses BytesView for request and response bodies. It provides:

  • Memory pooling.
  • Reduced copying.
  • A single view over multiple chunks.
  • Zero-copy operations when the data layout permits them.
  • Compatibility with the bytes crate.

BytesView implements Buf. Use BytesBuf, which implements BufMut, to build a mutable body.

// Require a successful status and return the body as BytesView.
let mut body_bytes = client.get("https://example.com").fetch_bytes_body().await?;

// Use methods from bytes::Buf.
let length = body_bytes.remaining();

// Copy part of the body when required.
let mut buffer = vec![0; 10.min(length)];
if !body_bytes.is_empty() {
    body_bytes.copy_to_slice(&mut buffer);
}

This representation reduces memory use and copies for large bodies.

Performance

Reuse clients, parse repeated URIs once, and keep binary data in BytesView:

// Create one client and reuse it.
let client = HttpClient::new_tokio();

// Parse repeated URIs once.
let users_uri: Uri = "https://api.example.com/users".parse()?;
let items_uri: Uri = "https://api.example.com/items".parse()?;

// Keep binary data in BytesView.
let bytes = client.get(users_uri.clone()).fetch_bytes_body().await?;
process_binary_data(bytes);

Creating an HttpClient sets up connection pools and security configuration. Create one client and share it across tasks. Parse repeated Uri values once. Converting BytesView to String allocates and copies data, so keep binary or large bodies in BytesView when possible.

Integration with the HTTP Ecosystem

fetch uses types from http and extensions from http_extensions. It re-exports common types from both crates.

Resilience

The seatbelt crate provides the built-in resilience layers.

Client handlers and seatbelt middleware implement Service, so the pipeline can combine them without an adapter.

The available patterns include:

  • Retries with configurable backoff.
  • Timeouts.
  • Circuit breakers.

The standard pipeline configures these patterns by default.

TLS Support

The client supports two TLS backends:

  • rustls is the default. It uses rustls with the aws-lc-rs crypto provider. The tls feature enables it.
  • native-tls uses SChannel on Windows, Security Framework on macOS, and OpenSSL on Linux. It is selected when native-tls is enabled without rustls.

With rustls, rustls-platform-verifier uses the platform trust store. It:

  • Validates certificates against the operating system root CAs.
  • Validates host names and certificate expiration.
  • Applies TLS security policies.

The default client applies the selected TLS backend:

use fetch::HttpClient;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = HttpClient::new_tokio();

    // Send an HTTPS request.
    let response = client.get("https://www.example.com").fetch_text().await?;

    Ok(())
}

Enable tls and tokio to use the default rustls transport:

fetch = { version = "*", features = ["tls", "tokio"] }

Enable native-tls and tokio to use native TLS:

fetch = { version = "*", features = ["native-tls", "tokio"] }

If both backends are enabled, use TlsOptions::builder_rustls() or TlsOptions::builder_native_tls() to select a backend for each client.

Automatic Response Decompression

Automatic response decompression is off by default. Enable the required compression-* features, or enable compression-all. Then select the formats through DecompressionOptions:

let client = builder.decompression(&[DecompressionFormat::Gzip]).build();

Requests list the formats in Accept-Encoding, with the most preferred format first. Before the client returns a response that uses a selected format, it removes Content-Encoding and Content-Length. It decodes the body lazily as the caller reads it. If the response uses a format that was not selected, the client returns it without changes.

DecompressionFormat::ALL asks for everything the build can decompress.

A malformed body fails when code reads it, not when the response arrives. This failure does not trigger a retry. The client uses the http_compression defaults. The decompressed output limit is 64 MiB. Each decompression stage accepts at most 1,024 concatenated streams. fetch does not override these defaults.

DecompressionOptions sets limits for decompressed output and compressed streams. Pass None to max_output_len to remove the output limit. The limits also apply to streamed bodies. Exceeding a limit returns the compression_limit_exceeded error label.

response_body_options separately limits memory used to buffer a body. Its default limit is 2 GiB.

For server-side compression and decompression, use the http_compression crate directly.

Features

Enable optional features in Cargo.toml:

[dependencies]
fetch = { version = "*", features = ["json", "tokio", "tls", "compression-gzip"] }

  • tokio enables the Tokio runtime integration. Combine it with rustls, native-tls, or the tls alias to provide HttpClient::builder_tokio, HttpClient::builder_tokio_with_options, and related APIs. The bundled transport requires a TLS feature even when the client permits HTTP.

  • json enables JSON serialization and deserialization. It provides methods such as HttpRequestBuilder::json and HttpRequestBuilder::fetch_json.

  • tls enables HTTPS through rustls and the aws-lc-rs crypto provider. It is an alias for rustls.

  • rustls enables the default TLS backend with aws-lc-rs.

  • native-tls enables SChannel on Windows, Security Framework on macOS, and OpenSSL on Linux. If both TLS backends are enabled, rustls is the default. Use TlsOptions::builder_native_tls() to select native TLS.

  • test-util provides fake responses and client behavior for tests without network I/O.

The compression-* features select formats for automatic response decompression. They are off by default. A build without them links no compression implementation. Enabling a feature makes its format available. The client does not decompress a response until DecompressionOptions::formats selects it and the client applies the options with decompression.

Note: Enable tokio and tls for HTTPS with the bundled transport. Enable json for JSON APIs.

This crate was developed as part of The Oxidizer Project. Browse this crate's source code.

Footnotes

  1. Advanced HTTP client features include file uploads, cookies, proxies, and redirects. ↩

  2. Future versions can add the missing cookies, redirects, and forms features. ↩