Summary

Closes #457

The gateway had only global concurrency bounds, so one caller could use up the upstream's capacity and shared web-search or code-interpreter quotas for everyone. This PR adds an opt-in, per-caller request rate limit. It is off by default: no middleware is installed and no headers change.

Configuration (CLI > environment > [server] file setting, following max_request_body_size_bytes):

• A burst without a rate, a zero value, or a burst over one hour of requests fails at startup, and the error names the setting.
• An explicit CLI value outranks a malformed inherited environment value.

Caller key (first match):

1. OIDC issuer and subject;
2. a keyed hash of the bearer token;
3. a keyed hash of x-api-key;
4. one shared anonymous bucket.

Raw credentials are never stored. A WebSocket is keyed by its upgrade request's own credentials, never by the server's OPENAI_API_KEY fallback.

Algorithm: GCRA (equivalent to a token bucket). Each caller holds one theoretical arrival time in a std::sync::Mutex<HashMap>, using tokio::time::Instant so tests can pause time. No lock is held across an await. Memory is bounded at 100,000 callers: idle callers are pruned first, and new callers share an overflow bucket while every tracked caller is active, with a warn at most once a minute. No new dependencies.

HTTP:

• The middleware is a route_layer on the protected routes, layered before OIDC so that it runs after authentication and sees the principal. /health, /ready and Swagger are exempt.
• A limited request never reaches the upstream. It gets 429 in OpenAI format (rate_limit_error / rate_limit_exceeded), or in Anthropic format with request-id on /v1/messages and count_tokens.
• Headers: retry-after, retry-after-ms, x-ratelimit-limit-requests, x-ratelimit-remaining-requests: 0, and x-ratelimit-reset-requests. The last is the time until the full burst returns, formatted like 1.5s or 2m3s.

WebSocket:

• The upgrade is charged by the HTTP middleware, which also carries the caller's bucket into the connection.
• Each client response.create is charged before any lane or byte capacity is reserved. A limited create gets an error frame with status: 429, and the connection stays open.
• response.steer, response.inject and gateway-started continuations are not charged. The check sits where client messages arrive, not in WsMultiplexer::schedule, which also runs automatic continuations.

Structure:

• rate_limit.rs holds the limiter, middleware, 429 builder and retry formatting.
• server_settings.rs resolves the [server] table. Its first commit is a pure move of the existing body-size resolution out of main.rs, which was at its 644-line exception (now 616).
• ServerConfig gains rate_limit, so AppState and its 16 test literals are unchanged.
• auth.rs is at its line baseline. Its only change is visibility (AuthErrorFormat and for_path, reused for the 429 format), and bearer_token in auth/api_key.rs is also now crate-visible.

Scope:

• Per process: with N replicas, a caller can send up to N times the rate. The API, container and Kubernetes docs say so.
• Not in this PR:
  • distributed or token-based quotas (Enforce user and tenant ownership for persisted state #107, Add complete multi-round Messages usage accounting #315);
  • per-IP keying, which needs trusted-proxy configuration;
  • agentic launcher flags. The launcher's child process inherits the environment variables, and the configuration file is shared.

Conflicts: a trial merge with #436, #440 and #420 is clean.

Test Plan

Unit tests:

• rate_limit::tests (paused time):
  • burst, then one request per interval, including exact retry_after and reset;
  • steady state;
  • independent callers;
  • pruning before overflow;
  • credentials kept only as a hash (no token text in Debug; bearer and x-api-key share a key; Basic is anonymous);
  • retry rounding and x-ratelimit-reset-requests formatting;
  • the burst bound.
• server_settings::tests:
  • off by default;
  • file, then environment, then CLI, including a CLI value over a malformed environment value;
  • the burst defaulting to the rate;
  • five invalid inputs, each named in its error;
  • flag parsing;
  • the generated [server] table.
• config_file tests: the new keys parse, 0 is rejected, and an unset burst is not written.

tests/rate_limit_test.rs (one request per minute, so the outcomes are deterministic):

• a_caller_over_its_limit_gets_an_openai_429_with_retry_headers: exact body and all five headers; the limited request never reaches the upstream; another bearer is unaffected.
• callers_without_credentials_share_one_bucket
• health_endpoints_are_never_limited
• messages_routes_answer_in_the_anthropic_format_and_key_by_api_key: exact body; the request-id header matches the body.
• rate_limiting_is_off_by_default: 10 requests and no rate-limit headers.
• websocket_creates_are_charged_and_a_limited_create_keeps_the_connection: the upgrade and two creates use a burst of 3; the next create gets the exact 429 frame; then response.steer is answered on the same connection.
• http_and_websocket_share_a_bucket_and_the_upgrade_is_charged: a limited upgrade gets HTTP 429 before switching protocols.
• a_websocket_without_credentials_is_not_charged_to_the_server_key

oidc_auth_test::rate_limits_key_oidc_callers_by_subject_not_token: two tokens for one subject share a bucket, and another subject has its own.

Negative controls (each applied to the finished code):

Binary smoke test at 1 request per minute:

• --help lists both flags.
• AGENTIC_RATE_LIMIT_BURST without a rate fails at startup: AGENTIC_RATE_LIMIT_BURST requires AGENTIC_RATE_LIMIT_REQUESTS_PER_MINUTE.
• The first /v1/models request reaches the upstream; the second gets 429 with all headers; /health still returns 200.

Gate:

• cargo fmt -- --check
• cargo clippy --workspace --all-targets -- -D warnings
• Feature clippy (--features embedded-code-interpreter) for both crates
• The Rust file-size ratchet
• pre-commit run --all-files
• mkdocs build --strict
• cargo test --workspace --no-fail-fast: 2530 passed, 1 failed. The failure is tool::web_fetch::http::tests::the_deadline_bounds_a_slow_origin, which also fails on unmodified main when the whole lib suite runs in parallel.