💥 Require opt-in for automatic OpenTelemetry exporter setup by tiangolo · Pull Request #16476 · fastapi/fastapi · GitHub
Skip to content

💥 Require opt-in for automatic OpenTelemetry exporter setup - #16476

Merged
tiangolo merged 1 commit into
masterfrom
codex/otel-auto-configure-opt-in
Oct 8, 2026
Merged

tiangolo merged 1 commit into
masterfrom
codex/otel-auto-configure-opt-in

Conversation

@tiangolo

@tiangolo tiangolo commented Oct 8, 2026 •

Copy link
Copy Markdown
Member

💥 Require opt-in for automatic OpenTelemetry exporter setup

In short, to enable autoconfiguration of exporters, set the env var FASTAPI_OTEL_AUTO_CONFIGURE=true.

AI Disclaimer

Made with the help of AI, using Codex with gpt-6-astra, manually reviewed.

@codspeed

codspeed Bot commented Oct 8, 2026

Copy link
Copy Markdown

@github-actions

github-actions Bot commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

@tiangolo
tiangolo marked this pull request as ready for review October 8, 2026 11:51
@tiangolo
tiangolo merged commit af889e4 into master Oct 8, 2026
53 checks passed
@tiangolo
tiangolo deleted the codex/otel-auto-configure-opt-in branch October 8, 2026 11:51
iguehring-nvidia added a commit to NVIDIA-NeMo/Gym that referenced this pull request Oct 8, 2026
FastAPI 0.142.0 to 0.142.4 auto-configure a second OTLP span processor and
metric reader on nemo-lens's providers whenever OTEL_EXPORTER_OTLP_ENDPOINT
is set, so every span and metric left Gym servers twice. 0.143.0 makes that
opt-in (fastapi/fastapi#16476). Excluding the affected range is enough;
environments already built with 0.142 need a reinstall.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Ingo Guehring <iguehring@nvidia.com>
yaoyu-33 pushed a commit to yaoyu-33/Gym that referenced this pull request Oct 8, 2026
NVIDIA-NeMo#4216)

## Problem

FastAPI 0.142.0 to 0.142.4 ([release
notes](https://github.com/fastapi/fastapi/releases/tag/0.142.0),
[fastapi/fastapi#16403](fastapi/fastapi#16403))
configure OpenTelemetry export by themselves at ASGI startup whenever
`OTEL_EXPORTER_OTLP_ENDPOINT` is set. Gym sets that variable for every
server process so that nemo-lens finds the collector, and nemo-lens has
already registered providers with their own exporters. FastAPI attaches
a second `BatchSpanProcessor` and a second
`PeriodicExportingMetricReader` to those providers. Gym does not pin
FastAPI, so every job environment resolved since 2026-09-29 picked this
up.

Observed in every gym-native run since then:

- every span is exported twice: each trace appears twice in Tempo and
the collector's span-metrics counts are doubled;
- every metric is exported by two readers with separate aggregation
state, so the cumulative `_sum` of the Gym histograms carries last-digit
dips that Prometheus treats as counter resets; `increase()`/`rate()` on
a sum then adds the whole running value (mean rollout duration showed
hours instead of minutes, 60 of 60 runs affected).

## Change

FastAPI 0.143.0
([fastapi/fastapi#16476](fastapi/fastapi#16476),
from [discussion
#16473](fastapi/fastapi#16473)) makes the
automatic exporter setup opt-in, so only the 0.142 series
double-exports. This PR excludes that series in `pyproject.toml`
(`fastapi!=0.142.*`) and refreshes `uv.lock`; the locked version stays
0.135.2. An earlier revision wrapped the six `FastAPI()` sites in a
factory that switched the native telemetry off; the exclusion replaces
it because it also covers the bare `FastAPI()` apps outside the server
bases and needs nothing from future servers.

Environments already built with FastAPI 0.142 keep exporting twice until
they are reinstalled.

## Verification

A gym-native run with the factory revision as the job-side Gym
(terminal-bench miniswe, 8 rollouts, 1 node) showed one
`gym.rollout`/`gym.verify` span per rollout, one metric datapoint per 10
s with consistent sums, and
`resets(gym_rollout_duration_ms_milliseconds_sum[1d]) = 0`, against 2
spans per rollout and up to 684 resets per run on `main`. The exclusion
produces the same single export by resolving FastAPI to 0.143.0 instead
of switching the feature off.

Signed-off-by: Ingo Guehring <iguehring@nvidia.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant