{{ message }}
Repository navigation
💥 Require opt-in for automatic OpenTelemetry exporter setup - #16476
Merged
Merged
Conversation
Contributor
tiangolo
marked this pull request as ready for review
October 8, 2026 11:51
iguehring-nvidia
added a commit
to NVIDIA-NeMo/Gym
that referenced
this pull request
Oct 8, 2026
FastAPI 0.142.0 to 0.142.4 auto-configure a second OTLP span processor and metric reader on nemo-lens's providers whenever OTEL_EXPORTER_OTLP_ENDPOINT is set, so every span and metric left Gym servers twice. 0.143.0 makes that opt-in (fastapi/fastapi#16476). Excluding the affected range is enough; environments already built with 0.142 need a reinstall. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Ingo Guehring <iguehring@nvidia.com>
yaoyu-33
pushed a commit
to yaoyu-33/Gym
that referenced
this pull request
Oct 8, 2026
NVIDIA-NeMo#4216) ## Problem FastAPI 0.142.0 to 0.142.4 ([release notes](https://github.com/fastapi/fastapi/releases/tag/0.142.0), [fastapi/fastapi#16403](fastapi/fastapi#16403)) configure OpenTelemetry export by themselves at ASGI startup whenever `OTEL_EXPORTER_OTLP_ENDPOINT` is set. Gym sets that variable for every server process so that nemo-lens finds the collector, and nemo-lens has already registered providers with their own exporters. FastAPI attaches a second `BatchSpanProcessor` and a second `PeriodicExportingMetricReader` to those providers. Gym does not pin FastAPI, so every job environment resolved since 2026-09-29 picked this up. Observed in every gym-native run since then: - every span is exported twice: each trace appears twice in Tempo and the collector's span-metrics counts are doubled; - every metric is exported by two readers with separate aggregation state, so the cumulative `_sum` of the Gym histograms carries last-digit dips that Prometheus treats as counter resets; `increase()`/`rate()` on a sum then adds the whole running value (mean rollout duration showed hours instead of minutes, 60 of 60 runs affected). ## Change FastAPI 0.143.0 ([fastapi/fastapi#16476](fastapi/fastapi#16476), from [discussion #16473](fastapi/fastapi#16473)) makes the automatic exporter setup opt-in, so only the 0.142 series double-exports. This PR excludes that series in `pyproject.toml` (`fastapi!=0.142.*`) and refreshes `uv.lock`; the locked version stays 0.135.2. An earlier revision wrapped the six `FastAPI()` sites in a factory that switched the native telemetry off; the exclusion replaces it because it also covers the bare `FastAPI()` apps outside the server bases and needs nothing from future servers. Environments already built with FastAPI 0.142 keep exporting twice until they are reinstalled. ## Verification A gym-native run with the factory revision as the job-side Gym (terminal-bench miniswe, 8 rollouts, 1 node) showed one `gym.rollout`/`gym.verify` span per rollout, one metric datapoint per 10 s with consistent sums, and `resets(gym_rollout_duration_ms_milliseconds_sum[1d]) = 0`, against 2 spans per rollout and up to 684 resets per run on `main`. The exclusion produces the same single export by resolving FastAPI to 0.143.0 instead of switching the feature off. Signed-off-by: Ingo Guehring <iguehring@nvidia.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

💥 Require opt-in for automatic OpenTelemetry exporter setup
In short, to enable autoconfiguration of exporters, set the env var
FASTAPI_OTEL_AUTO_CONFIGURE=true.AI Disclaimer
Made with the help of AI, using Codex with
gpt-6-astra, manually reviewed.