{{ message }}
Repository navigation
✨ Add native OpenTelemetry support - #16403
Merged
Merged
Conversation
Contributor
📝 Docs previewLast commit daba280 at: https://db5507dc.fastapitiangolo.pages.dev Modified Pages |
tiangolo
marked this pull request as ready for review
September 29, 2026 16:27
This was referenced Sep 30, 2026
8 tasks
This was referenced Oct 2, 2026
4 tasks done
yaoyu-33
pushed a commit
to yaoyu-33/Gym
that referenced
this pull request
Oct 8, 2026
NVIDIA-NeMo#4216) ## Problem FastAPI 0.142.0 to 0.142.4 ([release notes](https://github.com/fastapi/fastapi/releases/tag/0.142.0), [fastapi/fastapi#16403](fastapi/fastapi#16403)) configure OpenTelemetry export by themselves at ASGI startup whenever `OTEL_EXPORTER_OTLP_ENDPOINT` is set. Gym sets that variable for every server process so that nemo-lens finds the collector, and nemo-lens has already registered providers with their own exporters. FastAPI attaches a second `BatchSpanProcessor` and a second `PeriodicExportingMetricReader` to those providers. Gym does not pin FastAPI, so every job environment resolved since 2026-09-29 picked this up. Observed in every gym-native run since then: - every span is exported twice: each trace appears twice in Tempo and the collector's span-metrics counts are doubled; - every metric is exported by two readers with separate aggregation state, so the cumulative `_sum` of the Gym histograms carries last-digit dips that Prometheus treats as counter resets; `increase()`/`rate()` on a sum then adds the whole running value (mean rollout duration showed hours instead of minutes, 60 of 60 runs affected). ## Change FastAPI 0.143.0 ([fastapi/fastapi#16476](fastapi/fastapi#16476), from [discussion #16473](fastapi/fastapi#16473)) makes the automatic exporter setup opt-in, so only the 0.142 series double-exports. This PR excludes that series in `pyproject.toml` (`fastapi!=0.142.*`) and refreshes `uv.lock`; the locked version stays 0.135.2. An earlier revision wrapped the six `FastAPI()` sites in a factory that switched the native telemetry off; the exclusion replaces it because it also covers the bare `FastAPI()` apps outside the server bases and needs nothing from future servers. Environments already built with FastAPI 0.142 keep exporting twice until they are reinstalled. ## Verification A gym-native run with the factory revision as the job-side Gym (terminal-bench miniswe, 8 rollouts, 1 node) showed one `gym.rollout`/`gym.verify` span per rollout, one metric datapoint per 10 s with consistent sums, and `resets(gym_rollout_duration_ms_milliseconds_sum[1d]) = 0`, against 2 spans per rollout and up to 684 resets per run on `main`. The exclusion produces the same single export by resolving FastAPI to 0.143.0 instead of switching the feature off. Signed-off-by: Ingo Guehring <iguehring@nvidia.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This was referenced Oct 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

✨ Add native OpenTelemetry support
This is designed from the beginning considering OpenTelemetry's contrib current implementation, Logfire, and Sentry.
Several features and data extraction parts in this implementation are there so that these tools can get the data they need, but would currently acquire by monkeypatching FastAPI.
Ideally, no third-party package should need/want to patch FastAPI internals (I'll change and refactor a lot of those internals soon 😅).
2026-09-29 update: checked with Sentry and Logfire, this doesn't block or limit their work, we can go with this. A follow up is probably worth with OpenTelemetry contrib to make use the new built-in integration.
AI Disclaimer
Made with the help of AI, using Codex with
gpt-6-astra, manually reviewed.