Repository navigation
[Backport 7.84.x] [CONTINT-3239] Revert "[automated] Upgrade embedded Python patch version to 3.13.16" - #57907
Conversation
…ion to 3.13.16" (#57902) ### What does this PR do? Reverts #57594, taking the embedded Python back from 3.13.16 to 3.13.15. It also makes the weekly automated bump (`dda inv python-version.update`) skip 3.13.16, so the bot doesn't reopen the same upgrade next Tuesday. The bump was also backported to `7.84.x` (#57866) and `7.85.x` (#57867), so this PR carries the matching `backport/*` labels. No tag ships 3.13.16 yet (`7.84.2` and `7.85.0-rc.4` still have 3.13.15). The next RC or patch release from those branches would. ### Motivation Since #57594 merged (Oct 8, 19:09 UTC), the Agent intermittently fails to load Python checks at startup. The import raises: ``` unable to import module 'datadog_checks.kube_apiserver_metrics': Traceback (most recent call last): ... File ".../datadog_checks/base/checks/openmetrics/base_check.py", line 6, in <module> import requests File ".../requests/__init__.py", line 47, in <module> from .exceptions import RequestsDependencyWarning File "<frozen importlib._bootstrap>", line 1366, in _find_and_load File "<frozen importlib._bootstrap>", line 418, in __enter__ File "<frozen importlib._bootstrap>", line 346, in acquire _frozen_importlib._DeadlockError: deadlock detected by _ModuleLock('requests.exceptions') ``` The check is never retried, so it stays missing for the whole life of the Agent. In container e2e tests this shows up as `TestKindSuite/TestControlPlane` (`kube_apiserver_metrics`, `kube_scheduler`, `kube_controller_manager`) and Kind/EKS `TestPrometheus` (`openmetrics`) failing with "Should NOT be empty, but was []". In the dddev logs of e2e Agents we also see it for `etcd`, `coredns` and `argo_rollouts`, and on `prometheus_client.samples` as well as `requests.exceptions`. Evidence that the bump is the cause: - `TestKindSuite/TestControlPlane` failed 62 times on `main` in the last 3 days, against once in the 11 days before. `8308aba1` (the parent of the bump) passes on every job; `a8c258b6` and every commit after it fail on some jobs. - `_DeadlockError` never appears in the dddev logs of e2e Agents during the 14 days before pipeline 143514636 (`a8c258b6`). It has appeared in every pipeline since. - Only metrics from Python checks go missing. Go-check metrics from the same runs keep arriving. #### Root cause: an upstream CPython regression 3.13.16 includes the backport (python/cpython#155915) of python/cpython#130094, a fix for a different concurrent-import race. That change makes `importlib._bootstrap._find_and_load_unlocked(name)` re-import the parent package when the parent's spec is still `_initializing`, while still holding the module lock of `name`. Until 3.13.15, a thread holding a submodule's lock never waited for a parent that another thread was still executing; it just loaded the submodule. It now does, which closes a cycle: - thread A runs `requests/__init__.py` (it holds the `requests` lock) and reaches `from .exceptions import ...`, which needs the `requests.exceptions` lock; - thread B imports `requests.exceptions` directly (for example `from requests.exceptions import ...` in a check). It takes the `requests.exceptions` lock, sees `requests` is still initializing, and now waits for the `requests` lock. importlib detects the cycle and raises `_DeadlockError` in one of the two threads. Any package whose `__init__` imports its own submodules can hit this, and that's most of them (`requests`, `prometheus_client`, `datadog_checks.base` through `lazy_loader`, ...). The Agent isn't doing anything unusual. It imports checks from several goroutines (the autodiscovery discoverer workers and the collector scheduler), which worked fine up to 3.13.15. We aren't the only ones hit by this. Other projects saw the same `_DeadlockError` / hang after moving to 3.13.16, or to 3.14, which has had the change since 3.14.0: - gke-labs/kube-agents#2446 and gke-labs/kube-agents#2464: hang in a concurrent-import test, traced to the 3.13.16 backport. They compared `_find_and_load_unlocked` from 3.13.15 and 3.13.16 to confirm it. - dgunning/edgartools#1325: `_DeadlockError` when one thread imports `edgar.reference` while another imports `edgar`. - NousResearch/hermes-agent#123103, NousResearch/hermes-agent#129507 and NousResearch/hermes-agent#132146: `_DeadlockError` on `openai.resources.*` under 3.14. They worked around it with lazy package exports. - marimo-team/marimo#11095: concurrent import deadlocks with bokeh. Nobody has reported it upstream yet: python/cpython#130094 is closed and no newer issue mentions this regression. We plan to file one. ### Describe how you validated your changes - `dda inv invoke-unit-tests.run --tests=python_version` passes, including a new test for the excluded version. - The container e2e jobs (Kind and EKS suites) are re-run on this PR to confirm `TestControlPlane` and `TestPrometheus` pass again. ### Additional Notes - `EXCLUDED_PYTHON_VERSIONS` only skips 3.13.16. The bot will propose 3.13.17 when it comes out. Whoever reviews that PR has to check that it no longer carries the change, or add it to the list. - #57594 got merged without the container e2e tests ever running on it: its `new-e2e-containers*` jobs were all skipped. They are manual on PRs, and dynamic test selection (`--impacted`) maps nothing to a `deps/cpython` change, so it skips `TestKindSuite` even when the jobs are played (it had to be forced with `RUN_E2E_TESTS=on` on this PR). Changes to the embedded Python should run the e2e suites that load Python checks. A dedicated test that loads many Python checks at once would also catch this directly. - This problem will come back with Python 3.14. Before moving to 3.14 (or to a 3.13 patch release that keeps the change), we'll need an Agent-side mitigation, such as importing shared dependencies on one thread before checks load concurrently, or an upstream fix. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: regis.desgroppes <regis.desgroppes@datadoghq.com> (cherry picked from commit c7f76a5) ___ Co-authored-by: Lénaïc Huard <L3n41c@users.noreply.github.com>
Files inventory check summaryFile checks results against ancestor b1f537e1: Results for datadog-agent_7.84.2.git.8.7251fc7.pipeline.143686757-1_amd64.deb:No change detected Results for datadog-iot-agent_7.84.2.git.8.7251fc7.pipeline.143686757-1_amd64.deb:No change detected |
Regression DetectorRegression Detector ResultsMetrics dashboard Baseline: b1f537e Optimization Goals: ✅ No significant changes detected
|
| perf | experiment | goal | Δ mean % | Δ mean % CI | trials | links |
|---|---|---|---|---|---|---|
| ➖ | quality_gate_metrics_logs | memory utilization | +1.46 | [+1.23, +1.68] | 1 | Logs bounds checks dashboard |
| ➖ | dsd_uds_10mb_3k_timestamped_contexts_memory | memory utilization | +0.69 | [+0.48, +0.90] | 1 | Logs |
| ➖ | quality_gate_idle | memory utilization | +0.16 | [+0.12, +0.21] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_security_idle | memory utilization | +0.11 | [+0.08, +0.15] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_security_mean_fs_load | memory utilization | +0.05 | [+0.02, +0.09] | 1 | Logs bounds checks dashboard |
| ➖ | dsd_uds_10mb_3k_timestamped_contexts_cpu | % cpu utilization | -0.03 | [-0.27, +0.21] | 1 | Logs |
| ➖ | quality_gate_security_no_fs_load | memory utilization | -0.04 | [-0.11, +0.04] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_idle_all_features | memory utilization | -0.08 | [-0.12, -0.04] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_private_action_runner | memory utilization | -0.30 | [-0.43, -0.18] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_logs | % cpu utilization | -1.79 | [-2.64, -0.94] | 1 | Logs bounds checks dashboard |
Bounds Checks: ✅ Passed
Explanation
Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%
Performance changes are noted in the perf column of each table:
- ✅ = significantly better comparison variant performance
- ❌ = significantly worse comparison variant performance
- ➖ = no significant change in performance
A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".
For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:
-
Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
-
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
-
Its configuration does not mark it "erratic".
CI Pass/Fail Decision
✅ Passed. All Quality Gates passed.
- quality_gate_security_no_fs_load, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_no_fs_load, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_idle, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
- quality_gate_private_action_runner, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_mean_fs_load, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_mean_fs_load, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
08f5458
into
7.84.x

Backport c7f76a5 from #57902.
What does this PR do?
Reverts #57594, taking the embedded Python back from 3.13.16 to 3.13.15. It also makes the weekly automated bump (
dda inv python-version.update) skip 3.13.16, so the bot doesn't reopen the same upgrade next Tuesday.The bump was also backported to
7.84.x(#57866) and7.85.x(#57867), so this PR carries the matchingbackport/*labels. No tag ships 3.13.16 yet (7.84.2and7.85.0-rc.4still have 3.13.15). The next RC or patch release from those branches would.Motivation
Since #57594 merged (Oct 8, 19:09 UTC), the Agent intermittently fails to load Python checks at startup. The import raises:
The check is never retried, so it stays missing for the whole life of the Agent. In container e2e tests this shows up as
TestKindSuite/TestControlPlane(kube_apiserver_metrics,kube_scheduler,kube_controller_manager) and Kind/EKSTestPrometheus(openmetrics) failing with "Should NOT be empty, but was []". In the dddev logs of e2e Agents we also see it foretcd,corednsandargo_rollouts, and onprometheus_client.samplesas well asrequests.exceptions.Evidence that the bump is the cause:
TestKindSuite/TestControlPlanefailed 62 times onmainin the last 3 days, against once in the 11 days before.8308aba1(the parent of the bump) passes on every job;a8c258b6and every commit after it fail on some jobs._DeadlockErrornever appears in the dddev logs of e2e Agents during the 14 days before pipeline 143514636 (a8c258b6). It has appeared in every pipeline since.Root cause: an upstream CPython regression
3.13.16 includes the backport (python/cpython#155915) of python/cpython#130094, a fix for a different concurrent-import race. That change makes
importlib._bootstrap._find_and_load_unlocked(name)re-import the parent package when the parent's spec is still_initializing, while still holding the module lock ofname. Until 3.13.15, a thread holding a submodule's lock never waited for a parent that another thread was still executing; it just loaded the submodule. It now does, which closes a cycle:requests/__init__.py(it holds therequestslock) and reachesfrom .exceptions import ..., which needs therequests.exceptionslock;requests.exceptionsdirectly (for examplefrom requests.exceptions import ...in a check). It takes therequests.exceptionslock, seesrequestsis still initializing, and now waits for therequestslock.importlib detects the cycle and raises
_DeadlockErrorin one of the two threads. Any package whose__init__imports its own submodules can hit this, and that's most of them (requests,prometheus_client,datadog_checks.basethroughlazy_loader, ...). The Agent isn't doing anything unusual. It imports checks from several goroutines (the autodiscovery discoverer workers and the collector scheduler), which worked fine up to 3.13.15.We aren't the only ones hit by this. Other projects saw the same
_DeadlockError/ hang after moving to 3.13.16, or to 3.14, which has had the change since 3.14.0:_find_and_load_unlockedfrom 3.13.15 and 3.13.16 to confirm it._DeadlockErrorwhen one thread importsedgar.referencewhile another importsedgar._DeadlockErroronopenai.resources.*under 3.14. They worked around it with lazy package exports.Nobody has reported it upstream yet: python/cpython#130094 is closed and no newer issue mentions this regression. We plan to file one.
Describe how you validated your changes
dda inv invoke-unit-tests.run --tests=python_versionpasses, including a new test for the excluded version.TestControlPlaneandTestPrometheuspass again.Additional Notes
EXCLUDED_PYTHON_VERSIONSonly skips 3.13.16. The bot will propose 3.13.17 when it comes out. Whoever reviews that PR has to check that it no longer carries the change, or add it to the list.new-e2e-containers*jobs were all skipped. They are manual on PRs, and dynamic test selection (--impacted) maps nothing to adeps/cpythonchange, so it skipsTestKindSuiteeven when the jobs are played (it had to be forced withRUN_E2E_TESTS=onon this PR). Changes to the embedded Python should run the e2e suites that load Python checks. A dedicated test that loads many Python checks at once would also catch this directly.🤖 Generated with Claude Code