[Backport 7.84.x] [CONTINT-3239] Revert "[automated] Upgrade embedded Python patch version to 3.13.16" by dd-octo-sts[bot] · Pull Request #57907 · DataDog/datadog-agent · GitHub
Skip to content

[Backport 7.84.x] [CONTINT-3239] Revert "[automated] Upgrade embedded Python patch version to 3.13.16" - #57907

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 1 commit into
7.84.xfrom
backport-57902-to-7.84.x
Oct 9, 2026
Merged

gh-worker-dd-mergequeue-cf854d[bot] merged 1 commit into
7.84.xfrom
backport-57902-to-7.84.x

Conversation

@dd-octo-sts

@dd-octo-sts dd-octo-sts Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

Backport c7f76a5 from #57902.


What does this PR do?

Reverts #57594, taking the embedded Python back from 3.13.16 to 3.13.15. It also makes the weekly automated bump (dda inv python-version.update) skip 3.13.16, so the bot doesn't reopen the same upgrade next Tuesday.

The bump was also backported to 7.84.x (#57866) and 7.85.x (#57867), so this PR carries the matching backport/* labels. No tag ships 3.13.16 yet (7.84.2 and 7.85.0-rc.4 still have 3.13.15). The next RC or patch release from those branches would.

Motivation

Since #57594 merged (Oct 8, 19:09 UTC), the Agent intermittently fails to load Python checks at startup. The import raises:

unable to import module 'datadog_checks.kube_apiserver_metrics': Traceback (most recent call last):
  ...
  File ".../datadog_checks/base/checks/openmetrics/base_check.py", line 6, in <module>
    import requests
  File ".../requests/__init__.py", line 47, in <module>
    from .exceptions import RequestsDependencyWarning
  File "<frozen importlib._bootstrap>", line 1366, in _find_and_load
  File "<frozen importlib._bootstrap>", line 418, in __enter__
  File "<frozen importlib._bootstrap>", line 346, in acquire
_frozen_importlib._DeadlockError: deadlock detected by _ModuleLock('requests.exceptions')

The check is never retried, so it stays missing for the whole life of the Agent. In container e2e tests this shows up as TestKindSuite/TestControlPlane (kube_apiserver_metrics, kube_scheduler, kube_controller_manager) and Kind/EKS TestPrometheus (openmetrics) failing with "Should NOT be empty, but was []". In the dddev logs of e2e Agents we also see it for etcd, coredns and argo_rollouts, and on prometheus_client.samples as well as requests.exceptions.

Evidence that the bump is the cause:

  • TestKindSuite/TestControlPlane failed 62 times on main in the last 3 days, against once in the 11 days before. 8308aba1 (the parent of the bump) passes on every job; a8c258b6 and every commit after it fail on some jobs.
  • _DeadlockError never appears in the dddev logs of e2e Agents during the 14 days before pipeline 143514636 (a8c258b6). It has appeared in every pipeline since.
  • Only metrics from Python checks go missing. Go-check metrics from the same runs keep arriving.

Root cause: an upstream CPython regression

3.13.16 includes the backport (python/cpython#155915) of python/cpython#130094, a fix for a different concurrent-import race. That change makes importlib._bootstrap._find_and_load_unlocked(name) re-import the parent package when the parent's spec is still _initializing, while still holding the module lock of name. Until 3.13.15, a thread holding a submodule's lock never waited for a parent that another thread was still executing; it just loaded the submodule. It now does, which closes a cycle:

  • thread A runs requests/__init__.py (it holds the requests lock) and reaches from .exceptions import ..., which needs the requests.exceptions lock;
  • thread B imports requests.exceptions directly (for example from requests.exceptions import ... in a check). It takes the requests.exceptions lock, sees requests is still initializing, and now waits for the requests lock.

importlib detects the cycle and raises _DeadlockError in one of the two threads. Any package whose __init__ imports its own submodules can hit this, and that's most of them (requests, prometheus_client, datadog_checks.base through lazy_loader, ...). The Agent isn't doing anything unusual. It imports checks from several goroutines (the autodiscovery discoverer workers and the collector scheduler), which worked fine up to 3.13.15.

We aren't the only ones hit by this. Other projects saw the same _DeadlockError / hang after moving to 3.13.16, or to 3.14, which has had the change since 3.14.0:

Nobody has reported it upstream yet: python/cpython#130094 is closed and no newer issue mentions this regression. We plan to file one.

Describe how you validated your changes

  • dda inv invoke-unit-tests.run --tests=python_version passes, including a new test for the excluded version.
  • The container e2e jobs (Kind and EKS suites) are re-run on this PR to confirm TestControlPlane and TestPrometheus pass again.

Additional Notes

  • EXCLUDED_PYTHON_VERSIONS only skips 3.13.16. The bot will propose 3.13.17 when it comes out. Whoever reviews that PR has to check that it no longer carries the change, or add it to the list.
  • [automated] Upgrade embedded Python patch version to 3.13.16 #57594 got merged without the container e2e tests ever running on it: its new-e2e-containers* jobs were all skipped. They are manual on PRs, and dynamic test selection (--impacted) maps nothing to a deps/cpython change, so it skips TestKindSuite even when the jobs are played (it had to be forced with RUN_E2E_TESTS=on on this PR). Changes to the embedded Python should run the e2e suites that load Python checks. A dedicated test that loads many Python checks at once would also catch this directly.
  • This problem will come back with Python 3.14. Before moving to 3.14 (or to a 3.13 patch release that keeps the change), we'll need an Agent-side mitigation, such as importing shared dependencies on one thread before checks load concurrently, or an upstream fix.

🤖 Generated with Claude Code

…ion to 3.13.16" (#57902) ### What does this PR do?

Reverts #57594, taking the embedded Python back from 3.13.16 to 3.13.15. It also makes the weekly automated bump (`dda inv python-version.update`) skip 3.13.16, so the bot doesn't reopen the same upgrade next Tuesday.

The bump was also backported to `7.84.x` (#57866) and `7.85.x` (#57867), so this PR carries the matching `backport/*` labels. No tag ships 3.13.16 yet (`7.84.2` and `7.85.0-rc.4` still have 3.13.15). The next RC or patch release from those branches would.

### Motivation

Since #57594 merged (Oct 8, 19:09 UTC), the Agent intermittently fails to load Python checks at startup. The import raises:

```
unable to import module 'datadog_checks.kube_apiserver_metrics': Traceback (most recent call last):
  ...
  File ".../datadog_checks/base/checks/openmetrics/base_check.py", line 6, in <module>
    import requests
  File ".../requests/__init__.py", line 47, in <module>
    from .exceptions import RequestsDependencyWarning
  File "<frozen importlib._bootstrap>", line 1366, in _find_and_load
  File "<frozen importlib._bootstrap>", line 418, in __enter__
  File "<frozen importlib._bootstrap>", line 346, in acquire
_frozen_importlib._DeadlockError: deadlock detected by _ModuleLock('requests.exceptions')
```

The check is never retried, so it stays missing for the whole life of the Agent. In container e2e tests this shows up as `TestKindSuite/TestControlPlane` (`kube_apiserver_metrics`, `kube_scheduler`, `kube_controller_manager`) and Kind/EKS `TestPrometheus` (`openmetrics`) failing with "Should NOT be empty, but was []". In the dddev logs of e2e Agents we also see it for `etcd`, `coredns` and `argo_rollouts`, and on `prometheus_client.samples` as well as `requests.exceptions`.

Evidence that the bump is the cause:
- `TestKindSuite/TestControlPlane` failed 62 times on `main` in the last 3 days, against once in the 11 days before. `8308aba1` (the parent of the bump) passes on every job; `a8c258b6` and every commit after it fail on some jobs.
- `_DeadlockError` never appears in the dddev logs of e2e Agents during the 14 days before pipeline 143514636 (`a8c258b6`). It has appeared in every pipeline since.
- Only metrics from Python checks go missing. Go-check metrics from the same runs keep arriving.

#### Root cause: an upstream CPython regression

3.13.16 includes the backport (python/cpython#155915) of python/cpython#130094, a fix for a different concurrent-import race. That change makes `importlib._bootstrap._find_and_load_unlocked(name)` re-import the parent package when the parent's spec is still `_initializing`, while still holding the module lock of `name`. Until 3.13.15, a thread holding a submodule's lock never waited for a parent that another thread was still executing; it just loaded the submodule. It now does, which closes a cycle:

- thread A runs `requests/__init__.py` (it holds the `requests` lock) and reaches `from .exceptions import ...`, which needs the `requests.exceptions` lock;
- thread B imports `requests.exceptions` directly (for example `from requests.exceptions import ...` in a check). It takes the `requests.exceptions` lock, sees `requests` is still initializing, and now waits for the `requests` lock.

importlib detects the cycle and raises `_DeadlockError` in one of the two threads. Any package whose `__init__` imports its own submodules can hit this, and that's most of them (`requests`, `prometheus_client`, `datadog_checks.base` through `lazy_loader`, ...). The Agent isn't doing anything unusual. It imports checks from several goroutines (the autodiscovery discoverer workers and the collector scheduler), which worked fine up to 3.13.15.

We aren't the only ones hit by this. Other projects saw the same `_DeadlockError` / hang after moving to 3.13.16, or to 3.14, which has had the change since 3.14.0:
- gke-labs/kube-agents#2446 and gke-labs/kube-agents#2464: hang in a concurrent-import test, traced to the 3.13.16 backport. They compared `_find_and_load_unlocked` from 3.13.15 and 3.13.16 to confirm it.
- dgunning/edgartools#1325: `_DeadlockError` when one thread imports `edgar.reference` while another imports `edgar`.
- NousResearch/hermes-agent#123103, NousResearch/hermes-agent#129507 and NousResearch/hermes-agent#132146: `_DeadlockError` on `openai.resources.*` under 3.14. They worked around it with lazy package exports.
- marimo-team/marimo#11095: concurrent import deadlocks with bokeh.

Nobody has reported it upstream yet: python/cpython#130094 is closed and no newer issue mentions this regression. We plan to file one.

### Describe how you validated your changes

- `dda inv invoke-unit-tests.run --tests=python_version` passes, including a new test for the excluded version.
- The container e2e jobs (Kind and EKS suites) are re-run on this PR to confirm `TestControlPlane` and `TestPrometheus` pass again.

### Additional Notes

- `EXCLUDED_PYTHON_VERSIONS` only skips 3.13.16. The bot will propose 3.13.17 when it comes out. Whoever reviews that PR has to check that it no longer carries the change, or add it to the list.
- #57594 got merged without the container e2e tests ever running on it: its `new-e2e-containers*` jobs were all skipped. They are manual on PRs, and dynamic test selection (`--impacted`) maps nothing to a `deps/cpython` change, so it skips `TestKindSuite` even when the jobs are played (it had to be forced with `RUN_E2E_TESTS=on` on this PR). Changes to the embedded Python should run the e2e suites that load Python checks. A dedicated test that loads many Python checks at once would also catch this directly.
- This problem will come back with Python 3.14. Before moving to 3.14 (or to a 3.13 patch release that keeps the change), we'll need an Agent-side mitigation, such as importing shared dependencies on one thread before checks load concurrently, or an upstream fix.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: regis.desgroppes <regis.desgroppes@datadoghq.com>
(cherry picked from commit c7f76a5)

___

Co-authored-by: Lénaïc Huard <L3n41c@users.noreply.github.com>
@dd-octo-sts dd-octo-sts Bot added backport bot changelog/no-changelog No changelog entry needed labels Oct 9, 2026
@dd-octo-sts
dd-octo-sts Bot requested a review from a team as a code owner October 9, 2026 12:13
@dd-octo-sts dd-octo-sts Bot added internal Identify a non-fork PR medium review PR review might take time qa/done QA done before merge and regressions are covered by tests labels Oct 9, 2026
@dd-octo-sts
dd-octo-sts Bot requested a review from a team as a code owner October 9, 2026 12:13
@dd-octo-sts
dd-octo-sts Bot requested a review from a team as a code owner October 9, 2026 12:13
@dd-octo-sts dd-octo-sts Bot added team/agent-devx team/agent-integrations team/container-integrations changelog/no-changelog No changelog entry needed qa/done QA done before merge and regressions are covered by tests backport bot medium review PR review might take time team/agent-build internal Identify a non-fork PR labels Oct 9, 2026
@datadog-official

datadog-official Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

@dd-octo-sts

dd-octo-sts Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor Author

Files inventory check summary

File checks results against ancestor b1f537e1:

Results for datadog-agent_7.84.2.git.8.7251fc7.pipeline.143686757-1_amd64.deb:

No change detected

Results for datadog-iot-agent_7.84.2.git.8.7251fc7.pipeline.143686757-1_amd64.deb:

No change detected

@cit-pr-commenter-54b7da

Copy link
Copy Markdown

Regression Detector

Regression Detector Results

Metrics dashboard
Target profiles
Job ID: 4701e5b5-b20a-4976-bc08-9fa0a01cb09a

Baseline: b1f537e
Comparison: 7251fc7
Diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment

perf experiment goal Δ mean % Δ mean % CI trials links
➖ quality_gate_metrics_logs memory utilization +1.46 [+1.23, +1.68] 1 Logs bounds checks dashboard
➖ dsd_uds_10mb_3k_timestamped_contexts_memory memory utilization +0.69 [+0.48, +0.90] 1 Logs
➖ quality_gate_idle memory utilization +0.16 [+0.12, +0.21] 1 Logs bounds checks dashboard
➖ quality_gate_security_idle memory utilization +0.11 [+0.08, +0.15] 1 Logs bounds checks dashboard
➖ quality_gate_security_mean_fs_load memory utilization +0.05 [+0.02, +0.09] 1 Logs bounds checks dashboard
➖ dsd_uds_10mb_3k_timestamped_contexts_cpu % cpu utilization -0.03 [-0.27, +0.21] 1 Logs
➖ quality_gate_security_no_fs_load memory utilization -0.04 [-0.11, +0.04] 1 Logs bounds checks dashboard
➖ quality_gate_idle_all_features memory utilization -0.08 [-0.12, -0.04] 1 Logs bounds checks dashboard
➖ quality_gate_private_action_runner memory utilization -0.30 [-0.43, -0.18] 1 Logs bounds checks dashboard
➖ quality_gate_logs % cpu utilization -1.79 [-2.64, -0.94] 1 Logs bounds checks dashboard

Bounds Checks: ✅ Passed

perf experiment bounds_check_name replicates_passed observed_value links
✅ quality_gate_idle intake_connections 10/10 4 ≤ 5 bounds checks dashboard
✅ quality_gate_idle memory_usage 10/10 172.71MiB ≤ 179MiB bounds checks dashboard
✅ quality_gate_idle total_bytes_received 10/10 745.78KiB ≤ 819.20KiB bounds checks dashboard
✅ quality_gate_idle_all_features intake_connections 10/10 4 ≤ 5 bounds checks dashboard
✅ quality_gate_idle_all_features memory_usage 10/10 516.31MiB ≤ 537MiB bounds checks dashboard
✅ quality_gate_idle_all_features total_bytes_received 10/10 1.13MiB ≤ 1.25MiB bounds checks dashboard
✅ quality_gate_logs intake_connections 10/10 18 ≤ 40 bounds checks dashboard
✅ quality_gate_logs memory_usage 10/10 211.47MiB ≤ 228MiB bounds checks dashboard
✅ quality_gate_logs missed_bytes 10/10 0B = 0B bounds checks dashboard
✅ quality_gate_logs total_bytes_received 10/10 263.06MiB ≤ 292MiB bounds checks dashboard
✅ quality_gate_metrics_logs cpu_usage 10/10 369.57 ≤ 2000 bounds checks dashboard
✅ quality_gate_metrics_logs intake_connections 10/10 20 ≤ 40 bounds checks dashboard
✅ quality_gate_metrics_logs memory_usage 10/10 441.57MiB ≤ 455MiB bounds checks dashboard
✅ quality_gate_metrics_logs missed_bytes 10/10 0B = 0B bounds checks dashboard
✅ quality_gate_metrics_logs total_bytes_received 10/10 0.94GiB ≤ 1.04GiB bounds checks dashboard
✅ quality_gate_private_action_runner memory_usage 10/10 71.96MiB ≤ 75MiB bounds checks dashboard
✅ quality_gate_security_idle cpu_usage 10/10 29.40 ≤ 100 bounds checks dashboard
✅ quality_gate_security_idle memory_usage 10/10 328.30MiB ≤ 355MiB bounds checks dashboard
✅ quality_gate_security_mean_fs_load cpu_usage 10/10 66.72 ≤ 200 bounds checks dashboard
✅ quality_gate_security_mean_fs_load memory_usage 10/10 301.71MiB ≤ 335MiB bounds checks dashboard
✅ quality_gate_security_no_fs_load cpu_usage 10/10 24.71 ≤ 100 bounds checks dashboard
✅ quality_gate_security_no_fs_load memory_usage 10/10 306.75MiB ≤ 345MiB bounds checks dashboard

Explanation

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

Performance changes are noted in the perf column of each table:

  • ✅ = significantly better comparison variant performance
  • ❌ = significantly worse comparison variant performance
  • ➖ = no significant change in performance

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

  1. Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.

  2. Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.

  3. Its configuration does not mark it "erratic".

CI Pass/Fail Decision

✅ Passed. All Quality Gates passed.

  • quality_gate_security_no_fs_load, bounds check memory_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_security_no_fs_load, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_security_idle, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_security_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_metrics_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_metrics_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
  • quality_gate_metrics_logs, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
  • quality_gate_metrics_logs, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_metrics_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
  • quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.
  • quality_gate_idle_all_features, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
  • quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
  • quality_gate_logs, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
  • quality_gate_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
  • quality_gate_private_action_runner, bounds check memory_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_security_mean_fs_load, bounds check memory_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_security_mean_fs_load, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_idle, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
  • quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.
  • quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.

@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot merged commit 08f5458 into 7.84.x Oct 9, 2026
375 of 390 checks passed
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot deleted the backport-57902-to-7.84.x branch October 9, 2026 18:43
@github-actions github-actions Bot added this to the 7.84.3 milestone Oct 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport bot changelog/no-changelog No changelog entry needed internal Identify a non-fork PR medium review PR review might take time qa/done QA done before merge and regressions are covered by tests team/agent-build team/agent-devx team/agent-integrations team/container-integrations

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant