Optimize CI hot-path setup without reducing coverage by zozo123 · Pull Request #73124 · apache/airflow · GitHub
Skip to content

Optimize CI hot-path setup without reducing coverage - #73124

Closed
zozo123 wants to merge 4 commits into
apache:mainfrom
zozo123:ci/fastest-ci-full
Closed

zozo123 wants to merge 4 commits into
apache:mainfrom
zozo123:ci/fastest-ci-full

Conversation

@zozo123

@zozo123 zozo123 commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Reduce repeated CI setup work while preserving the selected test commands and supported Python/backend environments.

  • Key prek environments by host, interpreter, tool versions, dependency configuration, and absolute paths. Validate restored hooks, recover from unusable archives, preserve unrelated caches, and use bounded fast compression with cache timing summaries.
  • Use the pnpm package-store cache instead of duplicate node_modules archives. Install only the selected Playwright engine, with support for all retained.
  • Run migration validation once per eligible Python/backend environment in an existing test shard, retaining failure propagation and the Python 3.14 exemption.
  • Bound parallel disk cleanup to four workers and avoid recursive ownership changes for empty directories while preserving cleanup targets and error handling.

Validation: all 133 cases in scripts/tests/ci/test_ci_setup_optimizations.py pass in the Python 3.10 Breeze environment. These exercise cache identity and recovery, archive permissions and symlinks, browser selection, cleanup failure propagation, and migration environment coverage.

The initial upstream run exposed formatting issues in the new tests, now corrected. It also encountered external download failures (Gradle HTTP 504 and a connection reset during Helm schema validation). Fresh upstream CI must pass before merge. No end-to-end speedup is claimed without comparable completed runs.


Was generative AI tooling used to co-author this PR?
  • Yes — ChatGPT and Codex (GPT-6)

Generated-by: ChatGPT and Codex (GPT-6) following the guidelines

zozo123 and others added 3 commits September 14, 2026 17:00
Keep selected test commands and supported matrices unchanged. Validate prek
hook environments after restore, use a host-aware cache identity and bounded
fast compression, and remove duplicate UI/browser archive transfers.
Preserve cleanup failure handling and migration environment coverage.
Add 133 targeted setup and workflow regression cases.

Generated-by: ChatGPT
Use one existing shard per selected Python/backend environment for migration
validation instead of provisioning another runner and restoring the image
again. Preserve the original test check names and propagate migration
failures through an existing check, without requiring branch-rule changes.
Keep all test commands, matrix exclusions and the unsupported-Python guard.

Generated-by: ChatGPT
Static checks failed on the new test module: ruff removed a redundant blank
line before the first-party import, and ruff format normalized the raw string
quotes, a comprehension, and an implicit string concatenation.

Also materialize the itertools.product argvalues for the cache-refresh-policy
parametrization as a list, so pytest does not emit PytestRemovedIn10Warning
for a non-Collection iterable.

All 133 cases in scripts/tests/ci/test_ci_setup_optimizations.py still pass,
including with -W error::DeprecationWarning.

Generated-by: ChatGPT

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Andrushika

Copy link
Copy Markdown
Contributor

@zozo123

zozo123 commented Sep 15, 2026

Copy link
Copy Markdown
Contributor Author

@zozo123 zozo123 closed this Sep 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants