-
Notifications
You must be signed in to change notification settings - Fork 1.5k
perf: run pyperformance in CI and close the gaps it finds #8663
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
4cb919a
10d327e
cf6b8c3
11f8331
d2e6d8b
1164140
7779043
f4781e8
66671d8
2bace67
b795621
de3006f
35fde2c
e0d6de6
16545f1
d9917ab
58e5797
5150706
71503be
105de35
a2fd0fb
187f1bd
8d82295
3ce7a3e
0f9bdb9
0a9b0ca
f1aaa09
b6f3528
c3f24db
4c7a728
d3372c4
f33054b
b97d542
e648e36
2d44f3c
0733c34
f949a12
23979f6
b88b571
a3df72c
a126a94
1b3d1fd
c5a05b0
5e244fa
df078a2
141f87c
c90081b
3c7b479
8d0ebf9
eade737
e5c7634
c5c8090
7b69fd0
1ac89d4
baafe8e
1059850
88e722f
d0b8045
6f4396f
65c509f
812d3f3
66a24e8
117826a
96c0bfc
b6f1fd7
0647ee9
ab86664
08a9726
e068bfb
3266846
06a3415
c727056
307e346
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,65 @@ | ||
| on: | ||
| workflow_run: | ||
| workflows: [pyperformance] | ||
| types: [completed] | ||
|
|
||
| name: pyperformance comment | ||
|
|
||
| # Same split as `codspeed-comment.yaml`: a `pull_request` run from a fork gets | ||
| # a read-only token however it is configured, so the comparison is computed | ||
| # there and handed over as an artifact, and this workflow -- which | ||
| # `workflow_run` always runs in the base repo's context -- is the one trusted | ||
| # with a token that may write the comment. | ||
| permissions: | ||
| contents: read | ||
| pull-requests: write | ||
|
|
||
| jobs: | ||
| comment: | ||
| if: ${{ github.event.workflow_run.conclusion == 'success' }} | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - name: Download the comparison produced by the pyperformance workflow | ||
| uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 | ||
| with: | ||
| name: pyperformance-diff | ||
| run-id: ${{ github.event.workflow_run.id }} | ||
| github-token: ${{ github.token }} | ||
| continue-on-error: true # absent when the job was skipped or dispatched | ||
|
|
||
| - name: Post or update the PR comment | ||
| uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0 | ||
| with: | ||
| script: | | ||
| const fs = require('fs'); | ||
| if (!fs.existsSync('diff.md') || !fs.existsSync('pr_number.txt')) { | ||
| core.info('No diff.md/pr_number.txt in this run; nothing to comment.'); | ||
| return; | ||
| } | ||
|
|
||
| const prNumber = parseInt(fs.readFileSync('pr_number.txt', 'utf8').trim(), 10); | ||
| const body = fs.readFileSync('diff.md', 'utf8'); | ||
| const marker = '<!-- pyperformance-local-diff -->'; | ||
|
|
||
| const comments = await github.paginate(github.rest.issues.listComments, { | ||
| owner: context.repo.owner, | ||
| repo: context.repo.repo, | ||
| issue_number: prNumber, | ||
| }); | ||
| const existing = comments.find((c) => c.body && c.body.includes(marker)); | ||
|
|
||
| if (existing) { | ||
| await github.rest.issues.updateComment({ | ||
| owner: context.repo.owner, | ||
| repo: context.repo.repo, | ||
| comment_id: existing.id, | ||
| body, | ||
| }); | ||
| } else { | ||
| await github.rest.issues.createComment({ | ||
| owner: context.repo.owner, | ||
| repo: context.repo.repo, | ||
| issue_number: prNumber, | ||
| body, | ||
| }); | ||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,220 @@ | ||
| on: | ||
| pull_request: | ||
| types: [labeled, opened, synchronize, reopened] | ||
| workflow_dispatch: | ||
|
|
||
| name: pyperformance | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| concurrency: | ||
| group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.sha }} | ||
| cancel-in-progress: true | ||
|
|
||
| env: | ||
| CARGO_INCREMENTAL: 0 | ||
| CARGO_TERM_COLOR: always | ||
| CI: true | ||
| # Pure-Python benchmarks only: anything whose workload needs a real C | ||
| # extension cannot install under RustPython at all, and the async/SSL ones | ||
| # are far too slow here to be worth a runner's time. Keep this list short -- | ||
| # every entry is paid for three times (CPython, base, head) in one job. | ||
| BENCHMARKS: >- | ||
| chaos,comprehensions,coroutines,deepcopy,deltablue,fannkuch,float, | ||
| generators,go,hexiom,json_dumps,json_loads,logging,mdp,nbody,nqueens, | ||
| pickle_pure_python,raytrace,regex_compile,regex_v8,richards,scimark, | ||
| spectral_norm,telco,typing_runtime_protocols,unpack_sequence, | ||
| unpickle_pure_python,xml_etree | ||
| # A benchmark that needs longer than this on the base commit is one this | ||
| # comparison cannot afford to wait for; it shows up as a failure in the | ||
| # table, which is itself the interesting signal. | ||
| BENCH_TIMEOUT: 300 | ||
|
|
||
| jobs: | ||
| compare: | ||
| # `skip:ci` mirrors the other workflows; `run:pyperformance` opts a PR in, | ||
| # because the job costs well over an hour and most PRs do not need it. | ||
| if: >- | ||
| github.event_name == 'workflow_dispatch' || | ||
| (!contains(github.event.pull_request.labels.*.name, 'skip:ci') && | ||
| contains(github.event.pull_request.labels.*.name, 'run:pyperformance')) | ||
| name: base vs. head vs. CPython | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 300 | ||
| permissions: | ||
| contents: read | ||
| # Honoured for a pull request opened from this repo, which is then | ||
| # commented on directly by the last step; a fork's run is handed | ||
| # read-only whatever is asked for here and goes the artifact route. | ||
| pull-requests: write | ||
| steps: | ||
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | ||
| with: | ||
| ref: ${{ github.event.pull_request.head.sha }} | ||
| fetch-depth: 0 | ||
| persist-credentials: false | ||
|
|
||
| - uses: dtolnay/rust-toolchain@stable | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔒 Security & Privacy | 🛡️ Analyzed with Security Review | 🟠 Major | ⚡ Quick win 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
zizmor .github/workflows/pyperformance.yamlRepository: RustPython/RustPython Length of output: 553 🤖 get_repo_knowledge executed:
Length of output: 4600 🏁 Script executed: #!/bin/bash
set -euo pipefail
sed -n '1,120p' .github/workflows/pyperformance.yamlRepository: RustPython/RustPython Length of output: 4922 Security Misconfiguration Reachability: Internal Pin
🤖 Prompt for AI AgentsSource: Coding guidelines |
||
|
|
||
| # Both the third interpreter under test and the host pyperformance | ||
| # itself runs on (it only runs under a real CPython; RustPython is just | ||
| # the `--python` target it drives). | ||
| - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 | ||
| id: cpython | ||
| with: | ||
| python-version: "3.14" | ||
|
|
||
| - name: Restore cache | ||
| uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 | ||
| with: | ||
| path: | | ||
| ~/.cargo/registry/index/ | ||
| ~/.cargo/registry/cache/ | ||
| ~/.cargo/git/db/ | ||
| target/ | ||
| key: pyperformance-${{ hashFiles('**/Cargo.lock') }} | ||
| restore-keys: pyperformance- | ||
| continue-on-error: true | ||
|
|
||
| # The base commit may predate these scripts, and `git checkout <base>` | ||
| # below would then delete them; run them from a copy outside the tree. | ||
| - name: Copy the benchmark scripts out of the checkout | ||
| run: cp -r scripts/pyperformance "${RUNNER_TEMP}/pyperformance-scripts" | ||
|
|
||
| # Each build is copied aside with the stdlib of its own commit, so the | ||
| # second does not overwrite the first and a PR that touches `Lib/` is | ||
| # still compared against the base commit's `Lib/`. A binary moved away | ||
| # from its checkout finds the stdlib only through RUSTPYTHONPATH, which | ||
| # `run_all.py` forwards through pyperf into the venv it builds. | ||
| - name: Build head | ||
| run: | | ||
| cargo build --release --locked | ||
| mkdir -p "${RUNNER_TEMP}/head" | ||
| cp target/release/rustpython "${RUNNER_TEMP}/head/rustpython" | ||
| cp -r Lib "${RUNNER_TEMP}/head/Lib" | ||
|
|
||
| - name: Build base | ||
| continue-on-error: true # a base that cannot build leaves head vs. CPython | ||
| env: | ||
| BASE_SHA: ${{ github.event.pull_request.base.sha }} | ||
| run: | | ||
| git checkout --quiet "${BASE_SHA}" -- | ||
| cargo build --release --locked | ||
| mkdir -p "${RUNNER_TEMP}/base" | ||
| cp target/release/rustpython "${RUNNER_TEMP}/base/rustpython" | ||
| cp -r Lib "${RUNNER_TEMP}/base/Lib" | ||
|
|
||
| - name: Restore head checkout | ||
| env: | ||
| HEAD_SHA: ${{ github.event.pull_request.head.sha }} | ||
| run: git checkout --quiet "${HEAD_SHA}" -- | ||
|
|
||
| - name: Save cache | ||
| if: always() | ||
| uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 | ||
| with: | ||
| path: | | ||
| ~/.cargo/registry/index/ | ||
| ~/.cargo/registry/cache/ | ||
| ~/.cargo/git/db/ | ||
| target/ | ||
| key: pyperformance-${{ hashFiles('**/Cargo.lock') }} | ||
| continue-on-error: true | ||
|
|
||
| # The three runs are deliberately adjacent, with no build between them, | ||
| # so nothing but the interpreter differs across the comparison. | ||
| - name: Measure CPython | ||
| env: | ||
| CPYTHON: ${{ steps.cpython.outputs.python-path }} | ||
| run: | | ||
| python3 "${RUNNER_TEMP}/pyperformance-scripts/run_all.py" \ | ||
| --python "${CPYTHON}" \ | ||
| --label cpython --no-psutil-stub \ | ||
| --benchmarks "${BENCHMARKS//[$'\n' ]/}" \ | ||
| --timeout "${BENCH_TIMEOUT}" --out "${RUNNER_TEMP}/pyperf-results" | ||
|
|
||
| - name: Measure base | ||
| continue-on-error: true # as above: the table falls back to head alone | ||
| env: | ||
| RUSTPYTHONPATH: ${{ runner.temp }}/base/Lib | ||
| run: | | ||
| python3 "${RUNNER_TEMP}/pyperformance-scripts/run_all.py" \ | ||
| --python "${RUNNER_TEMP}/base/rustpython" \ | ||
| --label base \ | ||
| --benchmarks "${BENCHMARKS//[$'\n' ]/}" \ | ||
| --timeout "${BENCH_TIMEOUT}" --out "${RUNNER_TEMP}/pyperf-results" | ||
|
|
||
| - name: Measure head | ||
| env: | ||
| RUSTPYTHONPATH: ${{ runner.temp }}/head/Lib | ||
| run: | | ||
| python3 "${RUNNER_TEMP}/pyperformance-scripts/run_all.py" \ | ||
| --python "${RUNNER_TEMP}/head/rustpython" \ | ||
| --label head \ | ||
| --benchmarks "${BENCHMARKS//[$'\n' ]/}" \ | ||
| --timeout "${BENCH_TIMEOUT}" --out "${RUNNER_TEMP}/pyperf-results" | ||
|
|
||
| - name: Render the comparison | ||
| env: | ||
| CPYTHON_VERSION: ${{ steps.cpython.outputs.python-version }} | ||
| PR_NUMBER: ${{ github.event.pull_request.number }} | ||
| run: | | ||
| python3 "${RUNNER_TEMP}/pyperformance-scripts/pr_diff.py" \ | ||
| --results-dir "${RUNNER_TEMP}/pyperf-results" \ | ||
| --cpython-version "CPython ${CPYTHON_VERSION}" \ | ||
| --md diff.md --json diff.json | ||
| echo "${PR_NUMBER}" > pr_number.txt | ||
|
|
||
| # A `pull_request` run triggered from a fork carries a read-only token no | ||
| # matter what this workflow grants, so the comment is posted from | ||
| # `pyperformance-comment.yaml`, which `workflow_run` runs in the base | ||
| # repo's context. This artifact is the hand-off. | ||
| - name: Upload the comparison | ||
| if: always() | ||
| uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 | ||
| with: | ||
| name: pyperformance-diff | ||
| path: | | ||
| diff.md | ||
| diff.json | ||
| pr_number.txt | ||
| retention-days: 14 | ||
| if-no-files-found: ignore | ||
|
|
||
| # A pull request opened from this repo can be commented on from here, | ||
| # which puts the table on the PR in the same run instead of waiting for | ||
| # the companion workflow. Forks fall through to that workflow instead. | ||
| - name: Post or update the PR comment | ||
| if: >- | ||
| github.event_name == 'pull_request' && | ||
| github.event.pull_request.head.repo.full_name == github.repository | ||
| uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0 | ||
| with: | ||
| script: | | ||
| const fs = require('fs'); | ||
| const body = fs.readFileSync('diff.md', 'utf8'); | ||
| const marker = '<!-- pyperformance-local-diff -->'; | ||
|
|
||
| const comments = await github.paginate(github.rest.issues.listComments, { | ||
| owner: context.repo.owner, | ||
| repo: context.repo.repo, | ||
| issue_number: context.payload.pull_request.number, | ||
| }); | ||
| const existing = comments.find((c) => c.body && c.body.includes(marker)); | ||
|
|
||
| if (existing) { | ||
| await github.rest.issues.updateComment({ | ||
| owner: context.repo.owner, | ||
| repo: context.repo.repo, | ||
| comment_id: existing.id, | ||
| body, | ||
| }); | ||
| } else { | ||
| await github.rest.issues.createComment({ | ||
| owner: context.repo.owner, | ||
| repo: context.repo.repo, | ||
| issue_number: context.payload.pull_request.number, | ||
| body, | ||
| }); | ||
| } | ||

There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Make
workflow_dispatchindependent of pull-request fields.workflow_dispatchhas nogithub.event.pull_request.HEAD_SHAis then empty, so the checkout on Line 111 fails. If a head fallback is added, the workflow still writes an emptypr_number.txt, and the comment workflow usesNaNasissue_number..github/workflows/pyperformance.yaml#L4-L4: removeworkflow_dispatch, or define inputs for the revisions that a manual run must compare..github/workflows/pyperformance.yaml#L110-L111: use a valid dispatch head SHA, or skip the base/head path for manual runs..github/workflows/pyperformance.yaml#L161-L167: do not create a PR-comment hand-off for a manual run..github/workflows/pyperformance-comment.yaml#L40-L40: skip the API call unless the upstream run has a valid associated pull request.📍 Affects 2 files
.github/workflows/pyperformance.yaml#L4-L4(this comment).github/workflows/pyperformance.yaml#L110-L111.github/workflows/pyperformance.yaml#L161-L167.github/workflows/pyperformance-comment.yaml#L40-L40🤖 Prompt for AI Agents