fix: make benchmark tests resilient to CodSpeed's repeated test body execution by Copilot · Pull Request #210 · cpp-linter/cpp-linter-hooks · GitHub
Skip to content

fix: make benchmark tests resilient to CodSpeed's repeated test body execution#210

Merged
shenxianpeng merged 3 commits into
mainfrom
copilot/fix-issue-from-main-branch
Apr 7, 2026
Merged

fix: make benchmark tests resilient to CodSpeed's repeated test body execution#210
shenxianpeng merged 3 commits into
mainfrom
copilot/fix-issue-from-main-branch

Conversation

Copilot AI commented Apr 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor
  • Investigate CI failure: CodSpeed benchmark run calls test body multiple times (warmup + measurement) but only runs fixtures once; also, TESTING_DIR = Path("testing") is relative, causing FileNotFoundError when CWD changes
  • Fix TESTING_DIR to use absolute path Path(__file__).parent.parent / "testing"
  • Fix "reformats" tests to reset file content at start of body and compare against MAIN_C.read_bytes() (handles repeated benchmark calls)
  • Apply ruff-format to long assertion line (pre-commit CI failure)
  • Validate changes pass all 25 integration tests + 49 other tests

@shenxianpeng
fix: use absolute TESTING_DIR and reset file in benchmark tests
4e54e3c 
- Change TESTING_DIR to Path(__file__).parent.parent / "testing" so all
  testing/ paths are absolute and CWD-independent
- Reset file to unformatted bytes at start of "reformats" test bodies
  and compare against MAIN_C.read_bytes() to handle pytest-codspeed
  calling benchmark bodies multiple times per test invocation

Agent-Logs-Url: https://github.com/cpp-linter/cpp-linter-hooks/sessions/e6f1960a-f8fc-4507-8baa-2b659f052346

Co-authored-by: shenxianpeng <3353385+shenxianpeng@users.noreply.github.com>
github-advanced-security[bot]
github-advanced-security AI found potential problems Apr 6, 2026
View reviewed changes
Comment thread tests/test_integration.py Dismissed
Comment thread tests/test_integration.py Dismissed
Copilot AI changed the title [WIP] Fix the issue from main branch fix: make benchmark tests resilient to CodSpeed's repeated test body execution Apr 6, 2026
Copilot AI requested a review from shenxianpeng April 6, 2026 22:23
@github-actions github-actions Bot added the bug Something isn't working label Apr 7, 2026
@codecov

codecov Bot commented Apr 7, 2026 •
edited
Loading

Copy link
Copy Markdown

Copilot AI mentioned this pull request Apr 7, 2026
Closed
@shenxianpeng

shenxianpeng commented Apr 7, 2026

Copy link
Copy Markdown
Member

@copilot Reference: //pull/210/checks?check_run_id=70156607477 and fix ci failures

Copilot AI commented Apr 7, 2026

Copy link
Copy Markdown
Contributor Author

@copilot Reference: //pull/210/checks?check_run_id=70156607477 and fix ci failures

Fixed in c792a06 — the pre-commit ruff-format CI failure was a long assertion line at line 71 that ruff wanted wrapped. Applied the formatting fix.

@sonarqubecloud

sonarqubecloud Bot commented Apr 7, 2026 •
edited
Loading

Copy link
Copy Markdown

Quality Gate Passed Quality Gate passed

Issues
0 New issues
2 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

@codspeed-hq

codspeed-hq Bot commented Apr 7, 2026 •
edited
Loading

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 63 untouched benchmarks
🆕 25 new benchmarks
⏩ 13 skipped benchmarks1

Performance Changes

Benchmark BASE HEAD Efficiency
🆕 test_clang_format_style_from_file_with_version[22] N/A 1.1 ms N/A
🆕 test_clang_format_style_from_file_with_version[21] N/A 1.1 ms N/A
🆕 test_clang_format_style_preset_is_idempotent[Chromium] N/A 1.4 ms N/A
🆕 test_clang_format_style_preset_is_idempotent[LLVM] N/A 1.4 ms N/A
🆕 test_clang_format_style_preset_reformats_unformatted_file[Chromium] N/A 810.8 µs N/A
🆕 test_clang_format_google_style_matches_good_c N/A 809.3 µs N/A
🆕 test_clang_format_style_preset_is_idempotent[WebKit] N/A 1.4 ms N/A
🆕 test_clang_format_style_preset_is_idempotent[Google] N/A 1.4 ms N/A
🆕 test_clang_format_style_from_file_with_version[16] N/A 1.1 ms N/A
🆕 test_clang_format_style_preset_is_idempotent[Mozilla] N/A 1.4 ms N/A
🆕 test_clang_format_style_from_file_reformats N/A 812.5 µs N/A
🆕 test_clang_format_style_preset_is_idempotent[Microsoft] N/A 1.4 ms N/A
🆕 test_clang_format_style_preset_reformats_unformatted_file[LLVM] N/A 813.2 µs N/A
🆕 test_clang_format_style_from_file_with_version[19] N/A 1.1 ms N/A
🆕 test_clang_format_style_preset_reformats_unformatted_file[Google] N/A 810.8 µs N/A
🆕 test_clang_format_style_from_file_with_version[20] N/A 1.1 ms N/A
🆕 test_clang_format_style_from_file_is_idempotent N/A 1.3 ms N/A
🆕 test_clang_format_style_from_file_with_version[18] N/A 1.1 ms N/A
🆕 test_clang_format_style_from_file_with_version[17] N/A 1.1 ms N/A
🆕 test_clang_format_style_preset_reformats_unformatted_file[Microsoft] N/A 810.3 µs N/A
... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Comparing copilot/fix-issue-from-main-branch (c792a06) with main (0d4c4db)2

Open in CodSpeed

Footnotes

  1. 13 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. No successful run was found on main (dedddc3) during the generation of this report, so 0d4c4db was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

@shenxianpeng shenxianpeng marked this pull request as ready for review April 7, 2026 05:22
@shenxianpeng shenxianpeng merged commit fd5cb74 into main Apr 7, 2026
20 checks passed
@shenxianpeng shenxianpeng deleted the copilot/fix-issue-from-main-branch branch April 7, 2026 05:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@shenxianpeng shenxianpeng Awaiting requested review from shenxianpeng
1 more reviewer
@github-advanced-security github-advanced-security[bot] github-advanced-security[bot] left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

@shenxianpeng shenxianpeng

Copilot code review Copilot

Labels

bug Something isn't working

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@shenxianpeng @github-advanced-security