Partition muzzle tasks with the same method as build and tests by smola · Pull Request #9077 · DataDog/dd-trace-java · GitHub
Skip to content

Partition muzzle tasks with the same method as build and tests#9077

Merged
smola merged 3 commits into
masterfrom
smola/muzzle-splits
Jul 7, 2025
Merged

Partition muzzle tasks with the same method as build and tests#9077
smola merged 3 commits into
masterfrom
smola/muzzle-splits

Conversation

@smola

@smola smola commented Jul 1, 2025

Copy link
Copy Markdown
Member

What Does This Do

  • Instead of writeMuzzleTasksToFile and a way to partition tasks externally in the CI config, just use the same partitioning scheme we use for build and test tasks. It's also a simpler setup.
  • ./gradlew :runMuzzle -PtaskPartitionCount=$NORMALIZED_NODE_TOTAL -PtaskPartition=$NORMALIZED_NODE_INDEX

Motivation

  • Aligning muzzle partitioning scheme with the others opens the possibility of partitioned build caches.
  • But even without that, it's simpler and we don't maintain 2 partition methods.

Additional Notes

Contributor Checklist

Jira ticket: [PROJ-IDENT]

@smola smola added tag: no release notes Changes to exclude from release notes comp: tooling Build & Tooling labels Jul 1, 2025
@pr-commenter

pr-commenter Bot commented Jul 1, 2025

Copy link
Copy Markdown

@smola smola marked this pull request as ready for review July 2, 2025 07:13
@smola smola requested review from a team as code owners July 2, 2025 07:13
@smola smola requested review from colin-higgins and dougqh and removed request for a team July 2, 2025 07:13

@PerfectSlayer PerfectSlayer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of curiosity, is there any downside to the build and test partitioning implementation?

@smola

smola commented Jul 2, 2025

Copy link
Copy Markdown
Member Author

@PerfectSlayer In theory yes: the approach used before ensured there were the same number of modules at each partition. While after the PR, it's based on a modulo of the gradle module path's hash code. So each partition might have different sizes. In practice, the precision of the former method is an illusion, since not every module takes the same time to process.
Accurate partitioning would require statistics on the time each task takes from historical runs. Circle CI had the tooling to do this, but I never got to make it work.

@bric3

bric3 commented Jul 3, 2025

Copy link
Copy Markdown
Contributor

@smola smola merged commit eeb36db into master Jul 7, 2025
675 of 676 checks passed
@smola smola deleted the smola/muzzle-splits branch July 7, 2025 07:17
@github-actions github-actions Bot added this to the 1.51.0 milestone Jul 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: tooling Build & Tooling tag: no release notes Changes to exclude from release notes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants