Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

/azp run runtime-coreclr libraries-jitstress, runtime-coreclr libraries-pgo

libraries-pgo failures are from profile inconsistencies during loop inversion. #111684 has a fix for this.

cc @dotnet/jit-contrib, @AndyAyersMS PTAL. Diffs show churn from 3-opt considering edges in a different order now.

This change isn't all that important on its own, but for a while now, I've wanted to consolidate all of our separate layout phases (RPO order, fgMoveHotJumps, cold code motion, and 3-opt) into something simpler. Instead of all these intermediate reorderings, I was thinking we'd feed a loop-aware RPO traversal that skipped all the cold blocks into 3-opt directly, and then reorder the block list once, while respecting EH invariants such that we don't need to completely rebuild EH regions. To get an idea of what this would look like, I have a prototype in #112004.

While working on this prototype, I found 3-opt's per-region ordering semantics to be problematic, since they implicitly depend on the initial layout keeping EH regions contiguous, or else the ordering of a nested region could change the end block of the parent region, breaking the bounds checking of the next 3-opt iteration. This seems like a fragile invariant, and getting 3-opt to only consider branches within a region is pretty simple, so I found it easiest to run 3-opt once for the entire hot region, hence this PR.

If my plan doesn't seem ill-advised to you, I'd succeed this PR with the following:

• Remove fgMoveHotJumps -- I don't think this has been all that important ever since we turned 3-opt on. Removing it churns the initial layout into 3-opt, which can lead to subtly different layouts that still have locally-minimal scores, so I don't think the churn should dissuade us too much. Regressions from this ought to serve as motivation for improving 3-opt's ability to find globally-minimal layout scores.
• Get rid of RPO layout and cold code motion. In their place, compute a loop-aware RPO that skips the cold blocks (such that they "sink" to the end of the method), and feed this into 3-opt as its initial layout. This has the nice side effect of never scrambling the block list in between phases, plus we can simplify our post-layout EH region repair to only fix the exception table, since we never split up the actual regions.
• Clean up any remaining invariants that can be relaxed.

Thanks!