They are all still quite generous, however, giving around 2-4x the average time.

If we wanted to be really precise, we could have step-bound timeouts with specific I/O (deps installs) or test runs capped. One other thing to consider could be that when you take measurements of the old job runs, sometimes they show cache-optimized timings and when that cache gets invalidated, the time will be different. My strategy is to find the slowest non-cached run and give it some percentage on top.

Do these jobs often stall such they frequently hit the timeout?

A lot of these aren't making it a huge amount of difference, like 60->40 or 30, 10->5 or 2.

The main point for timeout is that we don't fall back to the default 6 hours, and have something roughly reasonable.

I'm not sure we need to spend a lot of our own time fine-tuning and then maintaining slim timeouts, when (hopefully!) timeouts are rarely needed.

And going the other way, if something ends up running slower than normal, I don't think we should kill the job early, when it might still finish in some sort of reasonable time. We have had some Ubuntu image fetches running slower not so long ago.