Just drafting so that no one merges it by accident but I still want feedback and discuss this.

RHEL is on x86_64-v2 by default now: https://developers.redhat.com/blog/2021/01/05/building-red-hat-enterprise-linux-9-for-the-x86-64-v2-microarchitecture-level

Also we should probably add the x86_64-vX feature levels instead of using uarch names, because we want the featureset without the uarch specific codegen.

The baseline is mostly defined by the gcc-N package, which is configured to produce baseline binaries when options like -march= are not used. - https://wiki.debian.org/ArchitectureSpecificsMemo#Architecture_baselines

Is that what this change does?

x86-64-v2 (this PR) should be safe-ish to enable and brings a good performance boost.

I think Arch uses x86-64-v3 (haswell), but maybe this is too much.

x86-64-v2 (this PR) should be safe-ish to enable and brings a good performance boost.

I think Arch uses x86-64-v3 (haswell), but maybe this is too much.

AFAIK, most of Arch packages are still using x86-64, though x86_64_v3 architecture was added to devtools. The RFC is to keep both x86_64 and x86_64_v3 packages, instead of bumping the default architecture.

Also we should probably add the x86_64-vX feature levels instead of using uarch names, because we want the featureset without the uarch specific codegen.

I added a suggestion from me to this PR.

Is that what this change does?

Yes, on first look.

x86_64_v3 packages, instead of bumping the default architecture.

We are currently not doing this because it would be equivalent of adding a new architecture because we would build everything twice.

I am also currently (trying) to build my systems with haswell/skylake on my private hydra, to see how things go. I'll probably upstream some fixes along the way, too. The first broken package I encountered was libxcrypt which is already fixed on staging and that when bumping to skylake I couldn't run the jemalloc tests on a zenver3 machine.

Some more cpu features should be added: POPCNT, LAHF-SAHF, CMPXCHG16B...
https://en.wikipedia.org/wiki/X86-64#Microarchitecture_levels

The difference between v2 and higher feature levels is mostly vector extensions, so it's to be expected a compiler (which is mostly pointer chasing) doesn't gain much here.

I want to note that v3 DOES have some non-vector ops which benefits compilers, that includes 3-operand bitshift {S{H{L,R},AL},RO{L,R}}X. Since they have individual destination registers and does not limit CL for shift count, they can reduce register pressure and MOVs, especially when shift-count is not a constant. Bitshifts are heavily used anywhere, including time-sensitive hashing algorithms.
SHRX is also relied by a 1-cycle-per-byte shift-based-DFA algorithm, which can be used for lexing, UTF-8 validation or anything other small-state DFAs.

Here are some benchmarks from arch where the person running it kindly did a tl;dr which I am pasting here:

- there is no or negligible performance benefit of *-march=nehalem*, which corresponds to x86_64-v2,

- there is a moderate benefit of *-march=haswell* (x86_64-v3) - of around 10%-20% as compared to baseline for the tests performed

Link: https://lists.archlinux.org/pipermail/arch-general/2021-March/048739.html

I know it's a sample size of 1, but I would argue that the burden of proof with regards to demonstrating any benefit now lies with those proposing this change. The argument that "well, everyone else is doing it" doesn't carry much weight in this.

One thing I forgot (apologies for the comment spam) is if we do start looking at potential benefits and where we stand to gain something from this (full disclosure: I'm very much not a compiler guy).

1. either the act of building software is faster because the compilers benefit from this (use less energy, can get critical fixes shipped faster, can build more with less and so on), or
2. the resulting software runs faster (or uses fewer resources) - who doesn't like free speed?

I would imagine, that most of the software that really stands to benefit already has optimizations for various cpus/cpu features where we would gain nothing from fiddling with the baseline flags.

So could we compile variants of the compilers with a better baseline and use that when available? This way there would be no sudden breakage for end users?

You can't do this "when available" that easily. What would people with older computers do? Impurely switch their compiler variant? Or would we compile everything twice (e.g. base + -v3)?

Here are some benchmarks from arch where the person running it kindly did a tl;dr which I am pasting here:

- there is no or negligible performance benefit of *-march=nehalem*, which corresponds to x86_64-v2,

- there is a moderate benefit of *-march=haswell* (x86_64-v3) - of around 10%-20% as compared to baseline for the tests performed

Link: lists.archlinux.org/pipermail/arch-general/2021-March/048739.html

I know it's a sample size of 1, but I would argue that the burden of proof with regards to demonstrating any benefit now lies with those proposing this change. The argument that "well, everyone else is doing it" doesn't carry much weight in this.

This was mentioned in #202526 (comment) :).
It is not sufficient though.

I would imagine, that most of the software that really stands to benefit already has optimizations for various cpus/cpu features where we would gain nothing from fiddling with the baseline flags.

So could we compile variants of the compilers with a better baseline and use that when available? This way there would be no sudden breakage for end users?

Someone needs to do the rigorous work of looking into which packages benefit the most currently, see Guix prior art on that.

(full disclosure: I'm very much not a compiler guy).

There's a bit of a self-selection problem here; the people who really care about this stuff are already building everything themselves. I build with -march= and -mcpu= set to exactly what -mnative would choose, on four different architectures.

But my nix is patched to remove the Hydra key, so I don't really care what cache.nixos.org does.

It looks like CentOS is investigating what a jump to v3 might look like: https://blog.centos.org/2023/08/centos-isa-sig-performance-investigation/

I'd be intrerested to revisit reasons of other distros for switching to x86-64-v2, e.g. RedHat. Do they apply in NixOs' case? Or are they completely unapplicable to the NixOs? It's "do it because everyone else is doing it" kind of argument, but my point is why is everyone else doing it? They stand dropping support for old hardware just like NixOs does but still they decide to go ahead. What makes NixOs's situation different from everyone else's?

AFAIK the other distros don't do it without replacement and still provide also binaries for older CPUs. This PR's proposal didn't do that. (And actually I don't think this is worth doubling build and storage costs for the x86 infra parts.)

https://www.phoronix.com/news/CentOS-ISA-Experiment-Perform
CentOS is looking at v3 baseline

As I said on Matrix, CentOS can afford to do that because CentOS has very long release cycles, so people on CentOS N-1 will remain supported for many years to come. For NixOS, the switchover would take at most 6 months, which is not nearly enough time to cut off support for all pre-2015 hardware.

Isn't there some kind of cflag to automatically create variants that use these features and decide in runtime which function sets will be used?

The examples cited here [1] have like explicit definitions but is there something that can kind of ramp up to more recent instructions?

For example, a math heavy library could generate more than one variant of loop heavy objects and some kind of logic to decide at runtime using the CPUID instruction which code variant will run.

IDK tbh if this is possible in single file binaries. The common approach for BLAS-like libraries for example is to generate a dynamic library for each variant then decide at runtime which one to dlopen.

[1] https://lwn.net/Articles/691932/

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/pre-rfc-moving-nixos-x86-64-baseline-to-x86-64-v3/35924/2

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/pre-rfc-gradual-transition-of-nixos-x86-64-baseline-to-x86-64-v3-with-an-intermediate-step-to-x86-64-v2/35924/1

Any news on this? Would really like to see a v2 baseline for Nixos.

See the discourse thread above.

To summarise:

To our current knowledge, there is no good data that proves a benefit from bumping the generic target to x86_64-v2. There might be a benefit but we don't actually know whether it exists or not. If it exits, it will be rather small though. (Note that there is a decent amount of bad data on this.)

There is no good data on how many users a bump beyond x86_64-v1 would exclude either. Anectdata suggests that there are at least some.

There are some packages where even the flawed existing data shows a significant benefit from march tuning beyond a reasonable doubt. glibc-hwcaps could be leveraged to enable such march tuning for those specific packages without excluding any users.

IMHO, a generic bump such as the one in this PR will require an RFC. Perhaps we should close this to signal this a bit clearer.

What about duplicated nixos channels (at least for stable), where one is the generic one and the other is compiled with x86-64-v3? This wouldn't deprecate any platform and it is also the way cachyos handles this.

The problem with selectively compiling packages with x86-64-v3, which could improve the most, is that if I for example compile gcc this will retrigger a recompilation of my whole system. I did not find a way around this problem.

Maybe there should be a flag to prevent such rebuild at the cost of reproducability?

We do not have the hardware to build multiple channels. You can use system.replaceRuntimeDependency if you really want to. But also, x86_64-v3 also does not improve things much, and we can likely get very close in terms of performance with some dynamic dispatch.