…geohot p2p mod

Port of tinygrad's 565.57.01-p2p patch (originally by geohot, simplified by
aikitoria) to 610.57.04. The 610 driver already contains NVIDIA's native
BAR1 P2P implementation end-to-end (nv_gpu_ops external PTE building with
SYS_NONCOH aperture + BAR1 DMA address encoding, UVM_GPU_LINK_PCIE_BAR1 in
nvidia-uvm, PCIE_BAR1 connection type in p2p_api), so the original patch's
manual PTE/aperture hacks in nv_gpu_ops.c and gmmu_fmt.c are no longer
needed and are dropped. What remains force-enables the native path on
GeForce boards:

- nv-pci.c: always attempt resizable BAR1 (ignore NVreg_EnableResizableBar),
  so BAR1 covers all of FB and static BAR1 mapping auto-enables.
- kernel_bif.c: default p2pOverride to 0x11 (force P2P read/write caps,
  bypassing the chipset allowlist) and default pcieP2PType to
  NV_REG_STR_RM_PCIEP2P_TYPE_BAR1. The old
  NV_REG_STR_RM_FORCE_P2P_TYPE_BAR1P2P constant no longer exists in 610;
  RMPcieP2PType=BAR1 is its replacement and is what
  kbusIsPcieBar1P2PMappingSupported_GH100 checks. This also prevents
  mailbox P2P BAR1 allocation, which would otherwise conflict.
- g_kern_bus_nvoc.c: point the default (pre-Hopper) HAL entries for the
  five BAR1 P2P functions at the GH100 implementations, which are chip-
  independent (static BAR1 memdescs + IOMMU mappings). Covers GA102 (3090)
  and AD102 (4090); GB202 (5090) already uses the GH100 entries natively.
- kern_bus_gp100.c: dispatch _PCIE_BAR1 connection type to the BAR1 P2P
  create/remove HALs, mirroring kbusCreateP2PMapping_GH100.

Requires resizable BAR enabled in SBIOS. GPUs are still expected to
require the large-BAR vBIOS/board support that ReBAR provides.

Original-patch-by: George Hotz <geohot@comma.ai>
Simplified-by: aikitoria <aikitoria@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DmU5QfcAq8kxqGGCu6BucR

BAR1 must cover the framebuffer for NCCL's persistent peer mappings; a
256 MiB BAR1 shows an all-OK P2P matrix but hangs NCCL at the first
collective. Server firmware without a Resizable BAR toggle needs the
MMIO aperture raised and the GPUs re-enumerated: resize-bar1.sh programs
the ReBAR control register, drops the GPU root-port subtrees, rescans,
and reloads the driver (with a systemd unit, since the register resets
each boot). Also documents iommu=pt and NCCL_P2P_LEVEL=SYS with measured
8x3090 numbers: 24.7 GB/s all-reduce busbw vs 2.7 GB/s SHM.

Added an image to the README for better visual context.

…8x 3090

+36% single-stream and +58% at concurrency 8, same model and config,
driver and setup steps the only change. Collective microbenchmarks move
below the serving table as the explanation rather than the headline.

…ain audit

Audited the entire BAR1 P2P chain for chip dependence:
- kbusIsPcieBar1P2PMappingSupported_GH100: regkey + static BAR1 + no
  mailbox conflict, no chip checks.
- kbusCreateP2PMappingForBar1P2P_GH100 / Remove: refcounting + IOMMU
  mappings only, no Hopper register access.
- p2p_caps (_kp2pCapsGetStatusOverPcieBar1) and UVM
  (UVM_GPU_LINK_PCIE_BAR1, SYS/SYS_NONCOH aperture): chip-independent.
- Static BAR1 (kbusIsStaticBar1Supported/Enable/Disable) is implemented
  as _TU102 for Turing and newer; pre-Turing gets stubs, but pre-Turing
  can't run the open (GSP) modules anyway.

Conclusion: the existing reroute of the default HAL entries plus the
GP100 dispatch already wires every open-driver-supported pre-Hopper die.
The only per-card gate is hardware: BAR1 must cover the framebuffer, so
Turing needs a strap-level ReBAR unlock (e.g. NVStrapsReBar) since its
vBIOS predates Resizable BAR. Documented all of this in the README.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016b4qnyRqi83vakLKPFgmxR

…MemoryPoolRetainMB)

Since 590 the open module keeps freed system pages in per-node, per-order
pools and only the kernel's memory-pressure shrinker gives them back. On a
unified-memory GPU (GB10 / DGX Spark: every device allocation is system
memory) a process that exits leaves its whole working set in the pools,
invisible in /proc/meminfo and subtracted from MemAvailable. Measured on
stock 610.43.02: MemAvailable 107.0 GiB -> 51.7 GiB ten seconds after a
64 GiB CUDA process exited, with no process in nvidia-smi. User-space
free-memory checks (vLLM startup) fail until someone runs drop_caches.

Add nv_trim_page_pools(retain_pages): walk pools from the largest order
down, keep entries until the running total reaches the watermark, free the
rest through a new nv_mem_pool_reclaim() helper that the shrinker scan now
shares (dirty list first, then clean, mutex dropped before freeing, WARN_ON
if the count would go negative). The scrubber's in-flight entry is left
alone and picked up by a later trim or the shrinker; nv_mem_pool_destroy
is untouched.

NVreg_SystemMemoryPoolRetainMB (default 0) is the watermark in MiB;
0xFFFFFFFF disables trimming and keeps the old behaviour. The trim runs
from nvidia_close_callback (after ldata_lock is released, non-removal
branch) and from nvidia_ctl_close: RM clients and their allocations belong
to the control file, so a process that closes its device fd first would
otherwise be trimmed before its pages reached the pools. Both sites are
sleepable, hold no driver lock, and can only run while a file is open, so
never during module teardown.

Spec: drain-on-last-client-close.md. Acceptance runs are staged in
/home/eric/spark-driver-ab (blocked on MOK enrolment on the Spark box).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012CyNUyiJGhDRELovGpv8kH

On an integrated (ZERO_FB) GPU such as GB10 the residency chosen for an ATS
fault is always a CPU NUMA node, yet service_ats_requests() still routes
the range through uvm_migrate_pageable(): migrate_vma_setup() plus one
order-0 alloc_pages_node() per 4 KiB page. That can never form a
transparent huge page and costs about 9 us per page.

Measured on DGX Spark (610.43.02, kernel 6.17, THP=madvise), identical CUDA
kernels, 4-8 GiB anonymous mmap buffers:

  GPU first touch, THP off or MADV_HUGEPAGE:  0.43 GiB/s, 0 MiB THP,
      then 16 M random 4K-page accesses/s (cudaMalloc pool: 3515 M/s)
  CPU first touch with MADV_HUGEPAGE:         28 GiB/s, fully THP-backed,
      then 208 M random accesses/s from the GPU

The faults are serviced by UVM through the replayable fault buffer (4095
interrupts on the UVM line for a 2 GiB fault-in, none on the SMMU).

When servicing faults on an integrated GPU with CPU residency, populate the
range with uvm_populate_pageable_vma() (same permissions and flags the
migrate path used) and return. handle_mm_fault() honours THP, so the first
touch of a THP-eligible 2 MiB region fills the PMD. Access-counter
servicing keeps the migration path. Builds on 610.43.02 and 610.57.04.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012CyNUyiJGhDRELovGpv8kH

…d HAL defaults

The fork had commented out the NVreg_EnableResizableBar==0 check in
nv_resize_pcie_bars(), leaving a documented, parsed module parameter that
did nothing. Restore the check and flip the parameter's default to 1
instead, so behaviour is unchanged by default and =0 works again.

The five rerouted BAR1-P2P entries in g_kern_bus_nvoc.c have identical
if/else arms (both _GH100). That is intentional -- every pre-Hopper die
gets the chip-independent GH100 body instead of the NOT_SUPPORTED stubs --
but the generated file ships no generator, so say so at each site for
whoever rebases.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012CyNUyiJGhDRELovGpv8kH

…e BAR1 resize safe

install.sh: set -e first; build before unloading so a build failure leaves
the running driver alone; on a Secure Boot box sign every .ko with the MOK
and abort before unloading if there is no key or it is not enrolled
(unsigned modules are refused at boot and the box comes up GPU-less);
abort if nvidia is still loaded instead of installing over it; verify the
running version equals the build before printing nvidia-smi.

tools/resize-bar1.sh: enumerate every function under each GPU root port and
abort (reloading the driver at the old BAR size) if a non-NVIDIA device
would be removed with the subtree, unless RESIZE_BAR1_FORCE=1; after the
rescan verify BAR1 was assigned from sysfs and dmesg and exit non-zero if
the resize did not take; document that fuser -k kills a display server.

tools/nvidia-resize-bar1.service: order before nvidia-persistenced, the
display manager and multi-user.target, after systemd-modules-load, wanted
by sysinit.target; note that GPU consumers must declare After= themselves.

README: 0x11 leaves P2P atomics disabled (ForceP2P=0x211 to allow); the
new install.sh flow; MOK enrolment.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012CyNUyiJGhDRELovGpv8kH

Deep dive of the driver on DGX Spark (GB10): what the kernel modules
control during inference and what each lever measured; the two findings
with fixes (ATS fault servicing in nvidia-uvm, page-pool retention after
client exit); the P2P fork review and what was fixed; the A/B kit staged in
/home/eric/spark-driver-ab and the MOK enrolment it is blocked on.

drain-on-last-client-close.md is the request the pool-trim commit
implements, kept with the tree for the acceptance table and the upstream
prior-art notes.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012CyNUyiJGhDRELovGpv8kH

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012CyNUyiJGhDRELovGpv8kH

Stock -> patched -> stock with only nvidia-uvm swapped: GPU first touch of
system-allocated memory 0.42 -> 1.19 GiB/s at 4 KiB and 0.44 -> 19.6 GiB/s
under MADV_HUGEPAGE (now fully THP-backed, was 0), random access afterwards
17 -> 207 M acc/s. Controls flat, reversal leg matches stock.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VE6ghqCDH3YhRrNRXxHcXS

…ntinel

All acceptance tests from drain-on-last-client-close.md pass on 610.43.02
built with the patch: MemAvailable returns to within 0.1 GiB ten seconds
after a 64 GiB process exits (stock leaves 64 GiB unaccounted), the 4096 MiB
watermark retains 4.3 GiB, -1 restores the old behaviour, pools-disabled
composes, and 20 allocate/free/exit cycles against a holder process and a
host memory hog produce no NVRM warnings.

NVreg_SystemMemoryPoolRetainMB is parsed as a signed int, so the disable
sentinel is written as -1 on the module command line; 0xFFFFFFFF is
rejected with ERANGE. Say so in nv-reg.h and the README.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RGvBozT9aXajqz77TzAddQ

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RGvBozT9aXajqz77TzAddQ

…ged source

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UyfBF9hSFwkjT12ux8Ry8k