…ndices

_get_parent_visible_gpu_spec() gave up as soon as a CUDA_VISIBLE_DEVICES
token failed int() parsing, returning numeric_ids=None for any UUID mask
("GPU-<uuid>,..."). Every downstream consumer -- the per-model GPU picker
chief among them -- treats numeric_ids=None as "cannot resolve physical
IDs" and falls back to relative torch ordinals, which the frontend never
marks pinnable. A UUID mask is a legitimate, common NVIDIA configuration
(stable device ordering across PCIe reordering), so this silently hid the
picker on healthy multi-GPU CUDA hosts with no indication why.

Add nvidia.resolve_gpu_uuid_mask(), which maps a UUID mask to physical
indices via `nvidia-smi --query-gpu=index,uuid`, preserving mask order
since it defines the visible-ordinal mapping downstream. Wire it into the
CUDA branch of _get_parent_visible_gpu_spec() only -- ROCm hosts have no
nvidia-smi. Anything nvidia-smi can't resolve (a MIG instance UUID,
nvidia-smi missing entirely) keeps the previous unresolved behaviour
unchanged, so this is additive rather than a behavior change for the
cases that were already failing closed.

- Only trust nvidia-smi's UUID->index resolution when CUDA_DEVICE_ORDER
  is actually PCI_BUS_ID (the app's own default). nvidia-smi always
  numbers by PCI bus id regardless of CUDA_DEVICE_ORDER; if something
  upstream opted back into FASTEST_FIRST, index N from nvidia-smi and
  index N under FASTEST_FIRST can be different physical cards, so a
  resolved index handed back as explicit-selectable could select the
  wrong GPU once written to CUDA_VISIBLE_DEVICES for a worker.
- Resolve unambiguous UUID prefixes (e.g. "GPU-abcdef12"), which NVIDIA
  also accepts as a device identifier, not just full UUIDs. A prefix
  shared by more than one device is rejected rather than guessed at.
- Cache resolution results (including failures) per token tuple, so a
  missing or hung nvidia-smi doesn't pay its timeout on every poll --
  _get_parent_visible_gpu_spec() can be called multiple times per
  /api/system refresh.
- Update the gpu_ids field descriptions in models/inference.py and
  models/training.py, which still said explicit selection was
  unsupported for any UUID mask; a resolvable one is now supported.

- Derive the PCI-bus-order ordinal by sorting nvidia-smi's own
  pci.bus_id column ourselves, instead of trusting its index column to
  already match CUDA's PCI-bus enumeration. NVIDIA's own docs note
  device enumeration ordering isn't guaranteed consistent and
  recommend UUID or PCI bus ID for that; index was one bug away from
  being exactly the kind of value that guidance is warning about.
- Leave a MIG-enabled root GPU's UUID out of the resolution map. CUDA
  exposes MIG instances rather than the whole card in that case, so
  resolving the root UUID to a normal selectable index would silently
  pick a partitioned view of it.
- Resolve mixed numeric+UUID masks (e.g. "0,GPU-<uuid>"): numeric
  tokens pass through at the same trust level as the pre-existing
  numeric-only fast path, UUID tokens resolve as before. Previously
  any UUID token in the mask made int() parsing fail on the *whole*
  token list, but only pure-numeric or pure-UUID masks actually
  reached resolution.
- Require a prefix-matched token to actually look like "GPU-<hex>"
  before attempting the match, so a malformed token like "G" or "GPU"
  can't coincidentally string-prefix a real UUID on a single-GPU host.
- Cache failed resolutions for 30s instead of the process lifetime.
  The picker polls every 5s (floating-monitor.tsx); a transient
  nvidia-smi hiccup at Studio startup would otherwise hide the picker
  until restart. Successful resolutions still cache indefinitely --
  GPU topology doesn't change mid-process.

…etry probes

get_visible_gpu_utilization() and get_backend_visible_gpu_info() both key
their own nvidia-smi rows by its index column, on the codebase-wide
assumption (stated in hardware.py's "GPU index ordering" comment) that
nvidia-smi's index already matches PCI bus order. The prior commit derived
an independent ordinal by sorting pci.bus_id, which is safer in isolation
but could disagree with that index on a host where the assumption doesn't
hold -- silently attaching another card's memory/utilization to a
resolved UUID.

Verify the assumption instead of computing around it: sort by pci.bus_id
and confirm every row's index already equals its position in that sort.
If it does, resolve to nvidia-smi's own index -- consistent with every
other index-keyed probe in this file. If it doesn't, the invariant this
file leans on doesn't hold on this host and no ordinal from either source
can be trusted to line up with the others, so decline to resolve rather
than guess.

…n a TTL

_visible_ordinal_map() is keyed by physical ID, so a mask that resolves
to a repeated or aliased ID (e.g. "0,GPU-<uuid-of-0>") would collapse one
of the mask's distinct visible ordinals onto the other rather than giving
it its own device entry downstream. Reject any resolution containing
duplicate physical IDs instead.

Successful resolutions were being cached for the process lifetime.
Topology/MIG mode isn't provably immutable for a whole Studio session --
an admin can enable MIG on a previously plain card without a restart --
so a stale success could keep advertising a card as directly selectable
after that changes. Successes now revalidate on a 300s TTL; failures keep
their existing, much shorter 30s TTL.

MIG-enabled roots were excluded from the resolution map before prefix
ambiguity was checked, so a prefix shared by a normal GPU and a
MIG-enabled root would see only the normal GPU as a candidate and
resolve to it -- an otherwise-ambiguous prefix silently resolving anyway,
defeating the intended fail-closed behavior. Ambiguity is now checked
against every root UUID regardless of MIG state; only the final matched
UUID's own MIG state decides whether it's a valid resolution.

Real CUDA_VISIBLE_DEVICES semantics truncate enumeration at the first
invalid member of a mixed mask (negative, or not a real device) --
everything listed after it is hidden from the CUDA runtime entirely, per
NVIDIA's docs (e.g. "0,2,-1,1" makes only 0 and 2 visible). A numeric
token here was accepted purely on int() parsing, with no check against
what nvidia-smi actually reported, so a later UUID token in the same mask
could resolve to a GPU the real CUDA semantics would have hidden.

Rather than replicate CUDA's exact truncation point (which would mean
returning a shorter list than the mask, forcing every downstream consumer
to special-case it), a negative or unrecognized numeric member now fails
the whole resolution -- falling back to relative ordinals is safer than
resolving a UUID that real parsing would never have exposed.

valid_indices for the mixed-mask numeric path included every queried GPU
index, MIG-enabled roots included, even though UUID resolution
deliberately excludes those same roots. A mixed mask could route around
the MIG protection just by spelling the index instead of the UUID (e.g.
"0,GPU-<other-uuid>" where GPU 0 is MIG-enabled). valid_indices now
excludes MIG-enabled indices too, so both forms are rejected the same way.

Python's int() accepts spellings CUDA's own decimal parser wouldn't --
most notably PEP 515 digit-group underscores (int("0_0") == 0). A mixed
mask member like "0_0" would silently normalize to 0 and pass the
existing range/MIG checks, when real CUDA_VISIBLE_DEVICES parsing would
treat it as an invalid member and truncate enumeration there instead.
Numeric tokens are now required to match a plain decimal shape (ASCII
digits, optional leading "-") before conversion, so a malformed member
fails closed the same way an out-of-range or MIG-enabled index already
does, rather than being silently coerced into a valid-looking value.

Also fixes a copy-paste bug from the previous commit: the UUID-token
branch was appending the leftover numeric_idx from the prior loop
iteration instead of its own resolved idx.

…escriptions

The gpu_ids field descriptions said any nvidia-smi-resolvable root UUID
mask was supported, which was accurate after the first review round but
went stale once later rounds added the CUDA_DEVICE_ORDER=PCI_BUS_ID guard
and the nvidia-smi-index-vs-PCI-order consistency check -- a client
following the documented contract could submit gpu_ids for a UUID mask
that's actually still unsupported under FASTEST_FIRST or a mismatched
host, and get a validation error the docs didn't warn about.