… test

On a worker with a cold flashinfer trtllm_comm JIT cache the two ranks
spend ~200s compiling three CUDA sources under a cross-process FileLock
(warm cache: ~20s). The parent previously joined each rank with a 120s
timeout and snapshot exitcodes the moment the second join returned, so
rank 0 -- the TCPStore owner that tears down last -- was recorded as
None and the [0, 0] assertion failed although both ranks had passed
every barrier.

- Raise the per-rank join budget to 480s to cover cold JIT compilation.
- Add a shared 90s grace join before reading exitcodes so a rank that
  is merely slow in teardown is not misclassified as hung.
- Bump the bazel timeout class from moderate to long so a genuinely
  stuck rank trips the join timeout instead of a bazel TIMEOUT.

Co-Authored-By: Claude Code <noreply@anthropic.com>

The package-copy cache in ~/.cache is keyed on package version and bazel
pip variant, but not on the interpreter. A host running mixed cp310/cp312
jobs from the same pip variant (ppu312 pool swap, 2026-10-23) lets the
first job's staged copy be imported by the other interpreter: a cp312-abi3
tvm_ffi core.abi3.so references Py_Version (3.11+) and every cp310 PPU test
then dies at flashinfer import (run 82355513, 11/11).

Add cpXY to the cache key so each interpreter stages its own copy from its
own runfiles. Existing unversioned entries simply go stale.

Co-Authored-By: Claude Code <noreply@anthropic.com>

Scratch mode mmaps every shard and faults the checkpoint into page
cache, which stays charged to the pod memcg until reclaim. For
~865GB-class checkpoints under shared-worker memory pressure, reclaim
cannot keep pace and the kernel OOM-kills ranks mid-load
(v4_pro_cp4_ep4_basic_sm100: CONSTRAINT_MEMCG with ~180GB file-rss,
2026-10).

Shed clean file pages with posix_fadvise(POSIX_FADV_DONTNEED) once a
shard has supplied a threshold of reads (default 2GB), plus a final
shed when the shard handle closes. The hint is advisory and only
evicts clean pages: a co-located rank that still needs a page simply
re-faults it from backing store, so correctness is unaffected and the
load schedule needs no cross-rank coordination. LOADER_DROP_PAGE_CACHE=0
is an emergency killswitch; LOADER_DROP_PAGE_CACHE_THRESHOLD_MB tunes
the window.

Verified: mincore shows a fully-cached file drop from 100% to 0%
resident after the shed on both xfs and overlayfs; new unit tests
cover the threshold trigger, the final shed on close, the killswitch,
and the fadvise-error fallback.

Co-Authored-By: Claude Code <noreply@anthropic.com>

moe_nvfp4_deepep_normal_dp2_sm100 flaked on cold machines: the legacy
DeepEP Buffer spin-waits a hardcoded 100s for EP peers at the first
dispatch collective ("CPU recv timeout", compiled into the deep_ep
wheel), while a fresh machine needs ~13min of cold JIT compilation on
the first forward. Under dp2 the first real request lands on one rank
while the idle rank forwards an empty batch, so asymmetric compile
work maximizes arrival skew; the waiting peer throws, the rank aborts
in NormalEngine::step, and the process manager tears the server down.
Warm reruns pass because the JIT caches are hot.

The engine already has the designed cold-start absorption: --warm_up
runs a fake prefill inside the NormalEngine constructor, before the
backend reports ready and /health goes live, so both ranks compile
the same kernel sequence symmetrically and arrive at the collective
seconds apart, not minutes. The blanket --warm_up 0 came in with the
suite consolidation (#1371) as a startup-time optimization, not a
per-case choice. Re-enable warmup for all four normal-mode DeepEP
cases (fp8 tp2, nvfp4 tp2/dp2, nvfp4 online dp2). The LL variants use
prebuilt low-latency kernels and have not flaked, so they stay at 0.
The 1600s health-wait budget covers the one-time cold compile.

Co-Authored-By: Claude Code <noreply@anthropic.com>

All smoke suites pinned --load_method explicitly (scratch or
fastsafetensors). Auto already picks fastsafetensors whenever the
checkpoint and host support it (bounded 2GB bounce buffer instead of
whole-shard mmap) and falls back to scratch otherwise; the pins only
served to bypass that choice — including v4_pro_cp4_ep4_basic_sm100
forcing scratch on a ~865GB checkpoint. Remove every per-case pin so
suites follow the default.

Co-Authored-By: Claude Code <noreply@anthropic.com>

Add rtp_llm/test/utils/gpu_health_probe.py (a GPU/node identity probe the
gpu_lock wrapper logs before running a test) and hang_watchdog.py (an
in-process progress watchdog used by device_resource.py to bound hung GPU
tests), plus the nvidia_wheel_env.sh hook that wires the probe into the
test environment.

The device lock coordinated wrapper users but not real device occupancy:
acquisition only looked at stale PIDs, so a card held by another live process
was still handed out and the test died on CUDA OOM (observed: GPU 1 with
268 MiB free while the new test itself held 14.76 GiB).  Cleanup could also
SIGTERM processes that were not ours, leaving the neighbour half-occupied.

* acquisition now requires the whole group to be idle: per card
  memory.used <= RTP_GPU_FREE_MEMORY_MIB (new, default 1024, <0 = only check
  processes) and no compute-app PID at all (missing /proc entry = zombie
  context, live PID = busy).  A dirty group is released and retried inside
  the existing acquire budget, and the timeout message reports per-card
  memory and live PIDs instead of just lock holders.
* the idle judgement happens while holding the candidate group: a dirty
  group is released and the next candidate tried.  Judging after the fact
  first-fit the same dirty card on every retry and starved the clean cards
  behind it (observed: six sm12x attention tests each burned their 1800s
  acquire budget re-locking a busy gpu0 while gpu1-7 sat idle).
* cleanup kills only the test's own process group (/proc pgrp/sid), warns
  "held by live foreign pid" and leaves it alone, and reports unreclaimable
  zombie contexts; it no longer waits 30s when only foreign/zombie holders
  remain.
* unit tests: 37 cases (21 new) covering memory-threshold rejection, the
  timeout report, "never kill a live foreign process", zombie diagnostics,
  dirty-group skip during selection, and own-group SIGTERM->SIGKILL.

test_embedding_tp4 keeps its main-branch form: no in-test hang_watchdog
marks, no watchdog env/timeout overrides — the watchdog helper lives in
rtp_llm/test/utils/ where device_resource.py uses it.

(cherry picked from commit 53eafd0, with the test_embedding_tp4 in-test
watchdog marks rolled back per 361cfc6; folded in c098196)

Co-Authored-By: Claude Code <noreply@anthropic.com>

DeepJIT (rtp-kernel's C++ disk cache) derives its cache directory from
$HOME and hard-fails when the variable is empty ("HOME environment
variable must not be empty", cache/disk.hpp). Some Bazel test actions
run without HOME in the environment, which flaked the decode cases; the
python layer exposes no cache-dir override, so DG_JIT_CACHE_DIR (the
deep_gemm knob) cannot redirect this one.

- BUILD: env_inherit HOME so Bazel runs reuse the per-user cache.
- test_xqa.py: set a writable fallback (TEST_TMPDIR, else /tmp) before
  any rtp-kernel/JIT import, for direct runs outside Bazel.

(cherry picked from commit 388a5f0)

Co-Authored-By: Claude Code <noreply@anthropic.com>

/health can be green while the /v1/models endpoint is still slow under
load (observed on a loaded worker: a single 5s timeout aborted the whole
bench run before any task executed). Wrap the probe in a bounded retry
window -- 6 attempts, 5s apart -- and only then give up, so a slow first
probe no longer terminates the run.

(cherry picked from commit 90ba252)

Co-Authored-By: Claude Code <noreply@anthropic.com>

…device

dsv4: the fp8 paged/prefill indexer tests now execute on any CUDA device and
pin which kernel produced the numbers: the shared capability predicates decide
between DeepGEMM and the Triton fallback, and every case asserts the matching
call count (DeepGEMM == 1 when the predicate is true, fallback == 1 otherwise)
instead of silently skipping on hosts without the required wheel.

dsv41: mirror the same device/block contract for its paged/prefill dispatch by
importing the dsv4 predicates (shared source, not copied). Combinations dsv41
cannot serve raise a RuntimeError carrying the device capability and block_kv,
because dsv41 has no Triton fallback; the FP4 variant gates are unchanged.

Verified on a local L20D pool: 4/4 targets PASSED with the path counters, and a
forced-predicate probe exercises the fallback branch end to end.

(cherry picked from commit 5699884)

Co-Authored-By: Claude Code <noreply@anthropic.com>

Two test-side defects fed the intermittent
`repackNormalizesInvalidFeatures[3]` failure (expected 15, got 63):

- the deferred status projection returned by `applyPreparedStatus` was
  dropped by the test, while production consumes it (wrapped as an
  `EndpointPublication` after the status version is published) — run it;
- Mockito stubbing for `seqLen`/`hitCache` could interleave with the
  registration-triggered async repack, leaving a mixed feature snapshot as
  the last write to the route projection. Wait for the repack to settle and
  stub `hitCache` before `seqLen`, so a residual mixed window collapses to
  the expected value through the clamps.

(cherry picked from commit f87bc23)

Co-Authored-By: Claude Code <noreply@anthropic.com>

The module imported cutlass_scaled_fp4_mm/scaled_fp4_quant eagerly at
import time. rtp_kernel.nvfp4 only ships in the cuda13 rtp-kernel wheel
pinned by deps/requirements_lock_torch_gpu_cuda13.txt; a stale wheel
stays importable while missing the submodule, and the CUDA linear
registration imports fp4_linear before registering the FP8 backends --
so the eager ImportError silently unregistered FP8 as well.

Guard the import instead: keep the module importable, log a warning
with the original error, and raise a clear RuntimeError from the
wrappers (naming the needed wheel) only when an FP4 kernel is actually
used.

(cherry picked from commit 20782b6)

Co-Authored-By: Claude Code <noreply@anthropic.com>

… and arch selection

The legacy host rtp-maga.cn-zhangjiakou.oss.aliyuncs.com is DNS-blackholed, so
every pinned wheel URL in deps/ and arch_config/ now points at the reachable
single-'oss' host; the cuda13_arm lock (which only had the typo host) is fixed
with it.  Same objects, same sha256, only the host changes -- verified with
HTTP 200/206 against the mirror.

(cherry picked from commit f049e7c)

Co-Authored-By: Claude Code <noreply@anthropic.com>

…s, bounded waits)

HttpClientTest asserted inside callbacks (which run on network threads, where
gtest assertions are invalid), spun in an unbounded while(true) until every
callback arrived, and raced the network thread when it checked the connection
pool right after submission.  CI run 81193999 failed on exactly that surface:
testRequestInvalidAddress asserted that submitting to an invalid address always
returns true, but sendPacketAsync() refuses the submission outright once the
async connect has already failed (closed connection), so a loaded worker failed
the ASSERT (the same test passed on the rerun, confirming the race).

Move every assertion back to the test thread, bound all waits so a lost
callback fails the test instead of hanging the gtest binary, keep the random
invalid port away from the ports the fixture bound, and assert the settled
connection-pool state after the callback instead of racing it.

Verified locally with --runs_per_test=15 on L20 (15/15 PASSED).

(cherry picked from commit 27aec4f)

Co-Authored-By: Claude Code <noreply@anthropic.com>

Only the H20 cases whose purpose is DeepEP itself are retired:
moe_deepep_continuous_dp2, moe_deepep_ll_tp2, moe_deepep_normal_tp2 and
next_cudagraph_deepep.  Each removal points at the unit tests that carry the
coverage instead: //rtp_llm/models_py/distributed/test:deepep_test (normal /
low-latency / expert alignment on 2xH20) plus the DeepEP auto-configuration
matrix in rtp_llm/test/auto_configure_deepep_test.py.

next_mtp_cudagraph_deepep is kept on purpose as the standing sentinel: besides
the DeepEP low-latency dispatch it also pins MTP + CUDA graph + continuous
batching on the same config, which the unit tests do not cover.  Cases that use
DeepEP as one part of broader coverage (mla_cp_pd, mla_pure_cp_pd, moe_headwise,
moe_fp8pt_load_quant, moe_cp_pd, next_mtp_pd_reuse, next_bf16_tp2_dp2[_pd],
qwen35_moe_vl_fp8) also stay, and no other hardware lane is touched.

(cherry picked from commit 11243f7)

Co-Authored-By: Claude Code <noreply@anthropic.com>

…r median

The per-bucket floor is tight by design (0.87 floor vs ~0.85 typical), so a
single machine-wide jitter spike on a shared CI worker flips it: run
81910697 failed T=16 at ratio=0.877, run 81589806 failed T=16/1024 the same
way. Gated buckets (T=16/64/256/1024) now run the 7-round paired benchmark
twice and keep the better median -- a real slowdown misses the floor in
both windows and still fails, while one jittered window no longer does.

Co-Authored-By: Claude Code <noreply@anthropic.com>
(cherry picked from commit a6a60fc)

Production-code restoration from old github-main commit cf8e489; the
accompanying harness self-tests under rtp_llm/test/smoke/test/ are
deliberately not restored (dropped by 361cfc6).

Co-Authored-By: Claude Code <noreply@anthropic.com>

Production-code restoration from old github-main commit 30942b7; the
accompanying harness self-test is deliberately not restored (dropped by
361cfc6).

Co-Authored-By: Claude Code <noreply@anthropic.com>

… KV deadline)

PD cases could start issuing queries before the role's model RPC ports were
accepting connections, and the MLA PD configs could expire the cold-start KV
load window on a busy worker.  Wait for the model RPC ports inside
SeparatedCaseRunner and give the MLA PD cases a cold-start KV load deadline
instead of the generic one.

Production-code restoration from old github-main commit de9893c; the
accompanying harness self-test is deliberately not restored (dropped by
361cfc6).

Co-Authored-By: Claude Code <noreply@anthropic.com>

…lashInfer eviction, startup heartbeat

v4_pro failed run 82762052 with the 1800s watchdog killing a still-progressing
run: 664s vanished before the first Package Setup log (eviction lock waits),
then the ~865GB checkpoint cold-read at degraded NAS throughput blew the
remaining budget. Historical real-run max is already 1441s, leaving only 360s
of headroom on the default watchdog.

- suites_cuda13.bzl: give v4_pro RTP_TEST_WATCHDOG_SECONDS=3600 via test_envs
  (reaches the wrapper process env; case envs only reach the engine).
- jit_sys_path_setup: _evict_dangling_flashinfer_ops lock wait 300s -> 3s so a
  busy op is skipped instead of silently stalling test startup; add scan/start/
  finish logs with counts and elapsed so the pre-test gap is attributable.
- util.wait_sever_done: INFO heartbeat every 60s with elapsed/timeout; during
  a long checkpoint load the port stays connection-refused and every poll was
  DEBUG-only, so the console showed nothing for up to 1600s.

Co-Authored-By: Claude Code <noreply@anthropic.com>

…e in tests

PrefillBatchTraceTest.FetchNotFoundCreatesRealServerSpanWithTransportStatus
flaked in run 82791631 (ut-sm8x): the test took exactly 5000ms and
TelemetryRuntime::shutdown(5000) returned false while the runtime was still
making progress.

Root cause is structural, not load: TelemetryConfig.schedule_delay_ms defaults
to 5000, exactly equal to the shutdown deadline these tests assert on. The
pinned OTel-cpp BSP notifies its worker without holding cv_m, so a flush
request can miss the wakeup window; recovery then costs one full
schedule-delay tick. With tick == deadline, a single lost wakeup (or one
5s scheduling stall) consumes the entire budget and the bounded shutdown
reports failure.

Set schedule_delay_ms = 200 in every test fixture that asserts shutdown(5000):
a lost wakeup is then recovered at the next 200ms tick with 25x margin.
telemetry_test.cc's fast-flush fixtures already used 1ms; this covers the
remaining default-delay sites in telemetry_test.cc, grpc_propagation_test.cc,
phase_span_synthesizer_test.cc, PrefillBatchRpcServerTest.cc and
RpcWriterCancellationTest.cc.

Co-Authored-By: Claude Code <noreply@anthropic.com>

The ARM PD decode role pinned DG_JIT_CACHE_DIR to a per-test tmpdir
(introduced in bcf4865 with the MegaKernel decode integration), which
made its deep_gemm cache structurally cold on every run: the tmpdir is
deleted at test end, and the prefill role of the same case already used
the framework default, so the decode engine re-compiled the same dsv41
kernels the prefill role had just compiled into $HOME/.deep_gemm.

The smoke framework already handles deep_gemm exactly like flashinfer
and triton: setup_jit_cache() configures DG_JIT_CACHE_DIR via
configure_writable_cache_env() to the pod-persistent $HOME/.deep_gemm,
the wrapper env propagates into engine subprocesses through
os.environ.copy(), and deep_gemm cache entries are keyed by full kernel
signature so sharing the directory across roles and cases is
collision-free. Removing the override restores that unified path.

Co-Authored-By: Claude Code <noreply@anthropic.com>