Get Repo tests (CPU) back to green: 113 failures from three test-harness gaps - #9344
Get Repo tests (CPU) back to green: 113 failures from three test-harness gaps#9344danielhanchen wants to merge 2 commits into
Conversation
- test_chat_autoload_failure_gate.py: stub resolvePreserveThinkingOnLoad, which chat-adapter.ts began importing into the sliced region. - test_new_chat_context_recount.py: stub findLatestUserVideoBase64, which refresh-context-usage.ts began importing (#9056), plus an import-coverage guard so the next one fails loudly instead of as 41 silent zero counts. - test_cross_platform_parity.py: make the install.sh fallback-block scan nesting-aware so the nested if #8670 added no longer ends the block early.
for more information, see https://pre-commit.ci
|
I verified all three independently rather than taking them on report, and they hold up.
The All four files together: 209 passed, 0 failed. The shape worth noting for the future is that two of these three were the same failure mode. A test that slices a region out of a TypeScript file and runs it against a hand-written preamble will break every time someone adds an import to that file, and the recount one broke silently: the unbound identifier threw inside a |

Main is red.
Repo tests (CPU)onstudio-backend-ci.ymlis at 113 failed / 8982 passed (run 32328722463, job 96305094412), which fails the required check on every open PR in the repo. This gets that job back to green so the backlog can move.Before and after
tests/studio/test_chat_autoload_failure_gate.pytests/studio/test_new_chat_context_recount.pytests/python/test_cross_platform_parity.pytests/test_python39_compatibility.pyI ran each file locally on
origin/mainand again on this branch with the same-n 4shape the job uses. The two studio files went from111 failed, 13 passedto124 passedtogether, and the parity file from1 failedto green.Root causes
Two of the three are the same failure mode. Both of those tests slice a region verbatim out of a frontend TypeScript source and run it under node against a hand-written JS preamble. The slice drops the import block, so anything the sliced source starts importing has to be re-declared in the preamble by hand. Two separate PRs added an import and did not.
1.
test_chat_autoload_failure_gate.py, 70 failureschat-adapter.tsnow callsresolvePreserveThinkingOnLoadinside the auto-load region, and the harness had no stub for it. This file already has a guard that catches exactly this and it did its job, which is why all 70 said so in plain words rather than failing as wrong-model assertions.I added the stub. The real resolver is
storedPreserveThinking ?? preserveThinkingDefaultFromLoad(resp): a stored answer from hydration or the composer toggle wins, otherwise the backend's family default applies. Nothing in the sliced region hydrates or toggles the preference, sostoredPreserveThinkingis always null there and the correct stub is exactly the default arm,Boolean(resp.supports_preserve_thinking && resp.preserve_thinking_default). A constant would have made these scenarios agree with the harness instead of with the adapter.2.
test_new_chat_context_recount.py, 41 failuresSame shape, silent version. #9056 taught
refresh-context-usage.tsto decline a recount on a video turn:findLatestUserVideoBase64is imported fromchat-adapter.ts, the harness slice drops imports, and the preamble had stubs for the image and audio siblings but not this one. So the line raised a bareReferenceErrorinsiderefreshContextUsage, whosecatchis deliberately empty because a background recount must not interrupt chat. The throw was swallowed, node exited 0, the count never happened, and every positive expectation in the file failed asassert 0 == 1or, in the two tests that assert usage before count,assert None == 62. All 41 are that one identifier. The 12 tests that stayed green are the ones assertingcounts == 0, which bail before the new line.The product change is right and stays:
toOpenAIMessageshas no video branch,/chat/count_tokens503s on video, and hashing up to 85 MB of base64 inbranchSignatureis synchronous main-thread work. I added the stub returningundefined, which is what the real helper answers when no message carries a clip, and no fixture in this file attaches one. The decline itself is still pinned bystudio/frontend/tests/pr9057-video-simulation.test.ts, which asserts the call is present in the source and that it is paid beforebranchSignature.I also added the import-coverage guard this file was missing, modelled on the one
test_chat_autoload_failure_gate.pyalready has: every namerefresh-context-usage.tsimports and the sliced body uses must be declared in the preamble, or the harness build fails saying which name is missing. Without it this class of drift is invisible by construction, since the bare catch converts any new unbound identifier into 41 mystery zeros. I checked it by deleting the new stub again: it fails immediately and namesfindLatestUserVideoBase64.3.
test_cross_platform_parity.py, 1 failureinstall.sh contains --torch-backend=auto outside the fallback block at lines: [5280]. The use at 5280 is legitimate: it is inside theelsebranch that runs when GPU detection produced no index URL, which is precisely the branch the guard exists to allow. The guard was wrong, notinstall.sh.It found the fallback block by scanning from the
GPU detection failedcomment to the next line equal tofi. That worked until #8670 added a nestedif [ -n "$_unsloth_desktop_install_spec" ]inside thecasearm that picks the desktop install spec. Its closingfiat line 5276 ended the block early, so the real fallback install one line below fell outside it.I made the scan nesting-aware: track
if/fidepth and stop at thefithat actually closes the branch. The rule is unchanged and nothing is relaxed. I checked that by inserting a--torch-backend=autoline into a primary branch ofinstall.sh, and the test still fails and points at it.4.
tests/test_python39_compatibility.py, 1 failuretest_studio_evaluated_unions_do_not_grow, 36 studio files evaluating PEP 604 unions on the 3.9 floor against a debt ceiling of 35. Already fixed on main by #9335, which putfrom __future__ import annotationsback intopath_utils. It passes on current main, so there is nothing to do here and no change in this PR for it. I am listing it so the count adds up.Notes
Nothing was skipped, xfailed, deleted or loosened.
install.shis untouched. The only changes are three test files: two missing preamble stubs whose values match what the real collaborators return, one guard taught to parse nested shell blocks, and one new assertion that makes the next missing stub fail loudly on line one instead of silently 41 times.