Get Repo tests (CPU) back to green: 113 failures from three test-harness gaps by danielhanchen · Pull Request #9344 · unslothai/unsloth · GitHub
Skip to content

Get Repo tests (CPU) back to green: 113 failures from three test-harness gaps - #9344

Closed
danielhanchen wants to merge 2 commits into
mainfrom
fix-repo-tests-cpu
Closed

Get Repo tests (CPU) back to green: 113 failures from three test-harness gaps#9344
danielhanchen wants to merge 2 commits into
mainfrom
fix-repo-tests-cpu

Conversation

@danielhanchen

@danielhanchen danielhanchen commented Aug 20, 2026

Copy link
Copy Markdown
Member

Main is red. Repo tests (CPU) on studio-backend-ci.yml is at 113 failed / 8982 passed (run 32328722463, job 96305094412), which fails the required check on every open PR in the repo. This gets that job back to green so the backlog can move.

Before and after

before after
tests/studio/test_chat_autoload_failure_gate.py 70 failed, 1 passed 71 passed
tests/studio/test_new_chat_context_recount.py 41 failed, 12 passed 53 passed
tests/python/test_cross_platform_parity.py 1 failed, 76 passed 77 passed
tests/test_python39_compatibility.py 1 failed already fixed on main by #9335
total 113 failed 0 failed

I ran each file locally on origin/main and again on this branch with the same -n 4 shape the job uses. The two studio files went from 111 failed, 13 passed to 124 passed together, and the parity file from 1 failed to green.

Root causes

Two of the three are the same failure mode. Both of those tests slice a region verbatim out of a frontend TypeScript source and run it under node against a hand-written JS preamble. The slice drops the import block, so anything the sliced source starts importing has to be re-declared in the preamble by hand. Two separate PRs added an import and did not.

1. test_chat_autoload_failure_gate.py, 70 failures

chat-adapter.ts now calls resolvePreserveThinkingOnLoad inside the auto-load region, and the harness had no stub for it. This file already has a guard that catches exactly this and it did its job, which is why all 70 said so in plain words rather than failing as wrong-model assertions.

I added the stub. The real resolver is storedPreserveThinking ?? preserveThinkingDefaultFromLoad(resp): a stored answer from hydration or the composer toggle wins, otherwise the backend's family default applies. Nothing in the sliced region hydrates or toggles the preference, so storedPreserveThinking is always null there and the correct stub is exactly the default arm, Boolean(resp.supports_preserve_thinking && resp.preserve_thinking_default). A constant would have made these scenarios agree with the harness instead of with the adapter.

2. test_new_chat_context_recount.py, 41 failures

Same shape, silent version. #9056 taught refresh-context-usage.ts to decline a recount on a video turn:

if (findLatestUserVideoBase64(runMessages)) return;

findLatestUserVideoBase64 is imported from chat-adapter.ts, the harness slice drops imports, and the preamble had stubs for the image and audio siblings but not this one. So the line raised a bare ReferenceError inside refreshContextUsage, whose catch is deliberately empty because a background recount must not interrupt chat. The throw was swallowed, node exited 0, the count never happened, and every positive expectation in the file failed as assert 0 == 1 or, in the two tests that assert usage before count, assert None == 62. All 41 are that one identifier. The 12 tests that stayed green are the ones asserting counts == 0, which bail before the new line.

The product change is right and stays: toOpenAIMessages has no video branch, /chat/count_tokens 503s on video, and hashing up to 85 MB of base64 in branchSignature is synchronous main-thread work. I added the stub returning undefined, which is what the real helper answers when no message carries a clip, and no fixture in this file attaches one. The decline itself is still pinned by studio/frontend/tests/pr9057-video-simulation.test.ts, which asserts the call is present in the source and that it is paid before branchSignature.

I also added the import-coverage guard this file was missing, modelled on the one test_chat_autoload_failure_gate.py already has: every name refresh-context-usage.ts imports and the sliced body uses must be declared in the preamble, or the harness build fails saying which name is missing. Without it this class of drift is invisible by construction, since the bare catch converts any new unbound identifier into 41 mystery zeros. I checked it by deleting the new stub again: it fails immediately and names findLatestUserVideoBase64.

3. test_cross_platform_parity.py, 1 failure

install.sh contains --torch-backend=auto outside the fallback block at lines: [5280]. The use at 5280 is legitimate: it is inside the else branch that runs when GPU detection produced no index URL, which is precisely the branch the guard exists to allow. The guard was wrong, not install.sh.

It found the fallback block by scanning from the GPU detection failed comment to the next line equal to fi. That worked until #8670 added a nested if [ -n "$_unsloth_desktop_install_spec" ] inside the case arm that picks the desktop install spec. Its closing fi at line 5276 ended the block early, so the real fallback install one line below fell outside it.

I made the scan nesting-aware: track if/fi depth and stop at the fi that actually closes the branch. The rule is unchanged and nothing is relaxed. I checked that by inserting a --torch-backend=auto line into a primary branch of install.sh, and the test still fails and points at it.

4. tests/test_python39_compatibility.py, 1 failure

test_studio_evaluated_unions_do_not_grow, 36 studio files evaluating PEP 604 unions on the 3.9 floor against a debt ceiling of 35. Already fixed on main by #9335, which put from __future__ import annotations back into path_utils. It passes on current main, so there is nothing to do here and no change in this PR for it. I am listing it so the count adds up.

Notes

Nothing was skipped, xfailed, deleted or loosened. install.sh is untouched. The only changes are three test files: two missing preamble stubs whose values match what the real collaborators return, one guard taught to parse nested shell blocks, and one new assertion that makes the next missing stub fail loudly on line one instead of silently 41 times.

danielhanchen and others added 2 commits August 20, 2026 04:10
- test_chat_autoload_failure_gate.py: stub resolvePreserveThinkingOnLoad,
  which chat-adapter.ts began importing into the sliced region.
- test_new_chat_context_recount.py: stub findLatestUserVideoBase64, which
  refresh-context-usage.ts began importing (#9056), plus an import-coverage
  guard so the next one fails loudly instead of as 41 silent zero counts.
- test_cross_platform_parity.py: make the install.sh fallback-block scan
  nesting-aware so the nested if #8670 added no longer ends the block early.
@danielhanchen

Copy link
Copy Markdown
Member Author

I verified all three independently rather than taking them on report, and they hold up.

test_new_chat_context_recount.py at main's version: 41 failed, 12 passed. With this change: 53 passed. Removing the findLatestUserVideoBase64 stub while keeping the new import-coverage guard makes it fail with the identifier named, which is the point of adding the guard at all:

AssertionError: refresh-context-usage.ts imports ['findLatestUserVideoBase64'] and the
sliced body uses them, but HARNESS_PRELUDE does not define them.

The install.sh scan still catches a genuine out-of-block use. I inserted a --torch-backend=auto line into a primary branch and the test failed as it should; restored afterwards.

All four files together: 209 passed, 0 failed.

The shape worth noting for the future is that two of these three were the same failure mode. A test that slices a region out of a TypeScript file and runs it against a hand-written preamble will break every time someone adds an import to that file, and the recount one broke silently: the unbound identifier threw inside a try whose catch is deliberately bare, so the count never happened and 41 tests failed as assert 0 == 1 rather than saying what was missing. The new guard turns that into a first-line failure that names the identifier, which is why it is worth more than the stub it sits next to.

@danielhanchen

Copy link
Copy Markdown
Member Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant