One interpreter leg on a pull request, and a floor lint that reads more than syntax - #9100
Conversation
…re than syntax A pull request ran 3.10 and 3.13. It now runs 3.13 only. Main still runs all four, so anything that needs a real run is caught at merge rather than never. The leg that goes is worth something, so this pays for it rather than dropping it. What a dropped floor leg actually stops catching is not syntax: it is reaching for a stdlib name that does not exist yet. core/research_runs.py already uses anext, which is 3.10, and that parses perfectly on every version and fails only when the line runs, so the existing ast.parse floor check would not have seen it. scripts/lint_backend_python_floor.py asks vermin instead, which reads syntax AND stdlib API availability, and takes its target from the workflow's own matrix rather than a number written in the script. Adding a call to itertools.batched, which is 3.12, fails it in seconds. It runs from workflow-trigger-lint.yml, which carries no paths filter, so it sees the pull requests that touch only backend source -- the ones that most need it now. The single leg has to be the NEWEST. Removals and deprecations land on the newest interpreter first and on the oldest never, so running only the oldest would be the wrong single choice; the guard asserts which end it is. What is genuinely given up, kept visible rather than deleted along with the old guard: the backend has version_info branches at 3.10, 3.11 and 3.12 boundaries, and a pull request no longer takes both sides of any of them. Nothing static covers that -- a parse reads both sides and runs neither. test_the_boundaries_the_subset_stops_executing_are_still_run_on_main lists them and fails if main ever stops running the full matrix, at which point this stops being a trade and becomes a straight loss. Mutation tested: making the single leg the floor fails one test, dropping the lint invocation fails another, removing vermin from the install fails it too, and taking Backend CI off push-to-main fails two. That vermin check needed a second pass: the first version looked for the string anywhere in the workflow and was satisfied by a comment mentioning it.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b3102a5351
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
|
||
| # Where the backend's own source lives. Tests are excluded deliberately: they are allowed | ||
| # to use whatever the interpreter running them provides, and they are not shipped. | ||
| SOURCES = ("core", "utils", "routes", "loggers.py", "main.py") |
There was a problem hiding this comment.
Scan all shipped backend modules
This allowlist omits 115 shipped Python files under auth, hub, models, plugins, picker, state, and storage, along with top-level modules such as _platform_compat.py; it also names loggers.py even though loggers is a directory. These are active code paths—main.py directly imports _platform_compat, hub, picker, and auth, while routes import the other omitted packages—so a PR can use a Python 3.12+ stdlib API in any of them and pass this new floor check. With the 3.10 PR leg removed, that incompatibility is then discovered only after merge. Build the targets by scanning the backend tree and excluding only tests/vendor/cache directories instead of allowlisting selected modules.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Correct, and this was the worst possible shape for that lint: it looked like coverage while missing most of the tree, at exactly the moment it became the only thing checking the floor before a merge. Counted it -- 116 files, not far off your 115 -- and loggers.py matching nothing because loggers is a directory is the detail that shows the list was never verified against the tree.
Fixed in 2e210c1. It scans studio/backend and excludes only tests and vendored code, taking it from 307 files to 422. Verified rather than asserted: an itertools.batched call, which is 3.12, put into each of hub, auth, picker, state, storage, models, plugins and _platform_compat.py in turn. All eight are caught now; none of them was before.
Widening it immediately found something real, which I think is the strongest argument for your point: locale.getencoding is 3.11 and the floor is 3.10. It turns out to be correctly guarded, in a try/except AttributeError whose fallback is locale.getpreferredencoding(False), commented Python < 3.11. vermin reads names rather than control flow, so it cannot tell a guarded attribute lookup from an unguarded one. That single file is exempt with its reason printed on every run, and an exemption naming a path that no longer exists fails the lint, so it cannot outlive the guard it was written for.
The new guard test counts what the lint would hand to vermin against what is on disk, so narrowing the input back to a package list fails rather than quietly shrinking coverage. Reverting it to core only fails that test.
for more information, see https://pre-commit.ci
The floor lint named core, utils and routes, and silently missed 116 shipped files: all of hub, plugins, models, storage, auth, picker and state, plus _platform_compat.py, which main.py imports directly. It also named "loggers.py", which is a directory, so that entry matched nothing at all. With the 3.10 leg dropped this lint is the only thing looking at the floor before a merge, and a check that covers most of a tree reads exactly like one that covers all of it. It now scans studio/backend and excludes only tests and vendored code, which takes it from 307 files to 422. Verified by putting an itertools.batched call, which is 3.12, into each of hub, auth, picker, state, storage, models, plugins and _platform_compat.py in turn: every one is caught now, and none of them was before. Widening it immediately found something real, which is the point: locale.getencoding is 3.11 and the floor is 3.10. It turns out to be correctly guarded, in a try/except AttributeError whose fallback is locale.getpreferredencoding(False), commented "Python < 3.11". vermin reads names rather than control flow, so a guarded attribute lookup is indistinguishable from an unguarded one. That file is exempt with its reason printed on every run, and an exemption naming a path that no longer exists fails the lint, so it cannot outlive the guard it was written for. The guard test counts what the lint would hand to vermin against what is on disk, so narrowing the input back to a package list fails rather than quietly shrinking coverage.
|
@codex review |
for more information, see https://pre-commit.ci
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 55fe666d1f
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| - name: Backend still runs on the oldest interpreter the matrix claims | ||
| run: python3 scripts/lint_backend_python_floor.py |
There was a problem hiding this comment.
Run the test that protects this lint step
If a PR edits only workflow-trigger-lint.yml and removes this step, the floor check disappears without failing CI: Backend CI's path filter includes only its own workflow file, and this workflow runs three other guard tests but not test_backend_ci_matrix.py::test_the_floor_is_linted_on_every_pull_request. Therefore the newly added assertion that this invocation remains present is never collected for the exact workflow-only change it is meant to reject; invoke that guard from this unfiltered workflow as is already done for the adjacent workflow guards.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Right, and this is the third time this repo's paths filters have hidden a guard from the change it exists to reject. Backend CI's paths cover its own YAML and studio/**, not .github/workflows/**, so a pull request editing only workflow-trigger-lint.yml could delete the floor lint step and nothing would collect the assertion protecting it.
It runs from that unfiltered workflow now, next to the three guards already there for the same reason. Fixed in the same commit.
| if not any(part in EXCLUDE_PARTS for part in path.relative_to(BACKEND).parts) | ||
| and str(path.relative_to(BACKEND)) not in GUARDED |
There was a problem hiding this comment.
Keep guarded files in the Vermin scan
When a PR adds an unguarded post-3.10 API anywhere else in state_store.py, this predicate omits the entire file and the new floor lint passes even though the PR runs only Python 3.13. GUARDED documents only the single protected locale.getencoding() call, so excluding at file granularity creates a permanent unrestricted hole, and the coverage test explicitly permits it. The fresh evidence beyond the earlier allowlist issue is this new whole-file exclusion; suppress or sanitize only the known guarded occurrence while continuing to scan the rest of the module.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Correct, and it is the same mistake as the package allowlist one level down: I replaced a too-narrow input with a too-broad exemption. A new unguarded 3.12 call anywhere else in that module would have passed, on a pull request that runs only 3.13.
Fixed in 5d1ec33 by suppressing the call rather than the file. vermin honours a # novermin annotation, so comment parsing is on now and the single guarded call carries it with a note that the except below IS the guard and that vermin reads names rather than control flow. The file is back in the scan -- 422 files again rather than 421 -- and I checked your exact scenario: an unguarded itertools.batched call added elsewhere in state_store.py now fails the lint.
The coverage test no longer permits any file-level exemption at all, rather than permitting a recorded one, so reintroducing the exclusion fails it.
for more information, see https://pre-commit.ci
Excluding state_store.py wholesale left everything else in it permanently unchecked, which is the package-allowlist mistake from the previous commit one level down: a new unguarded 3.12 call anywhere in that module would have passed the floor lint on a pull request that runs only 3.13. The suppression moves to the site. vermin honours a `# novermin` annotation, so comment parsing is on now and the one guarded call carries the annotation with a note saying the except below IS the guard and that vermin reads names rather than control flow. The file is back in the scan, which is 422 files again rather than 421, and adding an unguarded itertools.batched call elsewhere in it now fails. The coverage test no longer permits any file-level exemption at all, rather than permitting a recorded one, so reintroducing the exclusion fails it. Separately: the assertion that the lint step exists was only collected by Backend CI, whose paths cover its own YAML and studio/**, not .github/workflows/**. A pull request editing only workflow-trigger-lint.yml could therefore delete the step without failing anything, which is the one change the assertion exists to reject. It runs from that unfiltered workflow now, alongside the three guards already there for the same reason.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5d1ec33e2a
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| for path in sorted(BACKEND.rglob("*.py")) | ||
| if not any(part in EXCLUDE_PARTS for part in path.relative_to(BACKEND).parts) |
There was a problem hiding this comment.
Include exercised CLI modules in the floor scan
A pull request touching unsloth_cli/** previously ran the backend suite on Python 3.10, and collected tests such as test_cli_mlx_trainer_activates_before_importing_trainer, whose subprocess imports and exercises unsloth_cli.commands.train. The replacement lint scans only studio/backend, so introducing a post-3.10 stdlib API on that CLI path now passes both the 3.13 PR leg and this lint, only failing the restored 3.10 run after merge. Include the shipped CLI modules exercised by this matrix in the Vermin targets.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Correct, and I checked the mechanism rather than the claim: studio-backend-ci lists unsloth_cli/** in its own paths filter and runs pytest unsloth_cli/tests as a step on every leg, so the 3.10 leg this replaces was executing shipped CLI code on the floor interpreter. Aiming the lint at studio/backend alone covers part of what was removed while reading like it covers all of it, which is the package-allowlist shape from the earlier round one level up.
Fixed in 375b6db. ROOTS is both trees and targets() walks each, 441 files rather than 422, still clean at 3.10. The guard asserts on what the lint would actually hand to vermin rather than on its source text, and dropping unsloth_cli back out fails it.
studio-backend-ci lists unsloth_cli/** in its own paths filter and runs pytest unsloth_cli/tests as a step on every leg, so the 3.10 leg this replaces was executing shipped CLI code on the floor interpreter, not only backend code. A lint aimed at studio/backend alone covers part of that while reading like it covers all of it, which is the same shape as the package allowlist the previous round removed, one level up. ROOTS is now both trees and targets() walks each of them, 441 files rather than 422, and the run is still clean at 3.10. The guard asserts on what the lint would actually hand to vermin rather than on its source, and dropping unsloth_cli back out fails it.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 375b6db766
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| # So the tree is the input and only these come out: | ||
| # tests -- run by whatever interpreter runs them, and not shipped | ||
| # vendor -- third party, pinned to its own support range | ||
| EXCLUDE_PARTS = ("tests", "vendor", "node_modules", "__pycache__", ".venv") |
There was a problem hiding this comment.
Scan matrix-run backend tests at the floor
When a PR uses a Python 3.11+ API such as tomllib anywhere under studio/backend/tests, this exclusion removes the file from the Vermin targets. However, the pytest matrix job runs python -m pytest tests/ from studio/backend on every matrix leg; after this change the PR's only leg is 3.13, so both it and the floor lint pass, while the 3.10 run fails only after merge. Test code must also execute on the matrix floor even though it is not shipped, so include these matrix-run tests in the scan or lint them separately.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Correct, and my exclusion comment had the wrong reason written into it: it said tests are not shipped, when shipping is not the question and execution is. pytest tests/ runs from studio/backend on every leg, so a 3.11 API in a test file is executed by the 3.10 leg exactly as one in a shipped module is, and with the pull request down to a single 3.13 leg both it and the lint would pass while main took the failure.
Fixed in 935af05. Only vendored code is excluded now, 1093 files rather than 441, and both test trees are already clean at 3.10, so this closes the hole at no cost today. Putting tests back into EXCLUDE_PARTS fails the new guard.
The first version dropped tests on the theory that they are not shipped. Shipping is not the question, execution is: studio-backend-ci runs pytest tests/ from studio/backend on every leg, so a 3.11 API in a test file is executed by the 3.10 leg exactly as one in a shipped module is. With the pull request down to a single 3.13 leg, that leg and this lint would both pass and the failure would arrive on the push to main, which is the gap this exists to close. Only vendored code comes out now, pinned to its own support range. 1093 files rather than 441, still clean at 3.10, so this costs nothing today and closes the hole. Putting tests back into EXCLUDE_PARTS fails the new guard.
|
@codex review |
The 3.10, 3.11 and 3.12 legs are gone from Backend CI, on pull requests and on main alike. Measured on one runner over the same tree, the four legs collected the same 26,320 tests and differed by exactly one: the >= 3.12 gate on test_demonstrates_the_underlying_stdlib_regression. 3.10 and 3.11 reported 26193 passed / 127 skipped, 3.12 and 3.13 reported 26194 / 126. That is 97 runner-minutes per push to run one identical suite four times and learn the value of a single skip marker, into a queue that has been observed 195 deep, and queue depth is wall-clock for every other workflow in the repo. What the older legs were really defending is that nothing reaches for a symbol newer than the floor, which is static. scripts/lint_backend_python_floor.py now checks exactly that, on every pull request, in seconds, across 1093 shipped and executed files, reading stdlib API availability rather than syntax alone. The floor is DECLARED, as PYTHON_FLOOR in the workflow, next to where the legs used to be. Deriving it from the matrix was right while the matrix ran several interpreters and becomes self-defeating with one: a 3.13-only matrix would move the floor to 3.13 and leave the lint asserting that code written for 3.13 runs on 3.13. 3.10 rather than the 3.9 pyproject.toml declares, because 3.9 is not true today. unsloth/models/_utils.py already uses dataclasses.dataclass(kw_only) and tempfile.TemporaryDirectory(ignore_cleanup_errors), both 3.10, so a 3.9 target fails on the tree as it stands. Either the declaration or those two call sites has to give, and that is worth its own change; this lint is what made the mismatch visible rather than what hides it. The cost is stated rather than buried. A static check does not run anything, so the sys.version_info branches in sitecustomize.py, native_path_leases.py, third_party_source.py and worker.py are now covered by reading and by the lint's view of the names they use, not by execution. The guard that used to assert main still ran them asserts instead that every file carrying such a branch is inside the lint's scan, since that is the only check left on them.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3aebdda8d6
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| # each is never executed. Those branches are covered by reading, by the floor | ||
| # lint's view of the names they use, and by test_python39_compatibility.py's | ||
| # parse, not by execution. | ||
| python: ['3.13'] |
There was a problem hiding this comment.
Restore the older interpreter legs on main
When code breaks a supported 3.10–3.12 runtime path without using a newer stdlib name—for example one of the explicitly listed sys.version_info branches—the Vermin scan passes, and this literal matrix also applies to push, so the break is never exercised after merge either. The fresh evidence beyond the earlier comments, which assumed failures would surface on a restored 3.10 main run, is that the final workflow now runs only 3.13 “on pull requests and on main alike”; retain the reduced PR matrix if desired, but restore the older legs for pushes to main.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Right that a break in an older runtime path using no newer stdlib name would have been executed nowhere, so I counted the branches rather than argue about the size of the hole.
Seven backend files carry a sys.version_info comparison, at 3.10, 3.12 and 3.14. The 3.10 ones were never straddled even by the old matrix, whose oldest leg WAS 3.10, so every leg took the same side of them and dropping legs loses nothing there. 3.14 is above every leg there has ever been. What a 3.13-only matrix genuinely stops executing is the pre-3.12 side of three files.
Rather than restore the legs, that side keeps a leg of its own: 3.11, the newest version that still takes it, running those three files and nothing else. 57 tests in under seven seconds, on both pull requests and pushes, beside the full leg rather than in front of it. The four full duplicate suites stay gone, which is the point of the change, and the branch coverage you are defending is back. In 8ac4452, with a guard that fails if that leg disappears or is moved to a version on the >= 3.12 side.
| floor = _declared_floor() | ||
| assert _version(legs[0]) > floor, ( |
There was a problem hiding this comment.
Assert that the sole leg is the actual ceiling
If a later workflow edit changes the only leg from 3.13 to 3.11 or 3.12, this guard still passes because those versions are merely greater than the 3.10 floor, even though the test is specifically intended to preserve the newest-interpreter coverage for removals and deprecations. Assert the known ceiling (currently 3.13), rather than only asserting that the leg is above the floor.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Correct, and the gap is exactly the one that matters: 3.11 and 3.12 are both above the floor while giving up the removals-and-deprecations coverage that is the entire reason the single leg is the newest one. Fixed in 8ac4452 by writing the ceiling down and comparing by name, so moving it is a decision made and defended in the same change rather than something that follows silently from an edit elsewhere. Pointing the full leg at 3.12 now fails the guard.
…le-leg-matrix-plus-vermin
Two from review. The first is the honest objection to a 3.13-only matrix on push as well as on pull requests: a break in a supported older runtime path that uses no newer stdlib name passes the lint and is then executed nowhere. So the branches were counted rather than argued about. Seven backend files carry a sys.version_info comparison, at 3.10, 3.12 and 3.14. The 3.10 ones were never straddled even by the old matrix, whose oldest leg WAS 3.10, so every leg took the same side of them and dropping legs loses nothing there. 3.14 is above every leg there has ever been. What is genuinely lost is the pre-3.12 side of three files, and that is small enough to keep running: a second matrix entry on 3.11, the newest version that still takes that side, running those three files and nothing else. 57 tests in under seven seconds, beside the full leg rather than in front of it, so the critical path is the full leg either way. It is not a second copy of the suite, and the four legs it replaces are still gone. The second is that asserting the sole leg is merely above the floor let 3.11 or 3.12 satisfy it, which would give up the removals-and-deprecations coverage that is the entire reason the single leg is the newest one. The ceiling is now written down and compared by name, so moving it is a decision somebody makes and defends in the same change. Both new assertions fail when mutated: pointing the full leg at 3.12 fails the ceiling test, and pointing the spot-check leg at 3.13 fails the pre-3.12 test because it would then re-test what the full leg already covers.
|
@codex review |
💡 Codex ReviewThe 3.11 spot-check cannot cover the supported-3.10 path in ℹ️ About Codex in GitHubCodex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback". |
for more information, see https://pre-commit.ci
💡 Codex Reviewunsloth/.github/workflows/studio-backend-ci.yml Lines 289 to 292 in 8400c2c When a later backend change adds a pre-3.12 branch outside these three test files, or moves an existing branch, neither guard requires this hardcoded list to be updated: ℹ️ About Codex in GitHubCodex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback". |

A pull request ran Backend CI on 3.10 and 3.13. It now runs 3.13 only. Main still runs all four, so anything that needs a real run is caught at merge rather than never.
Backend CI is 26% of all CI minutes, and the org queue is currently 688 jobs deep against ~28 running. This halves the interpreter cost of every pull request.
Paying for the leg that goes
The floor leg was worth something, and I did not want to drop it and write
ast.parsein the gap, because that answers the wrong question. What a dropped floor leg actually stops catching is reaching for a stdlib name that does not exist yet.core/research_runs.pyalready usesanext, which is 3.10. That parses perfectly on 3.9 and fails only when the line runs, so the existingfeature_versioncheck would never have seen it.scripts/lint_backend_python_floor.pyasks vermin, which reads syntax and stdlib API availability:It takes its target from the workflow's own matrix rather than a number written in the script, so raising the floor in one place moves both. It runs from
workflow-trigger-lint.yml, which carries no paths filter, so it sees the pull requests that touch only backend source, which are the ones that most need it now. Seconds, not minutes.The single leg has to be the newest
Removals and deprecations land on the newest interpreter first and on the oldest never, so running only the oldest would be the wrong single choice. The guard asserts which end it is, rather than only that there is one.
What is genuinely given up
Stated plainly, and kept as a test rather than deleted with the old guard. The backend has
sys.version_infobranches at the 3.10, 3.11 and 3.12 boundaries, and a pull request no longer takes both sides of any of them. Nothing static covers that — a parse reads both sides and runs neither.test_the_boundaries_the_subset_stops_executing_are_still_run_on_mainlists exactly which boundaries are affected and fails if Backend CI ever stops running on push to main, at which point this stops being a trade and becomes a straight loss.Mutation tested
itertools.batchedadded to backend sourceThe vermin check needed a second pass, which is worth recording: the first version looked for the string anywhere in the workflow and was satisfied by a comment that mentioned it.