Derive warmup steps from a model default that gives a ratio - #8723
Conversation
model_defaults may express warmup either way, and the backend accepts both:
core/training/worker.py prefers warmup_steps and falls back to
round(warmup_ratio * max_steps). The form only ever read warmup_steps, and
BackendTrainingDefaults did not even declare warmup_ratio, so a config that
gives only the ratio left the Warmup Steps field on its generic default.
Ten of the shipped configs are in that shape, none of which also set
warmup_steps:
default.yaml warmup_ratio 0.1
gemma/unsloth_gemma-3-4b-pt.yaml warmup_ratio 0.03
gemma/unsloth_gemma-3n-E4B.yaml warmup_ratio 0.03
other/unsloth_tinyllama-bnb-4bit.yaml warmup_ratio 0.1
qwen/unsloth_Qwen2.5-1.5B-Instruct.yaml warmup_ratio 0.1
embedding/*.yaml (5 files) warmup_ratio 0.03
default.yaml is the fallback for every model without a specific config, so
this is not limited to the four named ones.
Read warmup_ratio when warmup_steps is absent and convert it the same way the
worker does. An explicit warmup_steps still wins, and a ratio with no usable
max_steps leaves the field alone rather than writing a zero.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1cd39eb21b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // studio/backend/core/training/worker.py does when it falls back. | ||
| const warmupRatio = toNumber(training?.warmup_ratio); | ||
| if (warmupRatio !== undefined && maxSteps !== undefined && maxSteps > 0) { | ||
| patch.warmupSteps = Math.round(warmupRatio * maxSteps); |
There was a problem hiding this comment.
Match the backend's tie rounding for warmup steps
When a model or imported YAML produces a half-step value, such as warmup_ratio: 0.05 with max_steps: 50, Math.round derives 3 steps, while the referenced backend fallback in worker.py uses Python's int(round(...)) and derives 2 because Python rounds ties to even. Since the frontend submits the derived value as warmup_steps, this silently changes the effective training configuration; use rounding semantics that match the backend.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Declining this one. The int(round(...)) at studio/backend/core/training/worker.py:2979 runs only when warmup_steps is None and warmup_ratio is not None, and it sits inside _run_mlx_training, which rejects embedding runs outright at worker.py:2612. mappers.ts:121-122 sends warmup_steps: isEmbedding ? null : config.warmupSteps and warmup_ratio: isEmbedding ? 0.03 : null, with config.warmupSteps a non-nullable number defaulting to 5 (src/config/training.ts:83). Every run that can reach that branch therefore arrives with a concrete warmup_steps and a null ratio, so the condition is never satisfied.
The comparison also assumes the alternative outcome is 2 steps. Without this change the field carries the generic default 5, not 2, which is the bug being fixed, so there is no competing derivation to diverge from.
Where the backend does own a ratio, the operative semantics are ceil, not round. Embedding runs send warmup_ratio (mappers.ts:122 into worker.py:5208), and trainer.py:3790 prefers warmup_ratio over warmup_steps when both are present. Both hand it to TrainingArguments.get_warmup_steps, which is math.ceil(num_training_steps * ratio). ceil(2.5) is 3 and agrees with Math.round, while int(round(2.5)) is 2 and is the outlier, so matching it would move the derived value away from what the backend actually applies to a ratio.
No shipped config produces a tie either: 0.1 * 30 is 3.0 and 0.03 * 30 is 0.8999999999999999.
|
@codex review |
1 similar comment
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1cd39eb21b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // studio/backend/core/training/worker.py does when it falls back. | ||
| const warmupRatio = toNumber(training?.warmup_ratio); | ||
| if (warmupRatio !== undefined && maxSteps !== undefined && maxSteps > 0) { | ||
| patch.warmupSteps = Math.round(warmupRatio * maxSteps); |
There was a problem hiding this comment.
Use ceiling semantics when materializing a warmup ratio
For a non-embedding CUDA config where warmup_ratio * max_steps is below a half-integer (for example, 0.01 * 30), this writes warmupSteps: 0, and mappers.ts then submits that concrete step count instead of the ratio. The CUDA _train_worker passes a ratio to SFTConfig, whose Hugging Face TrainingArguments derives warmup with ceil, so the same config should have one warmup step rather than none. This is fresh evidence beyond the earlier MLX-only rounding example: it affects custom/imported ratios on the generic CUDA training path.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Correct, and fixed in e18856c. Math.round writes 0 whenever the product is under 0.5, so an imported config such as warmup_ratio: 0.03 with max_steps: 10 asked for warmup and would have trained with none. That is the same outcome the max_steps guard on the branch below already exists to avoid, so the two halves of the derivation disagreed with each other.
Math.ceil is what TrainingArguments.get_warmup_steps uses to materialize a ratio, which is the semantics every backend path applies wherever it owns one, and it produces identical values for all 78 shipped configs (0.1 * 30 is 3 and 0.03 * 30 is 1 under either operator). An explicit warmup_ratio: 0 still maps to 0.
One correction on the route you describe: for a non-embedding run mappers.ts:137 sends warmup_ratio: null, so _train_worker takes the warmup_steps branch at trainer.py:4044 and passes the literal 0 rather than a ratio. The conclusion is unaffected, since 0 is wrong on its own terms.
Regression test added in tests/training-model-defaults-warmup-ratio.test.ts, "a ratio too small to reach one step still gets one". It fails against the previous Math.round with actual: 0, expected: 1.
|
Notes on the red board, since none of it came from this change.
I merged current On the merged tree the frontend suite is 3417 passed / 0 failed, |
|
@codex review |
|
Codex Review: Didn't find any major issues. 🚀 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
CI finished on the merged base: 31 of 34 green, and all three reds trace off this change.
Identical, and 52 + 2 is the 54 CI reports. Both files assert on the source text of
One thing worth separating out: My earlier note named |

Problem
model_defaultsmay express warmup either aswarmup_stepsor aswarmup_ratio, and the backend accepts both.studio/backend/core/training/worker.pyis explicit about it:The form only ever read
warmup_steps:and
BackendTrainingDefaultsdid not declarewarmup_ratioat all, so there was no way to read it. A config that gives only the ratio therefore never reaches the Warmup Steps field, which keeps the generic default of 5 fromsrc/config/training.ts.Ten of the 78 shipped configs are in exactly that shape, and none of them also sets
warmup_steps, so there is no second source the value could arrive from:default.yamlgemma/unsloth_gemma-3-4b-pt.yamlgemma/unsloth_gemma-3n-E4B.yamlother/unsloth_tinyllama-bnb-4bit.yamlqwen/unsloth_Qwen2.5-1.5B-Instruct.yamlembedding/*.yaml(5 files)default.yamlis the fallback for every model that has no specific config, so this is not limited to the four named models.The embedding five are the mildest case, because
mappers.tsseparately hardcodeswarmup_ratio: isEmbedding ? 0.03 : nullinto the start payload, so the backend ends up with the right number regardless of what the field showed. The other five have nothing filling the gap.Fix
Read
warmup_ratiowhenwarmup_stepsis absent, and convert it the way the worker already does.max_stepsis read first so the conversion has it.An explicit
warmup_stepsstill wins, matching the worker's precedence. A ratio with no usablemax_stepsleaves the field alone rather than writing a zero, since a zero would be a worse default than the existing 5.Deliberately not done: adding a Warmup Ratio field to the UI. That is a bigger change and a product decision, and the shipped configs only need the value to arrive.
Test
tests/training-model-defaults-warmup-ratio.test.ts, four cases. The last one walks all 78 shipped configs and asserts that any config declaring a warmup in either spelling produces a numericwarmupSteps, so a future config written with a ratio cannot regress silently. It also asserts it found more than 50 configs, so a move or rename fails loudly rather than leaving the test checking nothing.Two of the four pass either way on purpose: they pin the behaviour the change could have broken (precedence, and not writing a zero) rather than reproducing the bug.
Whole suite,
npm test: 2277 passed / 0 failed before, 2281 passed / 0 failed after, so the four new cases are the entire delta.npm run typecheckis clean.biome checkon the two changed source files reports the same 40 warnings before and after, and the new test file carries the samenoNodejsModulesanduseNamingConventionwarnings the neighbouring test files already do, with formatting applied.