{{ message }}
Repository navigation
fix(tokenizer): point users at from_hf_hub on unknown model (#229) - #231
Merged
juliendenize merged 3 commits intoJun 1, 2026
Merged
Conversation
…i#229) The static MODEL_NAME_TO_TOKENIZER_CLS map only covers older model snapshots. Newer models (mistral-small-3, mistral-medium-3, magistral, saba, ...) fall through to a bare "Unrecognized model: X" which doesn't tell the caller what to do next. Surface the available escape hatches in the error itself — point at MistralTokenizer.from_hf_hub() and .from_file(), and list the known names so typos are obvious.
juliendenize
requested changes
May 29, 2026
Collaborator
There was a problem hiding this comment.
Hey thanks for the contribution !
i think that the best would be to simply deprecate this from_model method and indeed point to from_file or from_hf_hub as we won't maintain the model mapping (it's been quite a while ^^)
raising as you did can be kept with minor change
lmk if you'd like to do the iteration otherwise i'll take over in another pr :)
Collaborator
There was a problem hiding this comment.
Suggested change
| f"Unrecognized model: {model!r}. The static mapping in MistralTokenizer.from_model " | |
| f"Unrecognized model: {model}. The static mapping in MistralTokenizer.from_model " |
| raise TokenizerException( | ||
| f"Unrecognized model: {model!r}. The static mapping in MistralTokenizer.from_model " | ||
| f"only covers older model snapshots; newer or custom models should be loaded with " | ||
| f"MistralTokenizer.from_hf_hub(repo_id=...) or MistralTokenizer.from_file(...). " |
Collaborator
There was a problem hiding this comment.
to be consistent
Suggested change
| f"MistralTokenizer.from_hf_hub(repo_id=...) or MistralTokenizer.from_file(...). " | |
| f"MistralTokenizer.from_hf_hub(...) or MistralTokenizer.from_file(...). " |
Comment on lines
+32
to
+34
| # The error should point users at from_hf_hub / from_file so they can | ||
| # handle model snapshots that aren't in the static mapping, and echo | ||
| # the bad name so typos are obvious. |
Collaborator
There was a problem hiding this comment.
Suggested change
| # The error should point users at from_hf_hub / from_file so they can | |
| # handle model snapshots that aren't in the static mapping, and echo | |
| # the bad name so typos are obvious. |
The static MODEL_NAME_TO_TOKENIZER_CLS map isn't kept current with new releases. Mark from_model with a FutureWarning pointing at from_hf_hub and from_file, drop the known-name listing from the unknown-model error (since the map isn't authoritative going forward), and migrate the repo's own callers to .v1() / .v3() or wrap them in pytest.warns so test output stays quiet. Addresses @juliendenize's review on mistralai#231.
Contributor
Author
|
Really appreciate the thoughtful review @juliendenize :D. The deprecation framing is a much cleaner direction than my original "better error" attempt, and the consistency notes on the suggestion blocks made the diff way tighter. Genuinely learned something from this iteration, so thanks for taking the time on it! Pushed the deprecation:
Local run is green ( |
juliendenize
reviewed
Jun 1, 2026
juliendenize
left a comment
juliendenize
left a comment
Collaborator
There was a problem hiding this comment.
Thanks for iterating looks good now
Co-authored-by: Julien Denize <40604584+juliendenize@users.noreply.github.com>
juliendenize
approved these changes
Jun 1, 2026
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
`MistralTokenizer.from_model` raises `TokenizerException("Unrecognized model: X")` whenever the model name isn't in `MODEL_NAME_TO_TOKENIZER_CLS`. That mapping only covers older snapshots (latest entry is `2411`), so anything new — `mistral-small-3`, `mistral-medium-3`, `magistral-medium`, `saba`, ... — hits this branch and the caller has no idea what to do next.
This swaps the bare message for one that points at the existing escape hatches and echoes the failing name so typos are obvious:
```text
Unrecognized model: 'mistral-small-3-2501'. The static mapping in
MistralTokenizer.from_model only covers older model snapshots; newer or
custom models should be loaded with MistralTokenizer.from_hf_hub(repo_id=...)
or MistralTokenizer.from_file(...). Known names: codestral-2405, codestral-mamba-2407, ...
```
I considered also adding new entries to `MODEL_NAME_TO_TOKENIZER_CLS` for the recent models, but I don't want to guess at the right tokenizer version (v3-tekken vs v7 vs v11 with image config) for each model — that decision really has to come from your side. Happy to send a follow-up once the mapping for the newer snapshots is published.
Refs #229.
Tests
Existing `tests/test_mistral_tokenizer.py::test_from_model` already asserts `TokenizerException` for `"unknown-model"`. Extended it to also assert the message includes the offending name and the two escape hatches:
```
$ pytest tests/test_mistral_tokenizer.py::TestMistralToknizer::test_from_model
1 passed in 0.40s
```