What does this PR do ?

Adds a "ko" entry to LANGUAGE_TOKENIZER_MAP so get_tokenizer_for_language("ko", ...) picks the korean_chartokenizer that MagpieTTS v2607 ships, instead of silently falling back to english_phoneme.

Collection: tts

Changelog

• nemo/collections/tts/parts/utils/tts_dataset_utils.py: add "ko": ["korean_chartokenizer", "korean"] to LANGUAGE_TOKENIZER_MAP.
• tests/collections/tts/parts/utils/test_tts_dataset_utils.py: two unit tests in TestGetTokenizerForLanguage — Korean resolves to korean_chartokenizer when it is available, and still falls back to the default when a checkpoint has no Korean tokenizer.

Usage

from nemo.collections.tts.parts.utils.tts_dataset_utils import get_tokenizer_for_language

get_tokenizer_for_language("ko", ["english_phoneme", "japanese_phoneme", "korean_chartokenizer"])
# before: 'english_phoneme'   after: 'korean_chartokenizer'

Why

The v2607 model card lists Korean among the 12 supported languages, and the checkpoint's aggregated tokenizer contains korean_chartokenizer. The static map has no "ko" key, so the lookup returns the default english_phoneme, which drops almost all Hangul: the sentence 처음 뵙겠습니다. 주말은 어떠셨어요? encodes to 5 tokens with english_phoneme and 51 with korean_chartokenizer.

On main, do_tts() first consults the checkpoint's language_to_tokenizer_mapping (#15882), so whether do_tts is affected there depends on the checkpoint config, which I did not verify. The static map is still what:

• do_tts() uses when the config has no language_to_tokenizer_mapping (and always in the 3.0.0 release, which does not contain Bring MagpieTTS Release Fixes into Main Branch #15882), and
• ChunkedTTSInferenceDataset._get_tokenizer_name() uses for every manifest entry without tokenizer_names.

What I measured (nemo_toolkit 3.0.0, nvidia/magpie_tts_multilingual_357m branch v2607, a single 8 GB consumer GPU)

do_tts(text, language="ko", apply_TN=False, speaker_index=4) on three Korean sentences, then an ASR round trip with nvidia/nemotron-3.5-asr-streaming-0.6b (ko-KR) and character error rate against the input:

The same failure appeared with the main (19806879) and v2602 revisions of the checkpoint on 3.0.0. English, Japanese and Chinese sentences were normal in the same runs.

Checks run

• git diff --check: clean.
• The new and existing get_tokenizer_for_language logic, extracted from the file with ast and run against the 15 tokenizer names of the v2607 checkpoint: before the change ko -> english_phoneme, after it ko -> korean_chartokenizer; all other languages unchanged; ko without a Korean tokenizer still returns english_phoneme.
• Not run: pytest tests/collections/tts/parts/utils/test_tts_dataset_utils.py and pre-commit (I could not install the full NeMo dependency set in my environment). The new lines are within the 119-column limit.

Not changed, but noticed

With the v2607 tokenizer names, five more languages also fall back to english_phoneme through the static map: it (checkpoint has italian_chartokenizer, map expects italian_phoneme), vi (vietnamese_chartokenizer vs vietnamese_phoneme), hi (hindi_phoneme vs hindi_chartokenizer), pt (portuguese_Brazilian_phoneme, no entry) and ar (arabic_{AE,SA,MSA}_chartokenizer, no entry). I only measured Korean, so I kept this PR to ko; I am happy to extend it if you can confirm the intended tokenizer for each (especially which Arabic variant should be the default).

GitHub Actions CI

Untrusted PR: needs /ok to test <head-sha> from a maintainer.

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you add or update any necessary documentation? (No user-facing docs list this map.)
• Does the PR affect components that are optional to install? (No.)

PR Type:

• New Feature
• Bugfix
• Documentation

Who can review?

Anyone in the community is free to review the PR once the checks have passed.

Additional Information

• Related to Bring MagpieTTS Release Fixes into Main Branch #15882, Add hi-IN , Ko-KR and pt-BR IPA tokenizer support #15567

This change was drafted with AI assistance and reviewed by me.

🤖 Generated with Claude Code

🤖 Generated with Claude Code