Technical Investigation: Case Sensitivity in DictionaryMetaData Lookup

Following review discussion regarding case sensitivity during dictionary lookup, we investigated how DictionaryMetaData::getCombinedBinaryType and underlying Trie lookups behave in the engine:

1. Trie Byte Matching

Dictionary entries in the binary dictionary (e.g. Frau, Weg, Strasse) are stored in a marisa::Trie (MarisaTrie.hpp), which performs exact byte-sequence lookups.

2. One-Way Lowercasing Fallback in getCombinedBinaryType

In DictionaryMetaData::getCombinedBinaryType (DictionaryMetaData.cpp):

int64_t* DictionaryMetaData::getCombinedBinaryType(int64_t* result, std::u16string_view word) const
{
    *npc(result) = 0;
    auto combinedType = dictionary->getWordType(word);
    if (!combinedType) {
        ::std::u16string normalized;
        transform(&normalized, word, dictionary->getLocale()); // Lowercases word
        if (normalized != word) {
            combinedType = dictionary->getWordType(normalized);
        }
    }
    if (!combinedType) {
        return nullptr;
    }
    *npc(result) = *combinedType;
    return result;
}

• Uppercase/Capitalized inputs (Frau / FRAU): First checks exact match Frau, then falls back to lowercasing frau.
• Lower-case inputs (frau): Searches frau (fails, as entry is stored as Frau), then transform lowercases frau -> frau (normalized == word), returning nullptr.

Conclusion for German Compound Tokenization

When a compound noun like Ehefrau is split into Ehe + frau, the head segment frau is lowercased. Because getCombinedBinaryType does not perform capitalization fallback for lowercased inputs, capitalizing frau -> Frau before getCombinedBinaryType is necessary for frau to resolve noun properties (dictionaryNoun | dictionaryFeminine) from capitalized dictionary entries.

• Lower-case inputs (frau): Searches frau (fails, as entry is stored as Frau), then transform lowercases frau -> frau (normalized == word), returning nullptr.

This is a bad assumption. Frau is first, and it gets 2 entries in buildDictionary. One is exact casing, and the other is lower cased. Then frau is encountered later. It's an exact match, and overwrites the grammemes from Frau as frau, but I believe that it pushes the possible inflection to the end of the list of inflections.

It does not return null in this scenario.

I remain skeptical on the letter casing of this pull request.

Correction + deeper investigation on dictionary lookup casing

@george you're right, and my earlier comment was wrong on the mechanism: getCombinedBinaryType("frau") does not return nullptr. Thanks for pushing on this. I verified empirically against the built German dictionary (temporary diagnostic test, since reverted):

=== frau (lowercase) result non-null: 1
=== frau (lowercase) properties: indefinite pronoun
=== Frau (uppercase) result non-null: 1
=== Frau (uppercase) properties: accusative dative feminine genitive nominative noun singular
=== frau has noun bit: 0
=== Frau has noun bit: 1

The lookup succeeds, but resolves to the wrong entry. dictionary_de.lst has two real entries:

Frau: singular accusative dative genitive nominative feminine noun inflection=1
frau: indefinite pronoun inflection=13b

frau is a genuine German word (indefinite pronoun), not a casing artifact. In buildDictionary (Dictionary.cpp::extractPartsOfSpeech):

• Frau is processed first: wordsToTypes["Frau"] = {noun,feminine,…} (operator[]), and since the normalized form differs, emplace("frau", {noun,feminine,…}) inserts a lowercase copy.
• frau is processed later: wordsToTypes["frau"] = {indefinite,pronoun} via operator[], which overwrites the normalized copy. (The emplace at line 195 can't reinstate it — the key already exists.)

Net: frau -> {indefinite,pronoun} (noun bit 0, paradigm 13b); Frau -> {noun,feminine,…} (paradigm 1). So when Ehefrau decompounds into Ehe + frau, the head-word lookup returns the pronoun homonym, (type & dictionaryNoun) == 0, and it inflects with the pronoun paradigm. The inflection test einer Ehefrau -> eine Ehefrau fails without the capitalization fix.

Could we instead fix buildDictionary so the lowercase form keeps the noun grammemes?

I tested this directly: changed both writes to merge grammemes (wordsToTypes[phrase] |= combinedType; and wordsToTypes[normalizedKey] |= combinedType;), rebuilt, and ran the full suite. It regresses other languages:

Folding a capitalized word's grammemes onto its lowercase homonym pollutes the POS/number/gender bits that downstream inflection rules rely on — which is exactly what the existing comment warns against ("Don't override previous value because this isn't an exact value").

And even ignoring the regressions, it's the wrong layer: the inflection lookup is separate and case-sensitive. DictionaryLookupInflector resolves patterns via getInflectionPatternsForWord(word) keyed by the exact string, retrying only with a lowercased variant (never capitalized), and InflectionDictionary does no case-folding. So frau can only ever resolve to paradigm 13b; the noun paradigm 1 lives exclusively under Frau. Merging grammemes would hand frau the noun bits but still leave it inflecting with the pronoun paradigm.

Conclusion: capitalizing the compound head word to Frau in DeGrammarSynthesizer is the correct, targeted layer — it fixes the lookup key for both the grammeme metadata and the inflection paradigm, scoped to German compounds rather than globally reshaping the dictionary data model. Not a null-lookup problem (my mistake), but a lowercase-homonym-shadowing problem.

We can discuss this more in the meeting.

I think that the tokenizer changes are generally fine. I am pushing back on the DeGrammarSynthesizer_DeDisplayFunction.cpp changes because it has worked fine in the past with a different German tokenizer that decompounds. It's an implementation that I can't open source.

It's probably a good idea to update the copyrights in these files too.

Now if your code were looking up/normalizing variations of ü, ä, and ö, that might be a compelling change to make. This code isn't looking up "ue" nor the NFD forms of ü. All languages pretty much assume NFC.

We can discuss this after top PRs in our meeting.

Updated the copyright years to 2026, removed the duplicate TokenizerTest#testGermanDictionary from TokenizerTest.cpp, tracked de/tokenizer.dictionary in Git LFS, and updated DeDictionaryTokenizerConfig.cpp to use forward Fugenelemente (es, en, er, ens).

In DeGrammarSynthesizer_DeDisplayFunction.cpp, tightened inflectTokenChain so non-compound words do no extra work or string copies, and when headWord is the lowercase tail of a split capitalized compound (compoundStartToken != headWordToken), it reuses capGrammemes instead of looking up headWord twice.

Capitalizing the split compound head is still required for three cases in dictionary_de.lst / inflectional_de.xml:

1. Lowercase non-noun homographs overwriting emplaced nouns in Dictionary.cpp (4,935 capitalized nouns, including Frau, Mann, Weg, Decken):
   • In Dictionary.cpp, Frau: is read first (A-Z before a-z) and emplaces u"frau" at lines 195–197, but when frau: indefinite pronoun inflection=13b is read later, wordsToTypes[phrase] = combinedType (line 189) and wordToPropertyValue[phrase] = std::move(propertyValues) (line 202) use operator[] and overwrite the emplaced entry.
   • Without capitalizing frau -> Frau, <source case="accusative" number="singular" gender="feminine">einer Ehefrau</source> in test/resources/inflection/dialog/inflection/de.xml:322 (tokenized as |einer| |Ehe|frau|) fails because u"frau" resolves to indefinite pronoun ((headBinaryType & dictionaryNoun) == 0). (Standalone Ehefrau on line 222 is in dictionary_de.lst so getDisplayValue inflects it directly without tokenizing.)
   • Dictionary.cpp keeps wordsToTypes[u"decken"] as a verb rather than merging Decken's noun bits because de.xml:237-243 explicitly tests that Decken inflects to Decke while lowercase decken does not inflect (<!--First word is capitalized, which indicates a noun and it should not use inflection rules for the verb-->).
2. Plural-to-singular compound inflection with an adjective (e.g. schönen Ehefrauen -> schöne Ehefrau):
   • Lowercase u"frauen" has dictionaryNoun (the pronoun frau has no plural), so inflectWord(u"frauen", ...) returns lowercase u"frau", and inflectAdjectiveNextToNoun (DeGrammarSynthesizer_DeDisplayFunction.cpp:345) then looks up u"frau" for gender agreement, hitting the genderless pronoun frau and failing.
3. Initial-umlaut noun patterns in inflectional_de.xml (e.g. Arzt -> Ärzte in pattern name="48c", Apfel -> Äpfel in pattern name="65b"):
   • Because the initial capital letter changes (A -> Ä), the <suffix> in inflectional_de.xml is capitalized (<suffix>Arzt</suffix> -> <t><stem/>Ärzte</t>). Even though wordsToTypes[u"arzt"] has dictionaryNoun, Inflector_InflectionPattern matches <suffix>Arzt</suffix> case-sensitively, so a lowercase compound head (Wald|arzt -> Wald|ärzte) fails suffix matching unless capitalized to u"Arzt".