Support textIndexVersion 3 via diacritic folding (unaccent), behind a GUC by gmprestes · Pull Request #692 · documentdb/documentdb · GitHub
Skip to content

Support textIndexVersion 3 via diacritic folding (unaccent), behind a GUC - #692

Open
gmprestes wants to merge 1 commit into
documentdb:mainfrom
gmprestes:text-index-version-3
Open

gmprestes wants to merge 1 commit into
documentdb:mainfrom
gmprestes:text-index-version-3

Conversation

@gmprestes

@gmprestes gmprestes commented Jul 29, 2026 •

Copy link
Copy Markdown
Contributor

What

Adds support for textIndexVersion: 3 on text indexes — MongoDB's default since 3.2 — implemented as diacritic folding through the unaccent extension, behind a new GUC (enableTextIndexVersion3, default off).

Why

Every mongodump taken from a modern MongoDB deployment carries "textIndexVersion": 3 in its index metadata. Today createIndexes hard-errors on it:

Failed: mydb.mycollection: error creating indexes ...
(Code(620757021)) ... Currently only textIndexVersion 2 is supported, not 3

mongorestore treats this as fatal for the collection's index phase, so migrations silently end with collections that have no text search at all. Since v3 is the upstream MongoDB default, this affects essentially every migration that includes a text index.

Version 3 differs from version 2 by diacritic insensitivity (cafe matches café), plus small tokenizer changes. For languages with heavy diacritics (Portuguese, Spanish, French…) this is also a real search-quality gap, not just a compatibility one: users overwhelmingly type unaccented queries.

How

  • New GUC enableTextIndexVersion3 (default off). Version 3 requires the unaccent extension, which not every deployment installs — hence opt-in.
    • GUC off: version-3 specs are accepted and built with version-2 semantics, with a NOTICE explaining the difference. Restores complete either way (this alone fixes the mongorestore failure).
    • GUC on: full version-3 semantics via diacritic folding.
  • New text opclass reloption textversion (2|3) recorded on the index itself, so build and query sides agree per index: a version-2 index created while the GUC is on stays diacritic sensitive; flipping the GUC later never changes existing indexes.
  • Build side (rum_bson_single_path_extract_tsvector path): document text is folded via unaccent(text) before parsetext when the index version is ≥ 3.
  • Query side (BsonTextGenerateTSQueryCore): the $text search string is folded the same way before websearch_to_tsquery, with a size guard for options blobs serialized before the reloption existed.
  • unaccent(text) is resolved through the search path with a public-schema fallback and cached; a missing extension raises a clear error with a hint (CREATE EXTENSION unaccent, or disable the GUC).

Folding runs before dictionary processing, which also fixes a subtle v2 gap we found while testing: accented stopwords (e.g. Portuguese "São") are dropped from version-2 indexes entirely, so searching sao (or even são) can never match "São Paulo". Under v3 the folded term sao survives the dictionary and is indexed:

to_tsvector('portuguese', 'São')  →  (empty — stopword)
to_tsvector('portuguese', 'sao')  →  'sao':1

Tests

New regress suite text_indexes_version3 covering:

  • invalid versions (1, non-numeric) still rejected;
  • GUC-off compatibility path (accepted, built as v2, NOTICE);
  • folded matches with the portuguese config — including the São-stopword case above;
  • the english config, where the stemmer folds nothing (this is the common broken case in the wild: mongodump metadata defaults to default_language: english, and under v2 cafe can never match Café);
  • per-index version semantics (v2 index created with the GUC on stays diacritic sensitive);
  • weighted multi-path v3 indexes;
  • updates keeping folded terms in sync.

Notes

  • No behavior change with the GUC off other than replacing a fatal error with an accept-as-v2 NOTICE.
  • Existing indexes are unaffected (reloption defaults to 2; pre-existing option blobs are size-guarded).
  • Exact v3 tokenizer parity (unicode delimiter changes) is out of scope here; folding covers the semantic difference users hit in practice.

@gmprestes
gmprestes requested a review from a team as a code owner July 29, 2026 05:23
… GUC

textIndexVersion 3 has been MongoDB's default since 3.2, which means every
dump of a modern deployment carries version 3 in its index metadata — and
the hard error here fails entire mongorestore index-restoration phases,
silently leaving migrated collections with no text search.

Version 3 differs from version 2 by diacritic insensitivity ('cafe'
matches 'café'). This change implements it end to end:

- New GUC enableTextIndexVersion3 (default off): version 3 requires the
  unaccent extension, which not every deployment installs. When the GUC is
  off, version-3 specs are accepted and built with version-2 semantics,
  with a NOTICE about the difference — restores complete either way.
- New text opclass reloption textversion (2|3) recorded on the index, so
  the build and query sides agree on the semantics per index; a version-2
  index created while the GUC is on stays diacritic sensitive.
- Build side: document text is folded via unaccent(text) before parsetext
  when the index version is >= 3.
- Query side: the $text search string is folded the same way before
  websearch_to_tsquery, guarded for options blobs serialized before the
  reloption existed.
- unaccent(text) is resolved via the search path with a public-schema
  fallback and cached; a missing extension raises a clear error with a
  hint.

Folding runs before dictionary processing, which also fixes a subtle v2
gap: accented stopwords (e.g. portuguese 'São') are dropped from v2
indexes entirely, so searching 'sao' can never match — under v3 the folded
term is indexed and found.

Includes a regress suite covering: invalid versions still rejected, the
GUC-off compatibility path, folded matches in portuguese and english
configs, per-index version semantics, weighted multi-path indexes, and
updates keeping folded terms in sync.

Co-authored-by: veluacode <no-reply@velua.ai>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Guilherme M. Prestes da Silva <gmprestes@gmail.com>
@documentdb-triage-tool

Copy link
Copy Markdown

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request write-operations

Projects

Status: Todo

Development

Successfully merging this pull request may close these issues.

1 participant