iframe-proxy

leeshengis · 2026-10-14T02:29:11Z

Summary

Fix local text tokenization dropping Unicode Alphabetic marks, which can make distinct words match the same filter.

For example, insert points whose words payloads are क and की, then scroll with:

models.Filter(must=[
    models.FieldCondition(key="words", match=models.MatchText(text="क"))
])

Before this change, local mode returns both points. Qdrant 1.19.2 returns only क. The same mismatch occurs with MatchTextAny(text_any="क") and MatchPhrase(phrase="क").

Rust's word tokenizer uses Unicode Alphabetic + Number. Python's re pattern [^\W_]+ excludes Alphabetic marks such as Devanagari vowel signs and U+0345. Use regex property classes instead, and preserve tokenization-before-lowercasing. Not all combining marks are Alphabetic: U+0301 must remain a separator. No normalization is introduced.

This adds the mandatory regex>=2024.11.6 dependency because stdlib re does not support these Unicode properties; str.isalnum() has the same property gap. This adds a native-extension dependency (wheels where available, otherwise a build is needed). The lock update adds only this package, without updating existing packages.

Unicode tables can differ between Python, regex and the server. This change fixes token boundaries; it does not promise complete parity across Unicode versions or change Python lowercasing. For example, Python 3.12's Unicode 15 tables do not lowercase the Unicode 16 U+1C89/U+1C8A pair like the tested server does.

Validation

Qdrant 1.19.2, commit 016542aa5deb6c66380bb137badf73d54f742bde: public-API comparisons across local mode, REST and gRPC, unindexed fields and Word text indexes with phrase matching
The reproducer's 12 queries × 3 match types agree across all five configurations after the fix
Regressions cover Devanagari, Alphabetic/non-Alphabetic combining marks, underscores, CJK, numeric categories, and tokenization before lowercasing
Python 3.12.14 + regex 2026.9.29: 155 local tests, 48 local-client tests and 63 complex-filter congruence tests passed
Python 3.12.14 + minimum regex 2024.11.6: 155 local tests and 13 text congruence tests passed
Python 3.10.22 + minimum regex 2024.11.6: 203 local/client tests and 63 complex-filter congruence tests passed
Original pre-commit configuration (Python 3.10), local_collection doctest and git diff --check passed
Full Docker-based CI/all-extras suite not run locally

All Submissions

Based on and targeting dev
Followed the repository's documented development instructions
Checked for existing relevant pull requests

Changes to Core Features

Explained the change and motivation
Added regression tests
Ran the relevant tests locally
Installed and ran pre-commit hooks

netlify · 2026-10-14T02:29:18Z

coderabbitai · 2026-10-14T02:31:49Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Source mode: Auto
Repository: Unavailable
Branch: Unavailable
File: No configuration file from this repository applied
Commit: Unavailable
Layers: Organization UI
Review profile: CHILL
Plan: Advanced
Run ID: 47fc7be1-c041-45ce-b792-3adc3746fab9

📥 Commits

Reviewing files that changed from the base of the PR and between e85f330 and c8f3642.

⛔ Files ignored due to path filters: 1

poetry.lock is excluded by !**/*.lock

📒 Files selected for processing: 4

pyproject.toml
qdrant_client/local/payload_filters.py
qdrant_client/local/tests/test_payload_filters.py
tests/congruence_tests/test_complex_filters.py

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 3 remain after this review.

📝 Walkthrough

The local payload filter tokenizer now uses the regex module to match Unicode Alphabetic and Number characters before lowercasing tokens. The project adds regex as a runtime dependency. New unit and congruence tests cover Unicode text, combining marks, and text filter matching.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Suggested reviewers: generall

Merge Risk: ⚪ Minimal · up to c8f36

No actionable issue remains in the reviewed Unicode tokenization change; it is ready for normal merge checks.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 22.22% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 9 functions across 3 files. (1 skipped: 1…	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Title check		The title names the main change: local text tokenization now keeps Unicode Alphabetic combining marks. It is short, specific, and matches the changeset (tokenizer switched to regex Unicode properties,…
Description check		The description states the bug (local tokenization drops Unicode Alphabetic marks, so distinct words such as क and की match the same filter). It gives a reproducer, names the regex dependency, and lis…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Full details: Title check

Explanation

The title names the main change: local text tokenization now keeps Unicode Alphabetic combining marks. It is short, specific, and matches the changeset (tokenizer switched to regex Unicode properties, plus new tests for Devanagari and combining marks).

Full details: Description check

Explanation

The description states the bug (local tokenization drops Unicode Alphabetic marks, so distinct words such as क and की match the same filter). It gives a reproducer, names the regex dependency, and lists regression tests and validation runs. This is directly related to the changeset.

Full details: Docstring Coverage

Explanation

Docstring coverage is 22.22% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 9 functions across 3 files. (1 skipped: 1 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create a new PR

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

chrikrah

@leeshengis from my perspective the tokenizer change can be merged. Local mode now tokenizes like 1.19.2 for MatchText and MatchTextAny, and the new tests fail on the base. I checked token boundaries only, so whether regex is acceptable as a new required native dependency stays open.

$ python -m pytest qdrant_client/local/tests/test_payload_filters.py -q   # head c8f3642, CPython 3.12, regex 2026.9.29
23 passed in 1.10s
# payload_filters.py from b64c22e (dev), tests kept:
6 failed, 17 passed in 1.31s

# probe: unindexed "words" payloads 1 "क", 2 "की", 3 "aͅb", 4 "áb"; scroll ids
base b64c22e      MatchText('क') -> [1, 2]   MatchText('a b') -> [3, 4]
head c8f3642      MatchText('क') -> [1]      MatchText('a b') -> [4]
server 1.19.2     MatchText('क') -> [1]      MatchText('a b') -> [4]   (MatchTextAny: same ids on head and server)

Has a maintainer agreed to that dependency, on the linked issue or elsewhere?

leeshengis · 2026-10-26T06:28:20Z

fix: preserve Unicode alphabetic marks in local text tokenization

e85f330

leeshengis requested a review from joein as a code owner October 14, 2026 02:29

Merge latest dev and resolve Poetry lock metadata conflict

c8f3642

chrikrah reviewed Oct 25, 2026

View reviewed changes

Sunbelt Computer Software

PL/B Language Development and Support

Navigation Menu

Repository navigation

fix: preserve Unicode alphabetic marks in local text tokenization - #1575

fix: preserve Unicode alphabetic marks in local text tokenization#1575
leeshengis wants to merge 2 commits into
qdrant:devfrom
leeshengis:fix/local-unicode-tokenization

leeshengis commented Oct 14, 2026

Uh oh!

netlify Bot commented Oct 14, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Oct 14, 2026 •

edited

Loading

Uh oh!

chrikrah left a comment

Uh oh!

leeshengis commented Oct 26, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Sunbelt Computer Software

PL/B Language Development and Support

Conversation

leeshengis commented Oct 14, 2026

Summary

Validation

All Submissions

Changes to Core Features

Uh oh!

netlify Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for poetic-froyo-8baba7 ready!

Uh oh!

coderabbitai Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chrikrah left a comment

Choose a reason for hiding this comment

Uh oh!

leeshengis commented Oct 26, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

netlify Bot commented Oct 14, 2026 •

edited

Loading

coderabbitai Bot commented Oct 14, 2026 •

edited

Loading