basicmachines-co
diff --git a/‎docs/multilingual-embedding-benchmark.md‎
Lines changed: 152 additions & 0 deletions b/‎docs/multilingual-embedding-benchmark.md‎
Lines changed: 152 additions & 0 deletions
diff --git a/‎docs/semantic-search.md‎
Lines changed: 27 additions & 2 deletions b/‎docs/semantic-search.md‎
Lines changed: 27 additions & 2 deletions
diff --git a/‎justfile‎
Lines changed: 24 additions & 0 deletions b/‎justfile‎
Lines changed: 24 additions & 0 deletions
diff --git a/‎test-int/BENCHMARKS.md‎
Lines changed: 17 additions & 0 deletions b/‎test-int/BENCHMARKS.md‎
Lines changed: 17 additions & 0 deletions
diff --git a/‎test-int/compare_search_benchmarks.py‎
Lines changed: 13 additions & 0 deletions b/‎test-int/compare_search_benchmarks.py‎
Lines changed: 13 additions & 0 deletions
diff --git a/‎test-int/semantic/conftest.py‎
Lines changed: 30 additions & 11 deletions b/‎test-int/semantic/conftest.py‎
Lines changed: 30 additions & 11 deletions
@@ -0,0 +1,152 @@
+# Multilingual Embedding Benchmark
+
+This benchmark supports the evidence-first model selection tracked by
+[#1372](https://github.com/basicmachines-co/basic-memory/issues/1372). It does not change Basic
+Memory's default embedding model.
+
+## Corpus and measurements
+
+`multilingual-retrieval-v1` contains 17 notes and 23 judged queries covering English, Chinese,
+Japanese, Korean, Arabic, Russian, Spanish, Thai, and mixed-language text. The query set includes
+same-language retrieval, English-to-non-English retrieval, mixed-language notes, a long-note
+chunk-boundary case, and four negative queries.
+
+Every run uses the production FastEmbed provider and one production storage pairing:
+SQLite/sqlite-vec, PostgreSQL/pgvector, or PostgreSQL/Milvus. Milvus runs use the first-party
+adapter against an isolated Milvus Lite database, while PostgreSQL continues to own search rows,
+the vector manifest, and FTS. Ranking quality is measured without a similarity cutoff. The same
+query is then repeated with the configured production cutoff to measure empty results and negative
+false positives. JSONL output includes:
+
+- recall@5, MRR@10, wrong-top-result rate, empty-result rate, and negative false-positive rate;
+- ranking and thresholded-query p50/p95 latency;
+- cold model load, indexing throughput, current/peak RSS, unique model-cache bytes, and vector
+  storage bytes;
+- model identity, dimensions, prefixes, license, corpus version, FastEmbed/Python versions, and
+  host information.
+
+Each model runs in a separate pytest process. This matters because loading multiple ONNX models in
+one process contaminates cold-load and resident-memory measurements.
+
+## Running the benchmark
+
+Python 3.12 is the authoritative environment for these measurements.
+
+```bash
+# One model/backend pair. Artifacts go to .benchmarks/.
+just benchmark-multilingual bge-small-en sqlite vector 0.55
+just benchmark-multilingual multilingual-minilm postgres vector 0.55
+just benchmark-multilingual multilingual-minilm milvus vector 0.55
+
+# Run and compare the current baseline and the first 384-dimensional candidate.
+just benchmark-multilingual-compare sqlite vector 0.55
+just benchmark-multilingual-compare postgres vector 0.55
+just benchmark-multilingual-compare milvus vector 0.55
+```
+
+The recipe installs the locked `milvus` optional extra as needed. The `milvus` backend label means
+PostgreSQL metadata/FTS plus Milvus vector storage; it is not a replacement SQL database.
+
+Supported model keys are `bge-small-en`, `multilingual-minilm`, `multilingual-mpnet`,
+`multilingual-e5-large`, and `jina-embeddings-v3`. E5's required `passage: ` and `query: ` prefixes
+are part of its benchmark contract. A model key being available to the harness does not mean it is
+approved for Cloud redistribution.
+
+## Initial screening results
+
+These results were collected on 2026-08-30 with Python 3.12.12, FastEmbed 0.8.0, and an Apple
+Silicon machine with 8 logical CPUs and 16 GiB RAM. PostgreSQL used the repository's
+`pgvector/pgvector:pg16` testcontainer. Milvus used PyMilvus 3.0.0 and Milvus Lite 3.1.0 through
+the production adapter. The small corpus proves provider/repository parity and gives directional
+model evidence; it is not a Cloud capacity test or a meaningful HNSW scale test.
+
+### SQLite vector quality at the current 0.55 cutoff
+
+| Slice | Metric | BGE small English | Multilingual MiniLM |
+| --- | --- | ---: | ---: |
+| Overall | recall@5 | 0.9474 | 1.0000 |
+| Overall | MRR@10 | 0.8474 | 1.0000 |
+| Overall | wrong top | 0.2105 | 0.0000 |
+| Overall | accepted empty | 0.0000 | 0.1579 |
+| Negative queries | false positive | 0.7500 | 0.0000 |
+| Same-language | MRR@10 | 0.9048 | 1.0000 |
+| Cross-language | recall@5 | 0.8571 | 1.0000 |
+| Cross-language | MRR@10 | 0.7762 | 1.0000 |
+| English baseline | MRR@10 | 1.0000 | 1.0000 |
+
+The PostgreSQL/pgvector and PostgreSQL/Milvus runs produced the same vector quality values for both
+models. That confirms the model comparison survives both hosted vector-storage paths.
+
+Hybrid retrieval remains backend-sensitive because SQLite FTS5 and PostgreSQL `tsvector` contribute
+their own ranks before reciprocal-rank fusion. The candidate still improved the complete hybrid
+path on both backends:
+
+| Backend | Model | recall@5 | MRR@10 | Wrong top | Negative false positive |
+| --- | --- | ---: | ---: | ---: | ---: |
+| SQLite hybrid | BGE small English | 0.8947 | 0.7531 | 0.3684 | 0.7500 |
+| SQLite hybrid | Multilingual MiniLM | 1.0000 | 0.8947 | 0.2105 | 0.0000 |
+| PostgreSQL hybrid | BGE small English | 0.8947 | 0.8070 | 0.2632 | 0.7500 |
+| PostgreSQL hybrid | Multilingual MiniLM | 1.0000 | 0.9211 | 0.1579 | 0.0000 |
+| PostgreSQL/Milvus hybrid | BGE small English | 0.8947 | 0.8070 | 0.2632 | 0.7500 |
+| PostgreSQL/Milvus hybrid | Multilingual MiniLM | 1.0000 | 0.9211 | 0.1579 | 0.0000 |
+
+Cross-language hybrid recall@5 increased from 0.7143 to 1.0000 on both backends. Hybrid MRR does
+not reach vector-only MRR because a lexical rank can still move the correct semantic result below
+an FTS result; that is fusion behavior, not a disagreement between sqlite-vec and pgvector.
+
+In vector-only retrieval, MiniLM ranks every positive query first and rejects every negative query
+at 0.55, but that cutoff also hides three correctly ranked positive results: Japanese watcher
+reconciliation, English to Spanish sourdough retrieval, and the Japanese mixed-language runbook
+query. At 0.50, the cross-language miss is recovered without introducing a negative-query false
+positive; the Japanese same-language and mixed-language queries remain below the cutoff. A model
+switch therefore needs an explicit similarity-threshold decision rather than inheriting 0.55
+without measurement.
+
+### Directional local runtime measurements
+
+| Backend | Model | Cold load | Index 17 notes | Notes/sec | Model RSS delta | Cache bytes | Vector storage |
+| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |
+| SQLite | BGE small English | 0.69 s | 1.86 s | 9.13 | 214,073,344 | 67,179,926 | 2,019,328 |
+| SQLite | Multilingual MiniLM | 1.59 s | 1.53 s | 11.14 | 436,699,136 | 252,141,023 | 2,019,328 |
+| PostgreSQL | BGE small English | 0.50 s | 8.21 s | 2.07 | 227,213,312 | 67,179,926 | 262,144 |
+| PostgreSQL | Multilingual MiniLM | 4.50 s | 11.53 s | 1.47 | 297,451,520 | 252,141,023 | 262,144 |
+| PostgreSQL/Milvus Lite | BGE small English | 0.25 s | 7.02 s | 2.42 | 260,882,432 | 67,179,926 | 40,930 |
+| PostgreSQL/Milvus Lite | Multilingual MiniLM | 1.01 s | 4.57 s | 3.72 | 684,883,968 | 252,141,023 | 40,930 |
+
+Local PostgreSQL query latency is dominated by testcontainer and `NullPool` connection setup and
+varied substantially between individual queries. Milvus Lite also opens short-lived local clients
+through the production repository boundary. The raw artifact preserves those samples, but neither
+path should be used to size Cloud workers or set a latency SLO.
+
+FastEmbed 0.8.0 reports that multilingual MiniLM now uses mean pooling instead of the CLS pooling
+used by older FastEmbed releases. Any rollout decision must therefore pin and record the tested
+FastEmbed/model combination; these results should not be treated as portable across provider
+version changes without rerunning the corpus.
+
+## Current decision
+
+Multilingual MiniLM advances as the first Cloud finalist because it materially improves every
+ranking slice, preserves the English baseline, remains at 384 dimensions, and has a permissive
+Apache-2.0 catalog license. It is not selected as the new default yet: its approximately 2x model
+RSS and 3.75x cache footprint need validation in the shared Cloud image and tenant-worker process,
+and the similarity cutoff needs calibration on a larger judged set.
+
+Do not benchmark the larger candidates by default. Advance MPNet or E5 only if MiniLM fails the
+Cloud resource/reindex test or a reviewed corpus exposes a material quality gap. The FastEmbed
+catalog labels Jina embeddings v3 as CC-BY-NC-4.0, so it is not a Cloud finalist without explicit
+license clearance.
+
+## Cloud follow-up
+
+The next bounded task is tracked in
+[`basic-memory-cloud#1898`](https://github.com/basicmachines-co/basic-memory-cloud/issues/1898):
+
+1. Bake MiniLM into the shared API/worker image and configure the same model, dimensions, and
+   prefixes in both process types.
+2. Run a representative tenant reindex on Cloud-equivalent worker hardware while measuring peak
+   RSS, duration, failure/retry behavior, image growth, vector-index size, and query latency. Run
+   the same corpus against Cloud's configured pgvector or Milvus service before comparing results.
+3. Calibrate the similarity threshold with reviewed multilingual judgments.
+4. If the evidence still favors MiniLM, deploy the new model identity and use the existing
+   revision-deduplicated fleet reindex. Accept temporary semantic incompleteness while FTS remains
+   available; do not add blue/green vector storage unless the measured rebuild window requires it.
@@ -150,6 +150,31 @@ Basic Memory passes `semantic_embedding_model` to FastEmbed, so you can select a
 embedding model registered by the installed FastEmbed version. Basic Memory does not maintain a
 second model allowlist or add arbitrary Hugging Face models to FastEmbed.
 
+The default remains `BAAI/bge-small-en-v1.5`. It is the smaller, lower-memory choice and avoids an
+automatic embedding rebuild for existing installations. Projects that need semantic retrieval
+across multiple languages can opt into multilingual MiniLM:
+
+```bash
+basic-memory config set semantic_embedding_provider fastembed
+basic-memory config set semantic_embedding_model \
+  sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
+basic-memory config set semantic_embedding_dimensions 384
+bm reindex --embeddings
+```
+
+Multilingual MiniLM covers many languages and keeps the default model's 384 dimensions.
+The matching dimensions avoid a vector-schema dimension change, but the two models' vectors are
+not interchangeable: switching in either direction still requires `bm reindex --embeddings`.
+
+Basic Memory's checked-in multilingual evaluation found that MiniLM preserved the English
+baseline while improving same-language, cross-language, and mixed-language ranking across
+Chinese, Japanese, Korean, Arabic, Russian, Spanish, and Thai. It also has a larger local artifact
+and materially higher memory use than the default. The current `0.55` similarity cutoff hid some
+correctly ranked MiniLM results in the small judged corpus, so deployments should measure their
+own queries before changing `semantic_min_similarity`. See the
+[multilingual embedding benchmark](multilingual-embedding-benchmark.md) for the corpus, results,
+limitations, and Cloud follow-up.
+
 List the models available in the same Python environment as Basic Memory. For a `uv tool`
 installation on macOS or Linux:
 
@@ -216,8 +241,8 @@ projects receive the same Postgres analysis through the managed full fleet reind
 do not run a local command.
 
 This is lexical matching, independent of the configured embedding model. Vector and hybrid search
-quality in these languages still depends on choosing a multilingual embedding model such as the
-Jina Chinese-English model or multilingual E5 configured above.
+quality in these languages still depends on choosing a multilingual embedding model such as
+multilingual MiniLM or multilingual E5 configured above.
 
 ### OpenAI
 
 
@@ -396,6 +396,30 @@ test-litellm-live *args:
 test-semantic-postgres:
     BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov {{PYTEST_FLAGS}} -m semantic -k postgres test-int/semantic/
 
+# Run one multilingual model/backend pair in its own process so cold-load and RSS
+# measurements are not contaminated by another FastEmbed model.
+benchmark-multilingual model="bge-small-en" backend="sqlite" mode="vector" threshold="0.55":
+    UV_PYTHON=3.12 \
+    BASIC_MEMORY_ENV=test \
+    LOGFIRE_IGNORE_NO_CONFIG=1 \
+    BASIC_MEMORY_MULTILINGUAL_MODEL="{{model}}" \
+    BASIC_MEMORY_MULTILINGUAL_BACKEND="{{backend}}" \
+    BASIC_MEMORY_MULTILINGUAL_RETRIEVAL_MODE="{{mode}}" \
+    BASIC_MEMORY_MULTILINGUAL_THRESHOLD="{{threshold}}" \
+    BASIC_MEMORY_BENCHMARK_OUTPUT=".benchmarks/multilingual-{{model}}-{{backend}}-{{mode}}-{{threshold}}.jsonl" \
+    uv run --extra milvus pytest -p pytest_mock -q --no-cov {{PYTEST_FLAGS}} \
+        test-int/semantic/test_multilingual_embedding_benchmark.py
+
+# Screen the current model and the 384-dimensional multilingual candidate, then
+# compare the latest records from their JSONL artifacts.
+benchmark-multilingual-compare backend="sqlite" mode="vector" threshold="0.55":
+    just benchmark-multilingual bge-small-en "{{backend}}" "{{mode}}" "{{threshold}}"
+    just benchmark-multilingual multilingual-minilm "{{backend}}" "{{mode}}" "{{threshold}}"
+    UV_PYTHON=3.12 uv run python test-int/compare_search_benchmarks.py \
+        ".benchmarks/multilingual-bge-small-en-{{backend}}-{{mode}}-{{threshold}}.jsonl" \
+        ".benchmarks/multilingual-multilingual-minilm-{{backend}}-{{mode}}-{{threshold}}.jsonl" \
+        --format table
+
 # View semantic benchmark results (rich formatted table)
 # Usage: just semantic-report [--filter-combo sqlite] [--filter-suite paraphrase] [--sort-by avg_latency_ms]
 semantic-report *args:
 
@@ -36,6 +36,23 @@ pytest test-int/test_search_performance_benchmark.py::test_benchmark_search_incr
 pytest test-int/test_search_performance_benchmark.py -v -m benchmark
 ```
 
+### Multilingual embedding model comparison
+
+The multilingual harness runs one FastEmbed model per process so model-load and RSS measurements
+remain isolated. It supports SQLite/sqlite-vec, PostgreSQL/pgvector, and PostgreSQL/Milvus. The
+Milvus case uses the production adapter with an isolated Milvus Lite database:
+
+```bash
+just benchmark-multilingual bge-small-en sqlite vector 0.55
+just benchmark-multilingual multilingual-minilm postgres vector 0.55
+just benchmark-multilingual multilingual-minilm milvus vector 0.55
+just benchmark-multilingual-compare sqlite vector 0.55
+just benchmark-multilingual-compare milvus vector 0.55
+```
+
+See [Multilingual Embedding Benchmark](../docs/multilingual-embedding-benchmark.md) for the corpus,
+model keys, metrics, initial results, and Cloud handoff.
+
 ### Write JSON benchmark artifacts
 ```bash
 BASIC_MEMORY_BENCHMARK_OUTPUT=.benchmarks/search-benchmarks.jsonl \
 
@@ -14,6 +14,17 @@
 HIGHER_IS_BETTER_SUFFIXES = ("_per_sec",)
 HIGHER_IS_BETTER_PREFIXES = ("hit_rate_", "recall_", "mrr_")
 EQUAL_IS_BETTER_KEYS = {"notes_indexed", "queries_executed"}
+LOWER_IS_BETTER_KEYS = {
+    "accepted_empty_rate",
+    "negative_false_positive_rate",
+    "wrong_top_rate",
+    "model_cache_bytes",
+    "peak_rss_bytes",
+    "rss_after_index_bytes",
+    "rss_after_load_bytes",
+    "rss_model_delta_bytes",
+    "vector_storage_bytes",
+}
 
 
 @dataclass(frozen=True)
@@ -27,6 +38,8 @@ def _preference_for_metric(metric_name: str) -> str:
     """Return optimization preference for a metric."""
     if metric_name in EQUAL_IS_BETTER_KEYS:
         return "equal"
+    if metric_name in LOWER_IS_BETTER_KEYS:
+        return "lower"
     if metric_name.startswith(HIGHER_IS_BETTER_PREFIXES):
         return "higher"
     if metric_name.endswith(HIGHER_IS_BETTER_SUFFIXES):
 
@@ -44,6 +44,7 @@
 from basic_memory.repository.entity_repository import EntityRepository
 from basic_memory.repository.project_repository import ProjectRepository
 from basic_memory.repository.search_repository import SearchRepository
+from basic_memory.repository.semantic_vector_index_factory import create_semantic_vector_index
 from basic_memory.services.file_service import FileService
 from basic_memory.services.search_service import SearchService
 
@@ -247,6 +248,7 @@ async def create_search_service(
     embedding_provider: EmbeddingProvider | None = None,
     *,
     reranker_enabled: bool = False,
+    benchmark_config: BasicMemoryConfig | None = None,
 ) -> SearchService:
     """Build a fully wired SearchService for a given combo."""
     engine, session_maker = engine_factory_result
@@ -265,27 +267,44 @@ async def create_search_service(
             },
         )
 
-    # Build app config
-    semantic_enabled = combo.provider_name is not None
-    app_config = BasicMemoryConfig(
-        env="test",
-        projects={"bench-project": str(tmp_path)},
-        default_project="bench-project",
-        database_backend=combo.backend,
-        semantic_search_enabled=semantic_enabled,
-        semantic_min_similarity=BENCHMARK_MIN_SIMILARITY,
-        reranker_enabled=reranker_enabled,
-    )
+    # A model benchmark supplies its complete provider contract so repository identity,
+    # dimensions, and prefixes match the provider that generated the stored vectors.
+    if benchmark_config is not None:
+        if benchmark_config.database_backend is not combo.backend:
+            raise ValueError("Benchmark config backend must match the search combo backend")
+        app_config = benchmark_config
+    else:
+        app_config = BasicMemoryConfig(
+            env="test",
+            projects={"bench-project": str(tmp_path)},
+            default_project="bench-project",
+            database_backend=combo.backend,
+            semantic_search_enabled=combo.provider_name is not None,
+            semantic_min_similarity=BENCHMARK_MIN_SIMILARITY,
+            reranker_enabled=reranker_enabled,
+        )
 
     # Create search repository (backend-specific)
     if combo.backend == DatabaseBackend.POSTGRES:
         from basic_memory.repository.postgres_search_repository import PostgresSearchRepository
 
+        vector_index_name = None
+        vector_index = None
+        if embedding_provider is not None:
+            vector_index_name, vector_index = create_semantic_vector_index(
+                session_maker=session_maker,
+                project_id=project.id,
+                app_config=app_config,
+                database_backend=combo.backend,
+                embedding_provider=embedding_provider,
+            )
         search_repo: SearchRepository = PostgresSearchRepository(
             session_maker,
             project_id=project.id,
             app_config=app_config,
             embedding_provider=embedding_provider,
+            vector_index_name=vector_index_name,
+            vector_index=vector_index,
         )
     else:
         from basic_memory.repository.sqlite_search_repository import SQLiteSearchRepository