Embedding Providers

Relevant source files

• docs/litellm-provider.md
• docs/semantic-search.md
• src/basic_memory/api/v2/routers/prompt_router.py
• src/basic_memory/config_models.py
• src/basic_memory/repository/embedding_provider_factory.py
• src/basic_memory/repository/fastembed_provider.py
• src/basic_memory/repository/fastembed_rerank_provider.py
• src/basic_memory/repository/litellm_provider.py
• src/basic_memory/repository/litellm_rerank_provider.py
• src/basic_memory/repository/milvus_config.py
• src/basic_memory/repository/openai_provider.py
• src/basic_memory/repository/rerank_provider.py
• src/basic_memory/repository/rerank_provider_factory.py
• src/basic_memory/repository/semantic_vector_index_factory.py
• test-int/semantic/conftest.py
• test-int/semantic/corpus.py
• test-int/semantic/litellm_live_harness.py
• test-int/semantic/metrics.py
• test-int/semantic/test_litellm_live_harness.py
• test-int/semantic/test_real_fastembed_reranker.py
• test-int/semantic/test_reranker_latency.py
• test-int/semantic/test_reranker_quality.py
• tests/api/v2/conftest.py
• tests/repository/test_fastembed_provider.py
• tests/repository/test_fastembed_rerank_provider.py
• tests/repository/test_litellm_provider.py
• tests/repository/test_milvus_config.py
• tests/repository/test_openai_provider.py
• tests/repository/test_rerank_provider_factory.py
• tests/repository/test_semantic_vector_index.py

Embedding providers are the core abstraction for transforming natural language text into high-dimensional vectors. These vectors enable semantic search capabilities, such as paraphrase matching and conceptual queries, by capturing the underlying meaning of notes and observations [docs/semantic-search.md:5-12]. Basic Memory supports local-first embedding via FastEmbed and cloud-based options via OpenAI and LiteLLM [docs/semantic-search.md:113-150].

Provider Abstraction and Lifecycle

All providers implement the EmbeddingProvider interface [src/basic_memory/repository/embedding_provider.py:6-25], which requires methods for embedding single queries (embed_query) and batches of documents (embed_documents). To prevent memory leaks and redundant model loading—especially critical for local ONNX models that can exceed 2GB—Basic Memory uses a process-wide singleton pattern managed by a provider factory [src/basic_memory/repository/embedding_provider_factory.py:142-146].

Provider Factory and Caching

The create_embedding_provider function [src/basic_memory/repository/embedding_provider_factory.py:177-186] utilizes a ProviderCacheKey [src/basic_memory/repository/embedding_provider_factory.py:27-41] to ensure that a provider is only instantiated once per unique configuration. This key includes the provider name, model name, dimensions, and sensitive digests (API keys/bases), but deliberately excludes execution-tuning knobs like thread counts to avoid cache drift in containerized environments, which previously caused massive memory leaks in the ONNX CPU arena [src/basic_memory/repository/embedding_provider_factory.py:19-26].

Data Flow: From Text to Vector

The following diagram illustrates how the SearchService coordinates with providers to populate the semantic index.

Semantic Indexing Data Flow

Sources: [src/basic_memory/repository/embedding_provider.py:6], [src/basic_memory/repository/embedding_provider_factory.py:177-186], [src/basic_memory/repository/prefixing_provider.py:44-53], [src/basic_memory/repository/semantic_chunking.py:60-85]

---

FastEmbedProvider (Local Default)

The FastEmbedEmbeddingProvider [src/basic_memory/repository/fastembed_provider.py:37-38] is the default implementation. It runs entirely locally using the ONNX Runtime, requiring no API keys or network access [docs/semantic-search.md:130-132].

• Default Model: BAAI/bge-small-en-v1.5 (aliased from bge-small-en-v1.5) [src/basic_memory/repository/fastembed_provider.py:40-42].
• Dimensions: 384 [src/basic_memory/repository/fastembed_provider.py:61].
• Memory Management: It explicitly disables the ONNX CPU memory arena (enable_cpu_mem_arena=False) because the arena grows to fit peak usage and never returns that memory to the OS, leading to leaks in long-running processes [src/basic_memory/repository/fastembed_provider.py:89-97].
• Automatic Corruption Recovery: The provider identifies interrupted downloads by checking for missing model_optimized.onnx files and purges the corrupt cache subdirectories before attempting a reload [src/basic_memory/repository/fastembed_provider.py:29-34], [src/basic_memory/repository/fastembed_provider.py:154-160].

FastEmbed Execution Logic

Sources: [src/basic_memory/repository/fastembed_provider.py:79-102], [tests/repository/test_fastembed_provider.py:56-74]

---

Cloud and API Providers

For users requiring higher-dimensional vectors or offloading compute, Basic Memory provides API-backed implementations.

OpenAIProvider

The OpenAIEmbeddingProvider [src/basic_memory/repository/openai_provider.py:12] targets OpenAI's embeddings API.

• Default Model: text-embedding-3-small [src/basic_memory/repository/openai_provider.py:17].
• Concurrency: Uses an asyncio.Semaphore to control request_concurrency (default 4) [src/basic_memory/repository/openai_provider.py:29], [src/basic_memory/repository/openai_provider.py:83].
• Validation: Validates that the returned vector dimensions match the configured dimensions (default 1536) [src/basic_memory/repository/openai_provider.py:122-127].

LiteLLMProvider (Experimental)

The LiteLLMEmbeddingProvider allows routing to various backends (Cohere, Bedrock, etc.) via the LiteLLM SDK [docs/litellm-provider.md:3-6].

• Asymmetric Models: Supports different input_type settings for "document" vs "query" roles, automatically remapping for known families like Cohere v3 (search_document/search_query) and NVIDIA NIM (passage/query) [src/basic_memory/repository/litellm_provider.py:24-45].
• Forward Dimensions: Can optionally pass the dimensions parameter to the provider to request reduced-size embeddings for supported models like OpenAI text-embedding-3 [src/basic_memory/repository/litellm_provider.py:48-63].

---

Prefixing and Reranking

PrefixingProvider Wrapper

The PrefixingEmbeddingProvider [src/basic_memory/repository/prefixing_provider.py:44] wraps any EmbeddingProvider to prepend literal text prefixes (e.g., passage: or query: ) to inputs before they reach the underlying provider. This is critical for models that require role-specific prompting [src/basic_memory/repository/prefixing_provider.py:66-74].

Rerank Providers

Beyond initial retrieval, Basic Memory supports reranking via cross-encoders to improve precision.

• FastEmbedRerankProvider: A local ONNX cross-encoder provider [src/basic_memory/repository/fastembed_rerank_provider.py:63-64]. It uses a sigmoid function to map raw logits to a [0, 1] relevance scale [src/basic_memory/repository/fastembed_rerank_provider.py:155-161]. The default model is jinaai/jina-reranker-v1-tiny-en [src/basic_memory/config_models.py:61].
• LiteLLMRerankProvider: An API-based reranking provider supporting external services like Cohere Rerank [src/basic_memory/repository/litellm_rerank_provider.py:11].

Reranking Data Flow

Sources: [src/basic_memory/repository/fastembed_rerank_provider.py:142-152], [src/basic_memory/repository/rerank_provider_factory.py:34-45]

---

Batch Processing and Synchronization

Embedding operations are optimized through the sync_entity_vectors_batch mechanism, which coordinates between the repositories and providers [src/basic_memory/repository/semantic_vector_sync.py:1-65].

Feature	Description
Batch Size	Number of texts per provider request. Default 64 for FastEmbed [src/basic_memory/repository/fastembed_provider.py:60], but defaults to 2 for LiteLLM/OpenAI [docs/litellm-provider.md:55].
Sync Batching	Governs how many prepared vector jobs are flushed through the sync pipeline together [docs/litellm-provider.md:57].
Sharding	Large entities are sharded into manageable chunks for embedding to avoid memory spikes [src/basic_memory/repository/semantic_vector_sync.py:206-225].

Vector Sync Sequence

Sources: [src/basic_memory/repository/semantic_vector_sync.py:161-167], [src/basic_memory/repository/openai_provider.py:73-127], [tests/repository/test_semantic_vector_sync.py:161-183]

Storage Schema

Vectors are stored in two primary tables, utilizing sqlite-vec for SQLite or pgvector for Postgres [docs/semantic-search.md:13-14].

1. search_vector_embeddings: Stores the high-level embedding for an entire entity (Note) [tests/repository/test_sqlite_vector_search_repository.py:155-156].
2. search_vector_chunks: Stores embeddings for individual observations or sections within a note to allow for granular retrieval [src/basic_memory/repository/semantic_chunking.py:96-108].

The search_vector_chunks table includes a source_hash to detect when vectors need to be recalculated due to content changes [src/basic_memory/repository/semantic_chunking.py:98-108].

Sources: [src/basic_memory/repository/embedding_provider.py:37-41], [src/basic_memory/repository/semantic_chunking.py:88-113], [docs/semantic-search.md:13-14]