File Synchronization

The File Synchronization subsystem ensures that Markdown-based notes on the local filesystem remain in perfect parity with the database (SQLite or Postgres). It handles the detection of new files, updates to existing notes, deletions, and complex operations like renames or moves using a "local-first" philosophy where Markdown is the canonical representation.

Overview of the Sync Pipeline

The synchronization process is managed through initialize_file_indexing src/basic_memory/services/initialization.py147-161 which orchestrates both one-shot initial indexing and continuous filesystem watching across all entry points (API, MCP, CLI).

The pipeline operates in three primary phases:

1. Initial Reconciliation: Upon startup, reconcile_projects_with_config ensures that the config.json project registry and the database Project table src/basic_memory/models.py33-41 are in sync src/basic_memory/services/initialization.py111-121
2. Startup Indexing: A one-shot scan catches changes made while the application was offline, utilizing a fanout runtime for parallel processing src/basic_memory/services/initialization.py35-54
3. Continuous Monitoring: The WatchService starts a long-running loop using watchfiles to detect real-time filesystem events and trigger incremental updates src/basic_memory/services/initialization.py172-183

Sync Architecture and Code Entities

The following diagram bridges the high-level sync concepts to the specific classes and functions in the codebase.

Sync Subsystem Components

Sources: src/basic_memory/services/initialization.py35-183 src/basic_memory/indexing/batch_indexer.py115-154 src/basic_memory/indexing/relation_persistence.py92-98 src/basic_memory/indexing/note_materialization_runner.py23-25

WatchService and Filesystem Monitoring

The WatchService src/basic_memory/services/initialization.py172 provides real-time monitoring of project directories. It is responsible for translating OS-level file events into database updates.

Key responsibilities include:

• Project Isolation: The service can be constrained to a specific project to prevent multiple MCP instances from racing on the same files src/basic_memory/services/initialization.py181-185
• Atomic Write Detection: It handles "Vim-style" atomic writes (delete + create) by grouping events.
• Move Detection: The system identifies moved files by matching checksums between "deleted" and "created" events in a single batch, preventing unnecessary re-indexing of content.

For technical details on move detection and gitignore handling, see WatchService and Filesystem Monitoring.

Sources: src/basic_memory/services/initialization.py147-185 src/basic_memory/indexing/models.py78-89 src/basic_memory/indexing/batch_indexer.py20-25

SyncService and Batch Indexer

The BatchIndexer src/basic_memory/indexing/batch_indexer.py115-154 is the core engine for reconciling the database state with the filesystem. It processes files in bounded-parallel batches to ensure high performance.

Key responsibilities include:

• Parallel Processing: Uses _run_bounded src/basic_memory/indexing/batch_indexer.py178-182 to manage concurrency for markdown parsing and database upserts src/basic_memory/indexing/batch_indexer.py160-163
• Relation Publication: The RelationGenerationPublisher src/basic_memory/indexing/relation_persistence.py92-98 commits observations and relation chunks under a "generation fence" src/basic_memory/indexing/file_indexer.py105-112 ensuring graph updates are tied to a specific version of note content.
• Note Reconciliation: The NoteContentReconciler src/basic_memory/indexing/note_content_reconciler.py24-25 applies update plans to note_content rows, managing version guards to prevent overwriting concurrent writes src/basic_memory/indexing/note_content_reconciliation.py27-31
• Forward Reference Healing: Once targets are indexed, RepositoryRelationResolutionRuntime src/basic_memory/indexing/relation_resolution.py16-22 resolves previously "unresolved" links.

For details on the reconciliation logic and batching strategies, see SyncService and Batch Indexer.

Sources: src/basic_memory/indexing/batch_indexer.py115-200 src/basic_memory/indexing/relation_persistence.py99-144 src/basic_memory/indexing/note_content_reconciler.py16-25 src/basic_memory/indexing/relation_resolution.py1-24

System Initialization Flow

The synchronization system is initialized through initialize_app, which orchestrates database setup and project reconciliation before starting background indexing and recovery of pending materializations.

Startup Sequence

Sources: src/basic_memory/services/initialization.py57-91 src/basic_memory/services/initialization.py111-140 src/basic_memory/services/initialization.py147-185 src/basic_memory/indexing/note_materialization_runner.py23-25