​

File Structure

​

1. Header (4 KB)

• Magic bytes: Identifies the file as .mv2 format
• Version: File format version
• WAL metadata: Position and size of write-ahead log
• Footer offset: Points to the table of contents

​

2. Embedded WAL

• All mutations are written to WAL first
• On recovery, uncommitted changes are replayed
• Size scales with file capacity (1 MB to 64 MB)

​

3. Segments (Frames)

• Text segments: Document content and metadata stored as frames
• Blob segments: Binary data (images, PDFs) as frames
• Media segments: Audio and video content as frames
• Vector segments: Embeddings for semantic search (optional)

​

4. Indices

• Lexical index (BM25): Full-text keyword search - works out of the box
• Time index: Temporal ordering of frames
• Vector index: Semantic similarity search - optional, add when needed

​

5. Table of Contents + Footer

• Segment locations and sizes
• Index offsets
• Checksums for integrity verification

​

Data Lifecycle

​

Writing Data

1. put() - Adds frames (documents) to pending state
2. Indices updated - Lexical and vector indices are built
3. Time entries queued - Timestamps recorded for timeline
4. WAL appended - Transaction logged for crash safety
5. seal() - Commits everything to disk with checksums

from memvid_sdk import use

mem = use('basic', 'knowledge.mv2')

# 1. Add documents (pending)
mem.put(title="Doc 1", label="docs", metadata={}, text="Your content")
mem.put(title="Report", label="docs", metadata={}, file="report.pdf")

# 2. Commit to disk
mem.seal()

​

Reading Data

1. Open file - Locate latest valid footer
2. Load TOC - Map segments and indices
3. Replay WAL - Apply any uncommitted changes
4. Query indices - Search lexical/vector/time indices
5. Return results - Ranked documents with snippets

​

Single-File Guarantee

​

What It Means

• No sidecar files: No .wal, .lock, .shm files
• No external state: Everything is in the .mv2 file
• Portable: Copy the file to transfer the entire memory

​

Why It Matters

# Your entire knowledge base
ls ~/project/
# → knowledge.mv2

# Share it anywhere
cp knowledge.mv2 /team/shared/
scp knowledge.mv2 user@server:/data/
git add knowledge.mv2

​

How It Works

​

Crash Safety

​

Write-Ahead Logging

1. Transaction written to WAL region
2. WAL synced to disk (fsync)
3. Changes applied to segments
4. Checksum updated

​

Recovery Process

1. Locates the last valid footer
2. Loads the table of contents
3. Scans WAL for uncommitted entries
4. Replays any pending transactions

​

WAL Sizing

​

Locking and Concurrency

​

File Locking

• Shared locks: Multiple readers allowed
• Exclusive locks: Single writer at a time

​

Read-Only Mode

# Multiple processes can read simultaneously
mem = use('basic', 'knowledge.mv2', read_only=True)
results = mem.find('query')

​

Writer Conflicts

from memvid_sdk import use, LockedError

try:
    mem = use('basic', 'knowledge.mv2')
except LockedError:
    print("File is locked by another process")

​

Determinism

​

Why Determinism Matters

• Reproducible builds: Same data → same file
• Reliable testing: Predictable behavior
• Easy debugging: Consistent results

​

How It’s Achieved

• Segments written in deterministic order
• Timestamps explicit, not system-derived
• Checksums verify integrity

​

Performance Considerations

​

Memory Usage

• Table of contents
• WAL handle
• Pending time entries

• Closing handles when done
• Using read-only mode for queries

​

Index Building

• Lexical index: BM25 tokenization and indexing
• Vector index: Graph construction for similarity search

memvid put knowledge.mv2 --input ./large-dataset/ \
  --vector-compression \
  --parallel-segments

​

Search Optimization

• Lexical search: Fast for exact keywords
• Vector search: Slower but more intelligent
• Hybrid search: Balances both

​

Next Steps