|
| 1 | +# Read-path load benchmark |
| 2 | + |
| 3 | +This benchmark measures repeated, direct-permalink `read_note` calls through the real |
| 4 | +`bm mcp` stdio server. It compares an authoritative warm baseline with the same workload after |
| 5 | +the standalone Redis read cache is warmed. |
| 6 | + |
| 7 | +## Workload |
| 8 | + |
| 9 | +- deterministic 1, 16, and 64 KiB Markdown notes; |
| 10 | +- 32 distinct notes per size; |
| 11 | +- 128 measured reads at concurrency 1, 8, 32, and 64; |
| 12 | +- corpus materialization, indexing, connection setup, and cache warmup outside measurement; |
| 13 | +- isolated Basic Memory config, database, project, and home directories for every run; |
| 14 | +- JSONL output with p50, p95, p99, throughput, response bandwidth, errors, and workload metadata; |
| 15 | +- a `manifest.json` beside each result with benchmark and Basic Memory SHAs, provider versions, |
| 16 | + selected Redis and database server versions, target and harness Python versions, dirty-worktree |
| 17 | + state, synthetic-corpus checksum, and runtime configuration. |
| 18 | + |
| 19 | +Every inherited `BASIC_MEMORY_*` setting is removed before the harness adds its explicit isolated |
| 20 | +configuration. The harness explicitly disables Basic Memory auto-update so measurement cannot |
| 21 | +start background package checks or upgrades. The cached run sets `BASIC_MEMORY_REDIS_URL` only in |
| 22 | +the spawned MCP process and sizes `BASIC_MEMORY_REDIS_MAX_CONNECTIONS` to the largest requested |
| 23 | +workload or seed concurrency. This prevents high-concurrency rows from exhausting the default pool |
| 24 | +and silently measuring the cache's authoritative-read fallback. Output records whether Redis was |
| 25 | +enabled and the non-secret pool size without recording the URL, because URLs may contain |
| 26 | +credentials. |
| 27 | + |
| 28 | +The Basic Memory SHA comes from the `basic_memory` module imported by the Python environment behind |
| 29 | +`--bm-command`, not from the console script's parent directory. Publishable per-ref environments |
| 30 | +must therefore use an editable install linked to the source checkout; the harness fails rather |
| 31 | +than attributing a wheel or unrelated enclosing repository to the wrong SHA. |
| 32 | + |
| 33 | +## Run a paired comparison |
| 34 | + |
| 35 | +Use the same Redis server for the cached repetitions, but use a distinct scratch directory for |
| 36 | +every run. The server must be ready before starting the benchmark; its startup time is not part |
| 37 | +of the measurement. |
| 38 | + |
| 39 | +```bash |
| 40 | +just bench-read-cache redis://127.0.0.1:6379/0 run-01 |
| 41 | +``` |
| 42 | + |
| 43 | +The recipe writes each side's `results.jsonl` and `manifest.json` under |
| 44 | +`.scratch/read-load-{authoritative,redis-warm}-run-01/`, then prints the Markdown comparison. Give |
| 45 | +each repetition a distinct run ID so its corpus, results, and provenance remain available. Use |
| 46 | +`just bench-read-load <label> [redis_url]` when running one side independently. |
| 47 | +When invoking the script with a custom `--output` outside `--scratch`, `manifest.json` follows the |
| 48 | +JSONL file into that output directory so retained results never lose their provenance. |
| 49 | +The output path must be new unless `--truncate` is explicit; otherwise the harness fails before |
| 50 | +starting runtime resources so one manifest cannot describe JSONL rows appended by multiple runs. |
| 51 | +Each custom output directory is one run-artifact boundary: a second output filename is rejected |
| 52 | +when that directory already contains `manifest.json`, preserving the first result's provenance. |
| 53 | +Custom output and manifest paths must also stay outside the benchmark's runtime-owned `config`, |
| 54 | +`project`, and `main-home` directories. Otherwise Basic Memory could index a result while the run |
| 55 | +is active and invalidate the Redis entries being measured. |
| 56 | + |
| 57 | +Use `just bench-read-smoke [redis_url]` for a tiny real-MCP run at concurrency 64 that verifies |
| 58 | +Redis pool sizing plus result and manifest generation before committing to the full matrix. |
| 59 | + |
| 60 | +Latency is evidence, not a CI threshold. Use at least six paired repetitions before making a |
| 61 | +performance claim, alternate run order, and discard runs with material host contention. The |
| 62 | +manifest must report clean benchmark and Basic Memory worktrees for a publishable comparison. The |
| 63 | +real-Redis integration suite remains the correctness gate for cache identity and invalidation. |
| 64 | +Run that gate with `just test-read-cache`. |
0 commit comments