You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Bug] Unreadable memory.json is silently overwritten instead of quarantined (data loss) #7072
MarkdownMemoryStorage (the deermem backend's markdown storage) silently destroys an unreadable memory.json. When the file cannot be decoded (e.g. UTF-16 BOM, invalid UTF-8, permission error), _load_summary...'s except (OSError, UnicodeError) branch logs a warning and returns None, and the next commit then rebuilds a fresh revision-1 manifest over the original bytes — the user's stored memories are gone with no backup.
The module documents a quarantine contract for exactly this case (memory.json.corrupt-<timestamp>), and the sibling decode/fence path already implements it; the summary-load except branch skips it.
Repro
Write a memory.json that is not valid UTF-8 (e.g. encode a valid summary as UTF-16 with BOM).
Load the storage (MarkdownMemoryStorage(...) + a read/load call).
Save anything (e.g. save(create_empty_memory())).
On current main (commit 62130caa):
WARNING ... Cannot read memory summary ...: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte
# load() returns an empty document, zero quarantine files
# save() returns True; the original bytes are absent everywhere under the storage root;
# no backup; the manifest is rewritten at revision: 1
Expected: the unreadable file is moved aside as memory.json.corrupt-* (best-effort, per the module's documented contract) and the original bytes remain recoverable; a subsequent save must not destroy them.
Suggested fix
Call the existing best-effort self._quarantine_unreadable(path) in the summary-load except (OSError, UnicodeError) branch before returning None — matching the sibling decode/fence path that already quarantines. One line; verified to preserve the original bytes (a single memory.json.corrupt-* with byte-identical content survives subsequent loads and saves; repeated loads do not create duplicates because the file is moved away).
Tests
backend/tests/test_memory_storage.py::TestMarkdownMemoryStorage::test_non_utf8_summary_is_quarantined_before_rebuild (UTF-16 BOM file; asserts one quarantine file with the original bytes that survive a subsequent save).
On base: fails — AssertionError: a non-UTF-8 summary must be quarantined, not silently rebuilt over / assert 0 == 1.
With the fix: passes; the six neighboring memory-backend modules are 189 passed / 1 skipped (base: 188/1, delta exactly the new test). Ruff clean.
Activity
fix(memory): quarantine unreadable memory.json in the read-error bran…
fix(memory): quarantine unreadable memory summaries (bytedance#7076)