[Bug] Unreadable memory.json is silently overwritten instead of quarantined (data loss) · Issue #7072 · bytedance/deer-flow · GitHub
Skip to content

[Bug] Unreadable memory.json is silently overwritten instead of quarantined (data loss) #7072

Closed
wannT-1/deer-flow
#3

Description

@BlueX888

Summary

MarkdownMemoryStorage (the deermem backend's markdown storage) silently destroys an unreadable memory.json. When the file cannot be decoded (e.g. UTF-16 BOM, invalid UTF-8, permission error), _load_summary...'s except (OSError, UnicodeError) branch logs a warning and returns None, and the next commit then rebuilds a fresh revision-1 manifest over the original bytes — the user's stored memories are gone with no backup.

The module documents a quarantine contract for exactly this case (memory.json.corrupt-<timestamp>), and the sibling decode/fence path already implements it; the summary-load except branch skips it.

Repro

  1. Write a memory.json that is not valid UTF-8 (e.g. encode a valid summary as UTF-16 with BOM).
  2. Load the storage (MarkdownMemoryStorage(...) + a read/load call).
  3. Save anything (e.g. save(create_empty_memory())).

On current main (commit 62130caa):

WARNING ... Cannot read memory summary ...: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte
# load() returns an empty document, zero quarantine files
# save() returns True; the original bytes are absent everywhere under the storage root;
# no backup; the manifest is rewritten at revision: 1

Expected: the unreadable file is moved aside as memory.json.corrupt-* (best-effort, per the module's documented contract) and the original bytes remain recoverable; a subsequent save must not destroy them.

Suggested fix

Call the existing best-effort self._quarantine_unreadable(path) in the summary-load except (OSError, UnicodeError) branch before returning None — matching the sibling decode/fence path that already quarantines. One line; verified to preserve the original bytes (a single memory.json.corrupt-* with byte-identical content survives subsequent loads and saves; repeated loads do not create duplicates because the file is moved away).

Tests

backend/tests/test_memory_storage.py::TestMarkdownMemoryStorage::test_non_utf8_summary_is_quarantined_before_rebuild (UTF-16 BOM file; asserts one quarantine file with the original bytes that survive a subsequent save).

  • On base: fails — AssertionError: a non-UTF-8 summary must be quarantined, not silently rebuilt over / assert 0 == 1.
  • With the fix: passes; the six neighboring memory-backend modules are 189 passed / 1 skipped (base: 188/1, delta exactly the new test). Ruff clean.

Activity

  1. added a commit that references this issue on Oct 18, 2026
    069e9bd
  2. added a commit that references this issue on Oct 20, 2026
    2174c1d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs-triageAwaiting maintainer triage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions