Take the SQLite write lock up front for asset scan and output-registr… · code/app-python-comfyui@9d3bfc3 · GitHub
Skip to content

Commit 9d3bfc3

Browse files
authored
Take the SQLite write lock up front for asset scan and output-registration writes (Comfy-Org#16480)
* Take the SQLite write lock up front for scan and output-registration writes The scanner's seeding and reference sync, and executed-output registration, write through a separate engine whose transactions open with BEGIN IMMEDIATE, and the database runs in WAL mode. On those paths stat, hashing and metadata extraction now happen before the write transaction opens, and reference-sync results are applied only to rows unchanged since they were observed. busy_timeout stays at pysqlite's 5s default. A fast-scan batch now commits as one transaction, so an unexpected error partway through discards the whole batch; the next scan recreates it. Enrichment, verification, uploads and tagging still write through the existing sessions. Migration backups use SQLite's backup API, and a legacy database is checkpointed before it is relocated, since in WAL mode committed pages can live in the -wal file that a plain file copy misses. * Skip relocating a legacy database whose WAL cannot be checkpointed The checkpoint can report busy without raising; moving the file then would leave committed pages behind in the -wal. Also keep the source's file mode on SQLite backups, as the plain copy did. * Replace run_write_txn with a create_write_session factory Write paths open the write engine's session the same way the rest of the code opens create_session(), and commit explicitly. Executed-output registration reads the new record's fields before committing, so expiry does not reload them in a second write transaction. * Document the scanner's pre-transaction observation types * Document that write sessions must not nest * Warn that a nested write session looks like lock contention * Seed a hashed spec with the stat its hash was verified against * Bound the SQLite backup so a locked destination cannot hang startup * Time out a backup only while it is blocked
1 parent 557597f commit 9d3bfc3

16 files changed

Lines changed: 452 additions & 109 deletions

‎app/assets/scanner.py‎

Lines changed: 129 additions & 72 deletions

‎app/assets/scanner_changes.py‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -44,8 +44,13 @@ def pending_recovery_count() -> int:
4444

4545

4646
def recover_missing_content(
47-
session: Session, path: str, stat_result: os.stat_result, hashing_is_enabled: bool
47+
session: Session,
48+
path: str,
49+
snapshot: tuple[str, os.stat_result] | None,
50+
hashing_is_enabled: bool,
4851
) -> Literal["recovered", "no_match", "unstable"]:
52+
"""``snapshot`` is ``snapshot_hash(path)``, taken by the caller before its write
53+
transaction opens so the file is never hashed while the write lock is held."""
4954
if not hashing_is_enabled:
5055
return "no_match"
5156
occupied = session.scalar(
@@ -55,7 +60,6 @@ def recover_missing_content(
5560
)
5661
if occupied is not None:
5762
return "no_match"
58-
snapshot = snapshot_hash(path)
5963
if snapshot is None:
6064
if path not in _pending_recovery_paths:
6165
_pending_recovery_paths.append(path)

‎app/assets/services/ingest.py‎

Lines changed: 7 additions & 6 deletions

0 commit comments

Comments
 (0)