fix(local_store): make DocStore.lock() a real mutex on Windows by Varshith-Kali · Pull Request #593 · VectifyAI/PageIndex · GitHub
Skip to content

fix(local_store): make DocStore.lock() a real mutex on Windows - #593

Open
Varshith-Kali wants to merge 1 commit into
VectifyAI:mainfrom
Varshith-Kali:fix/docstore-windows-lock
Open

Varshith-Kali wants to merge 1 commit into
VectifyAI:mainfrom
Varshith-Kali:fix/docstore-windows-lock

Conversation

@Varshith-Kali

Copy link
Copy Markdown

Fixes #561.

What

DocStore.lock() used fcntl.flock() for its cross-process mutex, but fcntl doesn't exist on Windows — the ImportError fallback silently yielded with no lock at all. Concurrent submit_document() calls on Windows could both pass the _unique_doc_name() check and save under the same filename, producing duplicate document names.

The Windows branch now takes a real cross-process lock via msvcrt.locking on the same .lock file, and an in-process threading.Lock keyed by storage directory additionally serializes this process's threads — including separate DocStore instances pointed at the same directory. The name-uniqueness check and the save already ran inside with self._store.lock() in submit_document(), so with a working lock the invariant holds. POSIX behavior (fcntl.flock) is unchanged.

Changes

  • pageindex/local_store.py: cross-platform lock as above (not re-entrant, documented in the docstring).
  • tests/test_local_store.py: regression tests simulating the Windows branch (hidden fcntl, fake msvcrt whose locking() truly serializes):
    • test_lock_windows_branch_serializes_threads: 8 threads x 25 iterations across two DocStore instances on one directory; asserts no lost increments and that the OS-level lock was actually acquired/released each time.
    • test_concurrent_same_name_submits_store_unique_names: the issue's repro — 8 concurrent same-name submit_document() calls receive sample.pdf, sample_1.pdf, ... sample_7.pdf.

Verified

  • Both new tests fail on main (duplicates observed, e.g. Document "sample.pdf" was stored as "sample_2.pdf") and pass with the fix.
  • Full suite: 712 passed; the 76 remaining failures are pre-existing environment issues (no running event loop / network-dependent LLM credential tests), identical counts on unmodified main.
  • The issue also suggests Windows CI coverage: I have a windows-latest job running tests/test_local_store.py ready, but my token lacks the workflow scope needed to push workflow files — happy to add it on request, or it can land as a follow-up.

fcntl is absent on Windows, so lock() silently yielded without acquiring
any lock and concurrent submit_document() calls could race the
check-then-save name-uniquing sequence, storing documents under
duplicate names. The Windows branch now takes a real cross-process lock
via msvcrt.locking, and an in-process lock keyed by storage directory
additionally serializes threads — including separate DocStore instances
aimed at the same directory. POSIX behavior (fcntl.flock) is unchanged.

Adds regression tests simulating the Windows branch (hidden fcntl, fake
msvcrt) covering thread serialization and concurrent same-name submits.
hw725 added a commit to hw725/PageIndex that referenced this pull request Oct 17, 2026
포크 main(d0b02ee)이 upstream 보다 147 커밋 뒤처져, upstream/main(f0db746) 에서
브랜치를 새로 따고 커스텀을 새 구조에 맞춰 다시 넣었다. upstream 은 그 사이
OpenAI SDK 분기를 없애고 litellm 단일 경로로 바꿨고, 재시도 소진 시 빈 문자열 대신
LLMRetriesExhausted 를 올린다. 커스텀 네 가지는 upstream 에 여전히 없다.

- 빈 본문 → EmptyLLMResponseError, 깨진 JSON·키 누락 → MalformedLLMResponseError 로
  올려 같은 재시도 사다리에 태운다. expect_json_keys 는 옵트인(호출부 5곳).
  키 검사는 존재만 본다 — {"physical_index": null} 은 응답이다. 예전처럼 null 까지
  재시도하면 이제는 사다리 소진 예외로 실행 전체가 죽는다.
- 이벤트 루프별 세마포어(PAGEINDEX_MAX_CONCURRENCY=12), timeout(PAGEINDEX_LLM_TIMEOUT=90,
  backend·호출부가 주면 그쪽이 이김), 지수 백오프(상한 10초).
- OPENAI_BASE_URL 이 로컬인데 꺼져 있으면 npx openai-oauth --port <포트> 를 띄운다.
- 기본 모델 gpt-5.6-terra(DEFAULT_INDEX_MODEL·DEFAULT_CHAT_MODEL). config.yaml 은 upstream
  그대로 둔다 — 거기 모델 키를 적으면 구체 키가 사용자 model= 보다 우선해 《model 은 모든
  역할을 정한다》 규칙이 깨진다(upstream 시험 9건이 잡았다). 예전 summary_model=luna 는
  terra 로 합쳐졌다.
- run_pageindex.py cp949 콘솔 대응, page_index_md 데모의 gpt-4.1 하드코딩 제거.

Windows 저장소 잠금(local_store.DocStore.lock): fcntl 이 없으면 잠금이 아무 일도 하지
않아, 같은 이름 PDF 두 개를 동시에 색인하면 둘 다 sample.pdf 로 저장돼 앞 문서를 가렸다
(upstream test_concurrent_same_name_submits_store_unique_names 가 Windows 에서 실패).
msvcrt 바이트 잠금으로 직렬화한다. upstream 에 같은 수정 PR 이 이미 열려 있다
(VectifyAI#593·VectifyAI#567·VectifyAI#456) — 그중 하나가 병합되면 다음 동기화 때 이 수정을 빼고 upstream 것을 쓴다.

검증: harness.contract.json C-001~C-003 (영수증 .harness/v2).
  pytest 799 passed / 0 failed(Windows 가 테스트 준비 자체를 허용하지 않는 4건 제외 —
  symlink 권한, 이름에 \ · " · CRLF 경로. 순정 upstream 에서도 실패한다).
  tests/test_fork_robustness.py 20건. scripts/fork_smoke.py — earthmover.pdf standard 모드 실제 LLM.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(local_store): DocStore.lock() no-op on Windows causes race condition and duplicate document names

1 participant