{{ message }}
Repository navigation
fix(local_store): make DocStore.lock() a real mutex on Windows - #593
Open
Varshith-Kali wants to merge 1 commit into
Open
Varshith-Kali wants to merge 1 commit into
Varshith-Kali wants to merge 1 commit into
Conversation
fcntl is absent on Windows, so lock() silently yielded without acquiring any lock and concurrent submit_document() calls could race the check-then-save name-uniquing sequence, storing documents under duplicate names. The Windows branch now takes a real cross-process lock via msvcrt.locking, and an in-process lock keyed by storage directory additionally serializes threads — including separate DocStore instances aimed at the same directory. POSIX behavior (fcntl.flock) is unchanged. Adds regression tests simulating the Windows branch (hidden fcntl, fake msvcrt) covering thread serialization and concurrent same-name submits.
hw725
added a commit
to hw725/PageIndex
that referenced
this pull request
Oct 17, 2026
포크 main(d0b02ee)이 upstream 보다 147 커밋 뒤처져, upstream/main(f0db746) 에서 브랜치를 새로 따고 커스텀을 새 구조에 맞춰 다시 넣었다. upstream 은 그 사이 OpenAI SDK 분기를 없애고 litellm 단일 경로로 바꿨고, 재시도 소진 시 빈 문자열 대신 LLMRetriesExhausted 를 올린다. 커스텀 네 가지는 upstream 에 여전히 없다. - 빈 본문 → EmptyLLMResponseError, 깨진 JSON·키 누락 → MalformedLLMResponseError 로 올려 같은 재시도 사다리에 태운다. expect_json_keys 는 옵트인(호출부 5곳). 키 검사는 존재만 본다 — {"physical_index": null} 은 응답이다. 예전처럼 null 까지 재시도하면 이제는 사다리 소진 예외로 실행 전체가 죽는다. - 이벤트 루프별 세마포어(PAGEINDEX_MAX_CONCURRENCY=12), timeout(PAGEINDEX_LLM_TIMEOUT=90, backend·호출부가 주면 그쪽이 이김), 지수 백오프(상한 10초). - OPENAI_BASE_URL 이 로컬인데 꺼져 있으면 npx openai-oauth --port <포트> 를 띄운다. - 기본 모델 gpt-5.6-terra(DEFAULT_INDEX_MODEL·DEFAULT_CHAT_MODEL). config.yaml 은 upstream 그대로 둔다 — 거기 모델 키를 적으면 구체 키가 사용자 model= 보다 우선해 《model 은 모든 역할을 정한다》 규칙이 깨진다(upstream 시험 9건이 잡았다). 예전 summary_model=luna 는 terra 로 합쳐졌다. - run_pageindex.py cp949 콘솔 대응, page_index_md 데모의 gpt-4.1 하드코딩 제거. Windows 저장소 잠금(local_store.DocStore.lock): fcntl 이 없으면 잠금이 아무 일도 하지 않아, 같은 이름 PDF 두 개를 동시에 색인하면 둘 다 sample.pdf 로 저장돼 앞 문서를 가렸다 (upstream test_concurrent_same_name_submits_store_unique_names 가 Windows 에서 실패). msvcrt 바이트 잠금으로 직렬화한다. upstream 에 같은 수정 PR 이 이미 열려 있다 (VectifyAI#593·VectifyAI#567·VectifyAI#456) — 그중 하나가 병합되면 다음 동기화 때 이 수정을 빼고 upstream 것을 쓴다. 검증: harness.contract.json C-001~C-003 (영수증 .harness/v2). pytest 799 passed / 0 failed(Windows 가 테스트 준비 자체를 허용하지 않는 4건 제외 — symlink 권한, 이름에 \ · " · CRLF 경로. 순정 upstream 에서도 실패한다). tests/test_fork_robustness.py 20건. scripts/fork_smoke.py — earthmover.pdf standard 모드 실제 LLM. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Fixes #561.
What
DocStore.lock()usedfcntl.flock()for its cross-process mutex, butfcntldoesn't exist on Windows — theImportErrorfallback silently yielded with no lock at all. Concurrentsubmit_document()calls on Windows could both pass the_unique_doc_name()check and save under the same filename, producing duplicate document names.The Windows branch now takes a real cross-process lock via
msvcrt.lockingon the same.lockfile, and an in-processthreading.Lockkeyed by storage directory additionally serializes this process's threads — including separateDocStoreinstances pointed at the same directory. The name-uniqueness check and the save already ran insidewith self._store.lock()insubmit_document(), so with a working lock the invariant holds. POSIX behavior (fcntl.flock) is unchanged.Changes
pageindex/local_store.py: cross-platform lock as above (not re-entrant, documented in the docstring).tests/test_local_store.py: regression tests simulating the Windows branch (hiddenfcntl, fakemsvcrtwhoselocking()truly serializes):test_lock_windows_branch_serializes_threads: 8 threads x 25 iterations across twoDocStoreinstances on one directory; asserts no lost increments and that the OS-level lock was actually acquired/released each time.test_concurrent_same_name_submits_store_unique_names: the issue's repro — 8 concurrent same-namesubmit_document()calls receivesample.pdf,sample_1.pdf, ...sample_7.pdf.Verified
Document "sample.pdf" was stored as "sample_2.pdf") and pass with the fix.windows-latestjob runningtests/test_local_store.pyready, but my token lacks theworkflowscope needed to push workflow files — happy to add it on request, or it can land as a follow-up.