Improve incremental indexing and resumable contract analysis by zzet · Pull Request #806 · zzet/gortex · GitHub
Skip to content

Improve incremental indexing and resumable contract analysis - #806

Open
zzet wants to merge 1067 commits into
mainfrom
fix/incremental-index-write-amplification
Open

zzet wants to merge 1067 commits into
mainfrom
fix/incremental-index-write-amplification

Conversation

@zzet

@zzet zzet commented Sep 14, 2026 •

Copy link
Copy Markdown
Owner

Small source edits could wait behind graph construction and contract completion. This PR allows guarded text edits and whole-file writes to proceed from verified checkout source bytes when the request does not require a fresh graph. Relationship queries retain their graph-freshness checks.

Write admission preserves checkout/root/HEAD identity, accepted ancestry, source and target guards, path serialization, parse checks and mutation ownership. Whole-file writes retain their existing optional base SHA and unconditional create/overwrite behavior. Optional admission can retry one ancestry-pin race before taking ownership; the write handler is never replayed. Explicit graph requirements, overlays and unsupported source-authority cases keep their existing admission path.

Incremental indexing publishes the requested repository first and updates companion repositories asynchronously. Core indexing no longer waits for contract completion. Contract-source projections and valid extraction work are reused, obsolete semantic attempts can be cancelled, base-edge scans stay kind-selective, and symbol-search reuse retains scoring and invalidation. Structural reparses preserve value references, external calls, builtins and unresolved-reference proofs. Retired views respect active leases. Upstream main 76c080e is incorporated, including shared disk walking and origin/seed timing.

Post-commit refresh admission creates an owned completion ticket before whole-checkout capture. The coordinator performs capture and publication asynchronously; completion proves the committed bytes or deletions in the actual published sample. Supersession, shutdown and ABA protections remain enforced. This fixes the observed failure where a disk write committed but synchronous checkout capture timed out before indexing was queued.

Retired worktree routes decline optional byte admission before coordinator activation. This preserves the existing bounded view_building refusal when no coordinator exists; both MCP interfaces verify that neither checkout is written. Completion tests also require surviving edits in different files to report the same verified publication, while same-file supersession and ABA checks remain strict.

Published candidate: ae1319b. Full local acceptance passed; CI and manual review remain pending.

The exact clean default macOS ARM64 CGO binary completed the original 14-stage campaign across 29 repositories and four linked worktrees: both 30-minute workloads, eight-agent capacity, separate 16-agent overload, restart, integrity, normal shutdown and fixture/source/process preservation. The three ordinary timed lanes recorded 93,630 correct operations, five separately counted pending observations, zero failures and zero inconclusive checks. The 16-agent diagnostic retained eight server_busy worker errors: six initialization failures and two later RPC failures. Those two timed RPC failures are included in eight, not additional normal-load failures.

The ACK union retained 7,532 acknowledged writes, zero unknown deliveries, timestamp ties or missing artifacts. Ninety surviving snapshots matched; 15 were superseded by later committed deletion ACKs. All 292 restart outcomes passed, including 112 original checks and 180 source/graph checks. The reader's static manual-review label remains; a finite peer audit reviewed the concrete exceptions and found no new correctness blocker. Prepared isolation checks remain separate from the ACK reader's source/graph assertions.

Selected separate wire cohorts retain material tails:

Lane and proven/requested scope N p95 ms p99 ms
Worktree-only symbols, actual WT service 7,310 1,005.663 1,681.629
Worktree-only symbols, PRIMARY requested; actual service unrecorded 310 3,706.297 5,356.567
Worktree-only whole-file-write ACK, actual WT service 667 501.307 989.292
Eight-agent whole-file-write ACK, actual WT service 18 710.829 710.829
Eight-agent whole-file-write ACK, PRIMARY requested; actual service unrecorded 22 1,329.814 1,896.100

The N310 symbols cohort reached 6,105.613 ms maximum; actual WT symbols reached 6,337.286 ms. Unrecorded actual service proves neither PRIMARY service, WT service nor fallback. The capacity write cohorts compare with historical b55 p95/p99 of 2,446.631/2,446.631 ms (actual WT) and 3,014.104/3,329.980 ms (PRIMARY requested, actual unrecorded). Small samples and separate-run host/time/case-mix differences prevent causal claims. Other unfavorable rows remain in the reports. There is no pooled or blanket latency claim, universal five-second guarantee, or current-head native recertification of all 259 operations/21 facades.

Focused normal/race checks and golangci-lint v2.13.1 passed; permanent regressions remain in CI. At 13:31 UTC on 21 October, current-head PR checks were 51 successful, four running, one queued, zero failed. Ubuntu and Windows were complete. A later main-run snapshot has 48 successful, three running, one queued and zero failed checks; MCP is now complete and only macOS indexer partitions 0/2/3/4 remain. The earlier full PR rollup is retained until independently refreshed. The interrupted 8f predecessor full run retains no acceptance credit.

A separate manual runtime is prepared and not started. Remaining readiness work is CI completion and manual acceptance, with the symbols tail disclosed. Expanded fault recovery, local-AI task-loop correctness and MCP response-state presentation remain agreed nonblocking followups. No merge, tag or release is implied.

@zzet
zzet force-pushed the fix/incremental-index-write-amplification branch from 303cbfe to aff1d47 Compare September 15, 2026 06:40
@zzet
zzet marked this pull request as draft September 25, 2026 11:10
@zzet zzet changed the title Incremental indexing: reuse committed generations and bound write amplification across worktrees Improve incremental indexing, scoped freshness, and symbol search Oct 2, 2026
@zzet
zzet marked this pull request as ready for review October 2, 2026 16:55
zzet added 20 commits October 15, 2026 20:21
Keep immutable raw records and coherent receipt proof clocks in a bounded actor cache. Derive subsequent extraction frontiers from selected receipt and census changes while preserving recorded-read convergence and fresh enrichment evidence.
Report whether actor checkpoints survive metadata validation and bounded
cache admission. Include charged sizes and bounded extraction frontiers so
rebuild mode changes can be traced without changing retention policy.
The first full-corpus checkpoint exceeded the global 64 MiB budget,
mostly through eager singleton declaration maps and repeated proof
strings. Retain sorted forward declarations and build inverse postings
only for queried keys, with full forward lookup and cached-miss updates.

Retain receipt digests and normalized query keys, canonicalize owned
proof strings per checkpoint, and charge shared ledger maps once after
explicitly preserving their alias. Nil maps carry no backing-table
charge. The cache budget and dependency proof checks remain unchanged.

Validate cached negative queries, multiple producers, immutable ownership,
receipt value changes, and real worker delta replay against cold results.
The compact full-corpus checkpoint still exceeded the global 64 MiB
budget. Its immutable owner and query reader indexes retained a hash
table for each posting list.

Store every posting in sorted unique slices with owned exact-capacity
backing. Preserve the planner's owner and negative-query dependency
edges, the accounting rules, and the global budget.

Verify multiple owners and readers, a new negative-query producer,
removed owners, replacement rebuilding, and independent old postings.
@zzet zzet changed the title Improve incremental indexing, scoped freshness, and symbol search Improve incremental indexing and resumable contract analysis Oct 16, 2026
peterkc added a commit to peterkc/gortex that referenced this pull request Oct 20, 2026
Carry the partition workflow and both scripts from zzet#806 unchanged.
Original work by Andrey Kumanyaev in d4698c8, c61c05a, 1eeeae0,
8e438dd, 4c6b803, 18eb3a5 and 839b3de.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants