{{ message }}
Repository navigation
Conversation
zzet
force-pushed
the
fix/incremental-index-write-amplification
branch
from
September 15, 2026 06:40
303cbfe to
aff1d47
Compare
zzet
marked this pull request as draft
September 25, 2026 11:10
zzet
marked this pull request as ready for review
October 2, 2026 16:55
This was referenced Oct 2, 2026
Closed
Keep immutable raw records and coherent receipt proof clocks in a bounded actor cache. Derive subsequent extraction frontiers from selected receipt and census changes while preserving recorded-read convergence and fresh enrichment evidence.
Report whether actor checkpoints survive metadata validation and bounded cache admission. Include charged sizes and bounded extraction frontiers so rebuild mode changes can be traced without changing retention policy.
The first full-corpus checkpoint exceeded the global 64 MiB budget, mostly through eager singleton declaration maps and repeated proof strings. Retain sorted forward declarations and build inverse postings only for queried keys, with full forward lookup and cached-miss updates. Retain receipt digests and normalized query keys, canonicalize owned proof strings per checkpoint, and charge shared ledger maps once after explicitly preserving their alias. Nil maps carry no backing-table charge. The cache budget and dependency proof checks remain unchanged. Validate cached negative queries, multiple producers, immutable ownership, receipt value changes, and real worker delta replay against cold results.
The compact full-corpus checkpoint still exceeded the global 64 MiB budget. Its immutable owner and query reader indexes retained a hash table for each posting list. Store every posting in sorted unique slices with owned exact-capacity backing. Preserve the planner's owner and negative-query dependency edges, the accounting rules, and the global budget. Verify multiple owners and readers, a new negative-query producer, removed owners, replacement rebuilding, and independent old postings.
peterkc
added a commit
to peterkc/gortex
that referenced
this pull request
Oct 20, 2026
3 of 7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Small source edits could wait behind graph construction and contract completion. This PR allows guarded text edits and whole-file writes to proceed from verified checkout source bytes when the request does not require a fresh graph. Relationship queries retain their graph-freshness checks.
Write admission preserves checkout/root/HEAD identity, accepted ancestry, source and target guards, path serialization, parse checks and mutation ownership. Whole-file writes retain their existing optional base SHA and unconditional create/overwrite behavior. Optional admission can retry one ancestry-pin race before taking ownership; the write handler is never replayed. Explicit graph requirements, overlays and unsupported source-authority cases keep their existing admission path.
Incremental indexing publishes the requested repository first and updates companion repositories asynchronously. Core indexing no longer waits for contract completion. Contract-source projections and valid extraction work are reused, obsolete semantic attempts can be cancelled, base-edge scans stay kind-selective, and symbol-search reuse retains scoring and invalidation. Structural reparses preserve value references, external calls, builtins and unresolved-reference proofs. Retired views respect active leases. Upstream main 76c080e is incorporated, including shared disk walking and origin/seed timing.
Post-commit refresh admission creates an owned completion ticket before whole-checkout capture. The coordinator performs capture and publication asynchronously; completion proves the committed bytes or deletions in the actual published sample. Supersession, shutdown and ABA protections remain enforced. This fixes the observed failure where a disk write committed but synchronous checkout capture timed out before indexing was queued.
Retired worktree routes decline optional byte admission before coordinator activation. This preserves the existing bounded
view_buildingrefusal when no coordinator exists; both MCP interfaces verify that neither checkout is written. Completion tests also require surviving edits in different files to report the same verified publication, while same-file supersession and ABA checks remain strict.Published candidate: ae1319b. Full local acceptance passed; CI and manual review remain pending.
The exact clean default macOS ARM64 CGO binary completed the original 14-stage campaign across 29 repositories and four linked worktrees: both 30-minute workloads, eight-agent capacity, separate 16-agent overload, restart, integrity, normal shutdown and fixture/source/process preservation. The three ordinary timed lanes recorded 93,630 correct operations, five separately counted pending observations, zero failures and zero inconclusive checks. The 16-agent diagnostic retained eight
server_busyworker errors: six initialization failures and two later RPC failures. Those two timed RPC failures are included in eight, not additional normal-load failures.The ACK union retained 7,532 acknowledged writes, zero unknown deliveries, timestamp ties or missing artifacts. Ninety surviving snapshots matched; 15 were superseded by later committed deletion ACKs. All 292 restart outcomes passed, including 112 original checks and 180 source/graph checks. The reader's static manual-review label remains; a finite peer audit reviewed the concrete exceptions and found no new correctness blocker. Prepared isolation checks remain separate from the ACK reader's source/graph assertions.
Selected separate wire cohorts retain material tails:
The N310 symbols cohort reached 6,105.613 ms maximum; actual WT symbols reached 6,337.286 ms. Unrecorded actual service proves neither PRIMARY service, WT service nor fallback. The capacity write cohorts compare with historical b55 p95/p99 of 2,446.631/2,446.631 ms (actual WT) and 3,014.104/3,329.980 ms (PRIMARY requested, actual unrecorded). Small samples and separate-run host/time/case-mix differences prevent causal claims. Other unfavorable rows remain in the reports. There is no pooled or blanket latency claim, universal five-second guarantee, or current-head native recertification of all 259 operations/21 facades.
Focused normal/race checks and golangci-lint v2.13.1 passed; permanent regressions remain in CI. At 13:31 UTC on 21 October, current-head PR checks were 51 successful, four running, one queued, zero failed. Ubuntu and Windows were complete. A later main-run snapshot has 48 successful, three running, one queued and zero failed checks; MCP is now complete and only macOS indexer partitions 0/2/3/4 remain. The earlier full PR rollup is retained until independently refreshed. The interrupted 8f predecessor full run retains no acceptance credit.
A separate manual runtime is prepared and not started. Remaining readiness work is CI completion and manual acceptance, with the symbols tail disclosed. Expanded fault recovery, local-AI task-loop correctness and MCP response-state presentation remain agreed nonblocking followups. No merge, tag or release is implied.