Repository navigation
[Tracking] Knowledge PR admission: model and effort policy #50
Description
Goal: filter ineligible agent-authored knowledge contributions before costly quality review.
Owner decision: knowledge must be materially produced by approved models at xhigh or max. An approved-model review/signature cannot qualify another model's output.
Status (2026-10-24): #98 is merged; the stale-success hardening is delivered. The owner approved the minimal required-check rollout without an additional synthetic noncompliant PR. Administrator configuration and verification remain pending; enforcement has not been confirmed. See the current rollout delta below.
Current rollout delta — 2026-10-24
This section supersedes the older checklist's demand for a new synthetic merge-blocking demonstration; the historical requirements and evidence below are retained for traceability. Event-delivery delays, unavailable GitHub reads/status writes, and the final-read/publication race remain bounded implementation limitations, not eliminated guarantees. A successful declaration status still does not establish knowledge quality or skill efficacy.
Admission path
flowchart LR
A["Protected-scope PR"] --> H{"Declared human-only?"}
H -- Yes --> J["No model allowlist; content review still required"]
H -- No / mixed --> B{"Effort: xhigh or max?"}
B -- No / unknown --> X["Block"]
B -- Yes --> C{"Model on allowlist?"}
C -- No / unknown --> X
C -- Yes --> D["Declaration check passes"]
D --> E["Content and source review"]
J --> E
E --> F["Skill Eval where required"]
A green per-PR knowledge-model-policy status means declaration compliance only, not proven execution settings or correct knowledge. A green sweep workflow means the scan completed successfully; individual PRs can still be rejected. Existing quality and merge gates still apply.
Policy
| Field | Requirement |
|---|---|
| Model allowlist | Claude Opus 5.5 · GPT-6-Astra · GPT-6.1 Sol |
| Effort | Accept only {xhigh, max}; use set membership, not string ordering. Other values fail by default. |
| Attribution | Tool/agent, exact model ID, effort, role, covered head; disclose every material author |
| Mixed models | Every material knowledge author must qualify; review, signing or cosmetic rewriting does not replace authorship |
| Human-only | Explicitly declare no material AI involvement; model allowlist does not apply. Content/source review still applies. Mixed contributions disclose every material AI author. |
| Mechanical / generated | No automatic exemption in v1. Running a generator does not erase AI authorship. Any exception needs an explicit approval bound to PR, head, scope and reason. |
| Unknown values | Disclose truthfully; they do not satisfy knowledge admission |
| Evidence | v1 checks self-declarations. Trusted executor attestations are a separate future enhancement |
| History | Prospective policy; do not invent identities or rewrite old commits |
Verified canonical IDs, explicit aliases, effort sets and primary-provider sources are recorded in the authoritative policy documentation and trusted configuration. The checker resolves model identity to interpret effort even though the diagram presents effort first.
Proposed trailer: Agent-assisted-by: <tool> / <model-id> [effort=<level>] (<role>).
Use one validated structured PR-body declaration for CI; human DCO remains separate. Never request private session logs, secrets or hidden reasoning.
Implementation checklist
- Confirm canonical IDs/aliases and implement the human-only and exception handling below; no inferred bypass.
- Update
docs/agent-collaboration.mdand PR template: model/version is currently optional. - Detect protected changes using an explicit path list:
src/inferrsi/data/skill_library/**,src/inferrsi/data/knowledge_packs/**,src/inferrsi/data/repo_packs/**/knowledge/**;docs/reference-matrix.md,docs/searchspace-matrix.md; knowledge contractsdocs/knowledge-base.md,docs/knowledge-operations.md,docs/knowledge-store.md; readers/loaderssrc/inferrsi/catalog.py,src/inferrsi/skill_catalog.py,src/inferrsi/knowledge_operations.py. Do not blanket-match all ofsrc/. Changes to this list require explicit policy review. - Add
knowledge-model-policy: default-deny allowlist + explicit effort set; optional explicit suspension overrides allowlisting. - Use trusted target-branch checker/policy. Include deletions and both rename paths; incomplete API results fail visibly.
- Refresh on commits/body edits/relevant base changes; bind declarations and results to current head. Always report the stable check, including non-knowledge “not applicable”.
- Test missing/invalid/mixed attribution, aliases, stale heads, renames, API pagination and attempts to relax policy inside the PR.
- Deploy report-only rollout: noncompliance is a failed policy status; the check is not yet required for merging. Real head/body refresh and status publication verified in test-only: verify knowledge policy lifecycle #63.
- Owner final acceptance; separately authorize and enable required
knowledge-model-policystatus formain, then verify repository rules and close this issue. Report-only is a rollout stage, not the final admission policy.
Safety and review boundaries
No privileged execution of PR code. If metadata events use pull_request_target, run only trusted base code, treat PR text as data, and use minimal permissions. Control-path changes receive ordinary content/agent review; #86 removed the separate machine policy-change approval protocol. policy_paths remains classification/reporting. Explicit knowledge exceptions still require authorized approval; a PR cannot grant itself an exception.
Keep review provenance separate from author declarations. A role switch is not independent review; a shared account alone neither proves nor disproves agent independence. New commits require refreshed coverage. Handle merge-queue/base-policy refresh explicitly if applicable.
Eligibility reduces review load but does not replace source accuracy, semantic fidelity, installed-reader checks or task-efficacy evaluation (#40). Model declarations do not automatically promote candidate knowledge.
Ownership and lifecycle
| Item | Plan |
|---|---|
| Implementation owner | @Dong1017 — Codex accountable agent, implementing under the owner's task-specific delegation. Final acceptance remains separate. |
| Exception approver | @Dong1017. Exceptions to their own contributions require a separately designated independent maintainer; no self-approval. |
| Final acceptance | @Dong1017 or an explicitly named delegate |
| While implementing | Keep open; maintain checklist and current decisions in this body |
| Once operational | Put authoritative rules in repository docs/config, link them here, then close; optionally pin as an entry point |
| Later policy changes | Bounded follow-up issues; preserve decision history rather than silently rewrite past evidence |
2026-10-21: qualified authorship confirmed; three models selected; effort corrected from the earlier wording to the allowed set {xhigh, max}. Issue publication does not authorize implementation, credentials, repository-rule changes or merging.
2026-10-21 clarification (owner confirmed): human-only declarations bypass the model allowlist, not content review; mixed material AI authors remain fully disclosed. Mechanical/generated changes receive no automatic exemption. Contract and reader paths are protected explicitly. Exceptions record PR/head/scope/reason and approval; a changed head requires renewed coverage. Report-only rollout precedes separately authorized required-check enforcement. @Dong1017 owns exception approval and final acceptance, with independent approval for their own exceptions.
Current implementation and remaining acceptance — 2026-10-22
| Item | Current state |
|---|---|
| Lifecycle and reporting correction | #86 merged as a1ee326, addressing #81's report-only lifecycle/reporting defects |
| Control-path review | Separate machine policy-change approval removed; classification retained. Model/effort admission and explicit exception governance remain unchanged |
| Live scan | New-main run: #31/#87 compliant, #51/#55 noncompliant, execution=ok |
| Previously misleading red jobs | #87 reruns 39741039321 and 39741260862 succeeded under new main without turning #51/#55 green |
| Candidate and integration CI | Candidate CI: both Python versions 432 passed, 9 skipped, 70 subtests passed; merge lane passed |
| Acceptance boundary | These observations verify reporting separation, not every failure mode or actual merge blocking. #81 remains separately subject to its closure acceptance |
Required-check readiness: still open
- Resolve and validate the stale-success risk for required enforcement: unreadable target metadata or publication-time exception-approval reads can leave an earlier success until retry. The checker reports uncertainty and uses bounded retry, but does not guarantee universal fail-closed invalidation.
- Review final-read/status-POST race handling and document the enforcement contract; observed closure/head changes already suppress stale publication.
- Obtain final owner acceptance and separate authorization to enable required
knowledge-model-policy. - Configure/verify the trusted publishing source and repository rules, then demonstrate that a genuinely noncompliant PR cannot merge. Workflow execution success alone must not satisfy admission.
- Publish a visual acceptance summary and close only after these conditions are met.
No repository protection settings were changed by #86 integration or these reruns. Historical failed attempts remain evidence; this is not authorization to bulk-green closed PRs. The affected historical heads are tracked in #81. The historical #68 failure must be read alongside its factual declaration correction and accepted #48 record, not treated as a current uncorrected authorship rejection.
Historical implementation and validation records — 2026-10-21 (superseded where stated above)
Operational evidence — 2026-10-21
- Implementation: PR #62, merged as
68c73977c87c0f87affe0cb4f4ca1d9a5ef22eca. Authoritative behavior now lives in the repository policy document, configuration and workflow. - Ready CI: Python 3.10 and 3.11 each passed 363 tests and 70 subtests, 9 skipped, Ruff and clean installed-wheel/reader checks passed. Local existing Windows environment passed 76 targeted tests. Independent delegated source review closed its concrete findings before merge.
- Post-merge base CI passed. First trusted-main policy sweep evaluated open PRs and wrote actual head statuses; PR Record MiniCPM-o throughput search, quality gates and peer baselines #51 exercised complete multi-page changed-file retrieval. Its red result reflects missing declarations in protected PRs, not an API/checker failure.
- Validation PR #63, closed without merging, verified non-protected success, protected missing-declaration failure, current-head unknown-model failure, and stale declaration failure after body-only edits. Exact heads and run receipts are in that PR. No eligible authorship or independent exception approval was fabricated. Positive admission/exception cases are automated-test evidence, distinct from these live rejection fixtures.
- Repository required-check settings remain unchanged. No merge queue is configured; queue integration must precede any future queue activation. A separately designated independent approver is still needed for exceptions/policy changes involving @Dong1017's material contribution.
Implementation and publication were explicitly delegated by the owner in the task session. Final closure is held for the requested owner check; this issue remains open.
Acceptance hold — shared-head statuses
Reviewer identified that commit statuses are keyed by SHA/context rather than PR. At bootstrap revision 68c73977c87c0f87affe0cb4f4ca1d9a5ef22eca, a compliant PR sharing a head with an ineligible PR can overwrite its failure with success. Earlier CI and individual live-fixture observations remain valid; they do not cover this orchestration defect.
- Reject ambiguous heads shared by multiple open PRs, including other target branches; refresh on their lifecycle changes. Implemented in fix: prevent shared-head knowledge admission overwrite #64.
- Add API-orchestration regressions covering both PR orders, PR-bound exception approval, and metadata/lifecycle refresh. Real HTTP adapter and actual status POSTs are exercised by fixtures.
- Repeat report-only lifecycle regression after correction; actual head-status receipts are in test-only: shared-head policy lifecycle (main) #65/test-only: shared-head policy peer (other base) #66.
- Reviewer/owner re-acceptance. Required-check enforcement, trusted publishing-source configuration and actual merge-blocking validation remain pending separate owner authorization.
Corrective implementation and fresh evidence — 2026-10-21
PR #64 is merged as 0032649b8d8e68f709c2a50e9fa31aa3f866dffb. A commit shared by multiple open repository PRs is ineligible, across target branches, regardless of a single PR's declaration or exception. Results are published only after a second complete PR-list scan revalidates uniqueness and admission metadata; approval comments are reread. Whole-collection requests are bounded to two scans per refresh. Closed and cross-base PR events refresh current main-target PRs.
- Current corrective-head CI, head
dbc56287c1f2aea384b6e34b8fa59a4c0ccebc49: Python 3.10 and 3.11 each 373 passed, 9 skipped, 70 subtests passed; Ruff and clean wheel/installed-reader checks passed. The merge tree matches this head. Local Windows policy tests: 73 passed. - API-orchestration fixtures cover compliant/ineligible PRs in both orders, an exception bound to only one PR, cross-base/paginated peers, close/reopen/head/body refresh, a peer appearing during evaluation, and earlier-PR body/approval changes during later evaluation. Six main-target PRs across two collection pages require exactly four collection GETs, addressing the automated review's repeated-scan finding.
- Independent delegated source re-review by
policy_reviewfound no remaining actionable issue at the exact corrective head. GitHub automatic review also completed on that head without new findings. These reviews do not replace final owner acceptance. - Post-merge base CI passed. The new main-policy sweep preserved non-applicable success and missing-declaration failure on existing PRs; its aggregate red result reflects those policy failures.
Fresh live fixture: #65 targets main; #66 targets another task-owned branch. Both use head 37103ad25f37d539c909e27d387fc5fe0f49b487 and the same non-protected root-file change.
| Lifecycle | Actual shared head status |
|---|---|
| Only #65 open | Success: Not applicable |
| #66 opens against another base | Failure: shared head |
| #66 closes, #65 stays open | Success: Not applicable |
| #66 reopens | Failure: shared head |
Both failure logs explicitly report that shared heads are ineligible. These are the actual stable commit-status receipts from github-actions[bot], not inference from the workflow's aggregate conclusion. Both fixtures are closed without merging. Positive model/human/exception admission scenarios remain automated-test evidence; this live test covers head uniqueness and cross-base lifecycle refresh.
Required-check settings were not changed; the current main ruleset has no required status check. Keep this issue open for re-acceptance, then separate authorization/configuration and verification of the required trusted check and actual merge blocking before closure.
Activity
Dong1017 commentedon Oct 24, 2026
@zengchuang-hw could you help complete the administrator-only rollout for #50? #98 is merged. The owner approved the minimal configuration change below; our current GitHub account can view but not edit repository rules.
knowledge-model-policybefore merging into main. Do not substitute the sweep workflow/job nameddeclarations: a successful sweep is not a successful admission verdict.Validation scope: no extra synthetic noncompliant PR, workflow dispatch or model experiment is requested. The owner explicitly deferred the synthetic merge-blocking test; record it as not run, not passed. Keep #50 open until the configuration receipts are reviewed; configuration alone does not establish knowledge quality or eliminate event/publication race windows.
Known operational caveat: #31 currently fails this status with
Status history could not be read(review). Once required, this infrastructure failure will block ordinary merges too. Do not clear it by manually publishing green, changing the declaration, weakening the check or adding a bypass; diagnose it separately if it persists.