Sunbelt Computer Software

Backstitch

Spec-to-code traceability and invariant checks for Python repositories.

$ uv tool install backstitch
$ cd your-project
$ mkdir -p docs/specs docs/plans src
$ backstitch check --spec-root docs/specs --code-root src

Backstitch verifies that spec requirements point to real implementation owners, code points back to the requirements it implements, and declared invariants are bound to real tests. It runs as two lanes under one policy layer:

Trace lane — deterministic facts about declared relationships. No model, byte-stable reports, hard-gate by default.
Semantic lane — bounded model review of finite, reproducible packets through the llm ecosystem, without open-ended repository access. Advisory by default; every verdict is evidence-bound and cached as a frozen, replayable artifact.
Policy layer — repository configuration decides which findings, at which evidence class, become blocking. The decision to block is always a reviewable config diff, never a model output.

Backstitch does not promise that models prove correctness. It promises that semantic review becomes repeatable enough to run in CI, diffable enough to audit, and structured enough to improve. The workflow stays deterministic even where one lane uses probabilistic judgment, because everything around the judgment is contract: what evidence the model sees, which requirement it judges, which prompt and model produced the verdict, what shape must come back, how uncertainty is represented, and how severities map to exit codes.

Backstitch is a standalone tool. Weft is a reference target and eventual consumer, not a package dependency.

Recommended For

Spec-driven Python projects. Stable requirement IDs and reciprocal links make intended behavior navigable from docs to code and back.
Repositories maintained by coding agents. Deterministic ownership, diagnostics, and acceptance gates replace prose-only compliance claims.
Critical behavior that needs explicit test bindings. Required and draft invariants make missing, malformed, duplicate, or misplaced bindings visible.
Teams that want bounded semantic review. Backstitch generates finite, reproducible packets before any model call and validates returned evidence.

Features

Bidirectional traceability - Resolves spec mappings and Python backlinks
Invariant checking - Connects required or draft guarantees to real tests
Stable diagnostics - Canonical names, short aliases, contexts, and levels
Configurable policy - Ordered selectors, fail_on, and suppressibility
Documented suppression - Governed exceptions require spec declarations and remain auditable with their rationales
Deterministic reports - Stable JSON, ordering, locators, and content hashes
Bounded semantic review - Section and invariant packets with evidence limits
Runtime-independent parsing - CommonMark and modern Python syntax support
Self-application - Backstitch's own repository is a strict acceptance corpus

Installation

# Install the CLI globally with uv (recommended)
uv tool install backstitch

# Or with pipx
pipx install backstitch

# Or add it to a project environment
uv add --dev backstitch
pip install backstitch

For a source checkout:

$ uv sync --extra dev
$ uv run backstitch --version
backstitch 0.3.0

Requirements:

Python 3.11+
llm, markdown-it-py, tree-sitter, and tree-sitter-python (installed automatically)

Repository commands currently require a POSIX system, including Linux and macOS, with the no-follow descriptor primitives defined in EVC-8.2; Windows is not supported.

Quick Start

For a project with application code under src, create the scan roots and add the repository profile:

$ mkdir -p docs/specs docs/plans src tests

# pyproject.toml
[tool.backstitch]
default_command = "check"

[tool.backstitch.profile]
name = "backstitch-style-v1"
spec_roots = ["docs/specs"]
plan_roots = ["docs/plans"]
code_roots = ["src", "tests"]
test_roots = ["tests"]

Give a requirement a stable ID and map it to an exact repository-relative implementation path:

## Authentication boundary [AUTH-1]

The request path must reject unauthenticated callers.

_Implementation mapping_:
- `src/auth.py`

Add the reciprocal backlink in the owning Python docstring or a nearby comment:

def authenticate(request: Request) -> User:
    """Authenticate one request.

    Spec: docs/specs/auth.md [AUTH-1]
    """

Run the deterministic checker from the repository root:

$ backstitch obligation list
$ backstitch obligation docs/specs/01-example.md#EXAMPLE-1 --summarize-evidence
$ backstitch obligation docs/specs/01-example.md#EXAMPLE-1 --find-evidence
$ backstitch guide alignment
$ backstitch
$ backstitch check
$ backstitch coverage
$ backstitch check --format json --output spec-trace.json
$ backstitch coverage --format json --output intent-coverage.json
$ backstitch check --show-suppressions

The obligation reads help bootstrap alignment before it is complete. Declared evidence comes only from source mappings, backlinks, invariant binds, and binding tests. Discovered candidates are deterministic review advice; a human or agent must still author an ordinary source diff, and Backstitch never applies that diff itself.

The built-in backstitch-style-v1 profile defaults to docs/specs for specs and backstitch plus tests for code. Override roots in configuration or with repeatable --spec-root, --code-root, and --test-root options. Bare backstitch delegates to the repository's configured default command.

Command Reference

Global Options

Global configuration options may appear before a command. Command-specific forms are also available where relevant.

--config PATH - Use exactly one configuration file; skip discovery
--no-config - Skip repository configuration; packaged defaults still load
--version - Show the installed Backstitch version
--help - Show help

Commands

Common deterministic examples:

$ backstitch obligation list --repo-root .
$ backstitch obligation docs/specs/01-example.md#EXAMPLE-1 --find-evidence
$ backstitch check --repo-root . --warnings-as-errors
$ backstitch check --repo-root . --format json --output spec-trace.json
$ backstitch coverage --repo-root . --format json --output intent-coverage.json
$ backstitch packets --repo-root . --kind invariant --output invariants.jsonl
$ backstitch packets --repo-root . --kind all --output packets.jsonl

Run backstitch <command> --help for the full option set.

coverage report mode works without Git and uses the same immutable snapshot and resolved trace graph as check. A whole-file edge is inherited coverage; an exact symbol, non-module owner, invariant declaration, or invariant binding is direct coverage. Reasoned inline or configured exemptions account for deliberate glue without manufacturing spec text. Ratchet mode compares the accepted snapshot with a repository-owned Git baseline and must be pinned in CI with --require-ratchet REF.

Exit Codes

0 - Command completed and no effective fail_on level was present
1 - The target repository has a finding at an effective failing level
2 - Arguments, configuration, input artifacts, output, or the tool failed

Exit 1 describes the target repository. Exit 2 describes the invocation or tool. Backstitch contains per-file failures where possible and never exposes a Python traceback as normal CLI output.

Core Concepts

Spec and Code Traceability

Backstitch parses Markdown sections with IDs such as [AUTH-1], implementation mapping blocks, and Python references in module, class, function, and method docstrings or comments. Mappings use exact repository-relative paths. Bare IDs resolve only when unique; Backstitch reports ambiguity instead of guessing.

The deterministic report contains normalized sections, code references, mappings, resolved edges, invariants, binds, issues, and summary counts. Identical inputs produce stable JSON ordering.

Invariant Bindings

Declare behavior in an owning Python docstring or inside an ID-bearing spec section:

def resolve() -> Report:
    """Build the deterministic report.

    Invariant: [INV.RES.1] Identical inputs produce byte-identical JSON.
    """

Bind the invariant inside a real test definition:

def test_report_is_stable() -> None:
    """Tests-invariant: [INV.RES.1]"""

Use Invariant (draft): for an advisory declaration. Under packaged defaults, an untested required invariant is an error and an untested draft invariant is a warning. packets --kind invariant includes bounded target and binding-test snippets for advisory semantic review.

Diagnostic Policy

Diagnostics have canonical names such as CODE_REF_UNMAPPED_FROM_SPEC and stable short aliases such as BSC008. Rules are ordered and the last matching rule wins:

[tool.backstitch.diagnostics]
fail_on = ["error", "warning"]
suppressible_levels = ["warning", "info"]

[[tool.backstitch.diagnostics.levels]]
select = ["BSC*", "INVARIANT_UNTESTED:draft"]
level = "warning"

Selectors accept canonical codes, short codes, *, family prefixes, and supported contexts. A diagnostic set to off is omitted from normal findings but remains available in suppressed_issues under --show-suppressions.

Repositories may opt into documented suppression governance:

[tool.backstitch.lint]
require_suppression_declarations = true

[[tool.backstitch.lint.suppressions]]
mechanism = "ignore"
path = "tests/*"
sections = []
codes = ["CODE_REF_UNMAPPED_FROM_SPEC"]
declaration = "docs/specs/04-exclusions.md#SUP-TEST-CITATIONS"

The referenced spec contains a CommonMark declaration such as _Traceability: suppression-declaration [SUP-TEST-CITATIONS] "Tests cite the contract but are not implementation owners."_. Strict mode leaves the original finding active when the declaration is invalid and reports every applied declaration and decoded rationale through --show-suppressions.

Configuration Discovery

Backstitch always starts with packaged defaults, then applies repository configuration, supported environment values, and explicit CLI options. It searches upward for the nearest .backstitch.toml or pyproject.toml with a [tool.backstitch] table. A standalone file uses the same key layout without the tool.backstitch prefix. Explicit --config PATH and extend may name any TOML filename; those names never become implicit discovery conventions.

[tool.backstitch]
default_command = "check"

[tool.backstitch.profile]
name = "backstitch-style-v1"
spec_roots = ["docs/specs"]
code_roots = ["src", "tests"]
test_roots = ["tests"]

[tool.backstitch.check]
format = "text"
warnings_as_errors = false

default_command accepts false, "check", or "analyze". Packaged defaults use false, so a repository must opt in. "check" is equivalent to backstitch check --repo-root .; "analyze" is equivalent to backstitch analyze --repo-root .. The value is one closed command name, not a shell command or a list, and it cannot contain arguments. Arguments typed after backstitch are forwarded to the selected command. A leading path is --repo-root shorthand, so backstitch . works, and an analyze default accepts the usual flags such as backstitch --model gpt-5.6-luna. Set false in a child config to disable an inherited default.

Bare "analyze" has the same credential reads, cache writes, bounded provider calls, and possible cost as explicit current-repository analysis. Do not use bare invocation in secret-bearing hostile-target automation. Such workflows must name analyze, the trusted config, and workflow-owned overrides explicitly.

Roll out a Backstitch version that supports default_command before adding the key to downstream repositories. Remove the key before downgrading to an older strict version, which will correctly reject it as unknown.

One invocation-scoped resolver applies the cascade `CLI > defined environment

selected config > extended config > packaged defaults`, then passes an immutable settings object into command code. Repeat a generic override before or after a config-consuming command:

backstitch --option analyze.cache_mode read-write analyze ...
backstitch config show --option check.warnings_as_errors true

KEY is a known runtime-consulted dotted leaf. VALUE is parsed as one TOML value when possible, otherwise as a bare string. Quote ambiguous strings. summarize-analysis, guide, and cache cleanup-lock reject configuration controls because they do not consume settings.

Test roots classify paths within code roots. Replacing code_roots without also supplying test_roots resets test roots for that configuration layer; every nonempty final test root must be contained by a final code root.

Semantic Review

Deterministic checks never call a model. Generate packets explicitly, then analyze and summarize them:

$ backstitch check --show-suppressions --format json --output spec-trace.json
$ backstitch packets --kind all \
    --output packets.jsonl \
    --report packet-report.json
$ backstitch analyze \
    --packets packets.jsonl \
    --packet-report packet-report.json \
    --output analysis.jsonl \
    --report analysis-report.json
$ backstitch summarize-analysis \
    --deterministic-report spec-trace.json \
    --analysis-results analysis.jsonl

Packets bind section, invariant, or documented-suppression intent; code snippets, tests, deterministic findings; and exact evidence regions shown to the model. The provider receives a packet-derived response schema, but model output remains untrusted: Backstitch owns packet identity, validates structured rows and evidence locality, and contains malformed output per packet.

The committed cloud default selects GPT-5.6 Luna through llm 0.33's OpenAI Responses adapter. Backstitch sends logical max_tokens = 16384 and reasoning_effort = "max"; the adapter serializes those as max_output_tokens and reasoning.effort. Temperature and seed are absent. Optional request fields have no omission sentinel: an absent key accepts the provider default, while any present value is validated, identity-bearing, and frozen before the adapter applies provider wire spelling. The provider's JSON Schema envelope may use strict = false; Backstitch's closed, packet-local normalizer remains authoritative.

Semantic verdicts can be stored in an immutable, content-addressed cache. cache_mode = "read-write" calls the provider only for misses; "require" forbids provider calls and fails on a miss. Cache identity includes the packet, prompt, provider, request controls, contract version, and explicit search epoch. Policy is deliberately excluded, so a policy-only change reprojects the same frozen result with zero calls.

result_reuse = "evidence-stable" is the default. For each unchanged packet/prompt/request/epoch review identity, the first complete result remains in force even when a later run selects a different model. Changed evidence is an ordinary miss under the newly selected model. Set result_reuse = "exact-inference" to require the selected provider's exact cache key, or change search_epoch to resample the evidence-stable decision. Reports preserve the original producing provider for carried results. Model age, provider qualification, and elapsed time do not invalidate an unchanged evidence-stable baseline. Release candidates qualify the exact current Luna and protected GPT-5.5 requests live; qualification is bounded event evidence, not a persisted or expiring Backstitch artifact.

Trusted model descriptors use Model Monster pkg:service PURLs as stable identity, for example pkg:service/openai.com/gpt-5.6-luna. The separate adapter_model_id is the raw name passed to llm. LLM_MODEL and --model may select a trusted descriptor by its PURL or by an unambiguous adapter model ID; provider identity remains the PURL.

The repository ignores .backstitch/. Treat its semantic cache as disposable acceleration state and its reports as fresh run outputs, not as source or reviewed evidence to commit. The usual local flows are:

update findings with bounded provider calls by explicitly selecting the trusted pyproject.toml and applying the reviewed static read-write options; valid hits are reused and only misses call the provider;
replay with zero provider calls by overriding cache_mode = "require"; a missing baseline or exact object fails with exit 2;
diagnose without reading or writing cache objects by using a trusted config whose [analyze] cache_mode is "off"; and
retain a deliberate new sample by changing the trusted search_epoch (and verifier search_epochs, when verification is enabled) before a read-write run.

When no Backstitch process is using it, the whole .backstitch/ directory can be removed. The next read-write run rebuilds needed objects; the next require run reports misses. Backstitch does not currently provide fine-grained pruning or concurrent whole-root deletion. A durable reviewed snapshot for indefinite provider-free replay would be a separate future feature, not a use of this cache.

The analysis report records completeness, cache hits and misses, provider calls, budgets, evidence-bound diagnostics, candidate debt, applied policy, problems, and the result digest. Exit 0 means the required review completed and policy allowed it. Exit 1 is reserved for policy-failing human or mechanically verified target findings. Exit 2 means the invocation, cache, provider, normalization, completeness, budget, or publication failed.

Measure one exact semantic identity against the committed mutation and negative-control corpus with:

$ backstitch eval \
    --corpus tests/semantic_eval/v3/manifest.json \
    --config pyproject.toml \
    --option analyze.cache_mode read-write \
    --option verify.enabled true \
    --option verify.cache_mode read-write \
    --output semantic-eval-report.json

The committed schema-3 smoke corpus exercises the current evaluator in report mode only. The eval report separates evidence sufficiency from conditional and end-to-end recall. It also records precision, false-positive and indeterminate rates, uncached flip rate, required-cache replay stability, cost, and latency. Report mode is observational. Enforce mode can grant stronger policy authority only after a reviewed historical corpus, a passing report for the exact analyzer/verifier composition, and an explicit human-pinned report digest.

Use backstitch doctor --probe before semantic analysis to check model registration, credentials, constrained decoding, and endpoint reachability. See the semantic gate implementation guide for cache trust, refresh, replay, evidence-authority, and rollout details.

Development and Contributing

Runtime

Backstitch uses markdown-it-py for CommonMark block structure and tree-sitter-python for Python ownership, comments, and docstrings. Running on Python 3.11 can therefore analyze newer target syntax such as PEP 695 generics and PEP 701 f-strings without relying on the host interpreter's ast grammar.

Testing

Backstitch's own committed repository config deliberately selects default_command = "analyze". Bare backstitch here is provider-capable; use explicit backstitch check --repo-root . for the hermetic self-corpus gate.

The repository pytest configuration runs the live cloud-provider contract test in the default local suite. A normal local run needs a working llm model and credential:

$ uv run pytest -q

For an intentionally hermetic run:

$ uv run pytest -q -m "not live_llm and not benchmark"
$ uv run pytest tests -q -n 0 -m benchmark
$ uv run pytest tests/live/test_live_llm.py -q -o run_live_llm=false

The serial benchmark lane runs each real self-corpus command once to warm up and five times for a reported median. It always fails command errors, timeouts, and catastrophic regressions. Relative regression checks remain explicitly unavailable until the repository pins a matching wall-clock runner contract and content-bound baseline.

To reproduce the CI coverage report locally:

$ uv run pytest tests -q -n auto --dist loadgroup -m "not live_llm and not benchmark" --cov=backstitch --cov-report=term-missing

The completion gate also includes:

$ uv run pytest tests/acceptance -q
$ uv run ruff check backstitch tests bin
$ uv run ruff format --check backstitch bin .github/scripts tests
$ uv run mypy backstitch bin/release.py tests
$ uv run backstitch check --repo-root . --show-suppressions

Live LLM Tests

tests/live/test_live_llm.py drives the real packets to analyze to check to summarize-analysis path. It asserts structured contracts, transport, and model success for cloud runs, not exact wording or classification. The cloud fixture includes a documented suppression packet and immediately proves a zero-call require replay after the live read-write miss.

# Store a provider key once, then run the local-default live test
$ uv run llm keys set openai
$ LLM_MODEL=gpt-4.1-mini uv run pytest tests/live/test_live_llm.py -q

# Or use a provider environment variable
$ OPENAI_API_KEY=... LLM_MODEL=gpt-4.1-mini uv run pytest -m live_llm -q

The same test supports a loopback OpenAI-compatible endpoint such as Ollama:

$ docker run -d --name backstitch-llm \
    -p 127.0.0.1:11434:11434 \
    -v "$PWD/.ollama-cache:/root/.ollama" \
    ollama/ollama
$ docker exec backstitch-llm ollama pull llama3.2:3b
$ BACKSTITCH_LOCAL_LLM_UPSTREAM=http://127.0.0.1:11434/v1 \
    BACKSTITCH_LOCAL_LLM_SERVED_MODEL=llama3.2:3b \
    BACKSTITCH_LIVE_LLM_KIND=local \
    uv run pytest -m live_llm -q

Cloud tests cost money and can fail because of provider outages, rate limits, or model retirement. Local endpoints avoid provider credentials but add model startup and output-quality variance. See docs/implementation/06-choosing-a-local-model.md for measured local-model guidance.

The hermetic CI matrix always deselects live tests and receives no provider secret. The manual semantic-refresh workflow binds its trusted tool checkout to the dispatch run's exact default-branch revision, requires the OPENAI_API_KEY repository secret, restores only validated immutable cache object trees, runs the bounded dogfood and eval refresh, and uploads fresh review reports. Cache service state is optional and untrusted. Missing credentials, incomplete analysis, or corrupt restored objects fail; cache eviction causes a bounded cold refresh. The workflow never commits or pushes cache or report content. It uses repository_dispatch, whose run ref is the default branch, instead of branch-selectable workflow_dispatch. Trigger it with write access:

$ gh api --method POST repos/{owner}/{repo}/dispatches \
    -f event_type=semantic-refresh

Repository collaborators with permission to create a repository dispatch can request a report-only analysis of an open pull request. Supply the PR number and the exact lowercase 40-hex head SHA shown by the GitHub API:

$ gh api --method POST repos/{owner}/{repo}/dispatches \
    -f event_type=semantic-pr-report \
    -F 'client_payload[pull_request_number]=123' \
    -f 'client_payload[head_sha]=0123456789abcdef0123456789abcdef01234567'

That workflow runs trusted Backstitch code, dependencies, prompts, provider controls, and configuration from the exact default-branch workflow revision. It treats the API-confirmed PR checkout as read-only data and never installs or executes its workflows, hooks, plugins, build configuration, or commands. The provider receives only the bounded packet projection selected by trusted Backstitch: relevant specification text, code and test evidence, deterministic findings, and exact evidence regions. Provider credentials exist only during the credential check and analysis step. The resulting artifact is observational; it does not comment on the PR, create a status, or gain merge authority.

The separate local-llm workflow owns the Ollama canary.

Releases

The local release helper runs regular, cloud-live, and local-live checks before creating and pushing a version tag. The tag-triggered workflow publishes to PyPI through Trusted Publishing and creates the GitHub Release.

$ bin/release.py --version X.Y.Z --dry-run
$ bin/release.py --version X.Y.Z

# When version files and CHANGELOG.md are already prepared
$ bin/release.py all --dry-run
$ bin/release.py all

See docs/implementation/05-release-publishing.md for release setup, rollback, and verification.

License

Backstitch is released under the MIT License. See LICENSE.

Name		Name	Last commit message	Last commit date
.github		.github
backstitch		backstitch
bin		bin
docs		docs
skills		skills
tests		tests
.codecov.yml		.codecov.yml
.gitignore		.gitignore
AGENTS.md		AGENTS.md
CHANGELOG.md		CHANGELOG.md
CLAUDE.md		CLAUDE.md
LICENSE		LICENSE
README.md		README.md
pyproject.toml		pyproject.toml
uv.lock		uv.lock

Command	Description
`obligation`	List obligations or inspect readiness, declared evidence, and deterministic candidates
`guide alignment`	Print the installed, versioned alignment quick start
`check`	Build the deterministic trace graph and report findings
`coverage`	Measure direct, inherited, exempt, and uncovered intent reach per Python definition
`packets`	Generate bounded section or invariant review packets; no model calls
`analyze`	Resolve immutable semantic results and apply repository policy
`eval`	Measure one semantic identity against a closed mutation/control corpus
`summarize-analysis`	Combine a deterministic report with semantic results
`doctor`	Diagnose model, credential, decoding, and endpoint readiness
`config show`	Print effective settings, layers, and diagnostic policy as JSON
`config path`	Print the discovered repository configuration path
`cache cleanup-lock`	Audit and remove one unchanged abandoned semantic lock

Sunbelt Computer Software

PL/B Language Development and Support

Navigation Menu

Repository files navigation

Backstitch

Recommended For

Table of Contents

Features

Installation

Quick Start

Command Reference

Global Options

Commands

Exit Codes

Core Concepts

Spec and Code Traceability

Invariant Bindings

Diagnostic Policy

Configuration Discovery

Semantic Review

Development and Contributing

Runtime

Testing

Live LLM Tests

Releases

License

About

Releases

Packages

Contributors

Languages

Sunbelt Computer Software

PL/B Language Development and Support

Latest commit

History

Folders and files

Repository files navigation

Backstitch

Recommended For

Table of Contents

Features

Installation

Quick Start

Command Reference

Global Options

Commands

Exit Codes

Core Concepts

Spec and Code Traceability

Invariant Bindings

Diagnostic Policy

Configuration Discovery

Semantic Review

Development and Contributing

Runtime

Testing

Live LLM Tests

Releases

License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages