Experiment guide

Complete setup and choose a benchmark first. This guide covers recursion budgets, task selection, reproducibility, and evaluation.

Recursion modes

Both modes start with k reference executions of the same base harness.

• Parallel recursion proposes and probes k general (invariant) candidates and k domain-specific candidates. Each of k independent compositions uses the full candidate pool and execution evidence to produce a refined harness.
• Sequential recursion runs k Propose–Probe–Compose steps, carrying the refined harness and accumulated execution evidence into the next step.

After adoption, reflection stores the adopted harness and evidence-supported lessons in a per-domain library. Tasks within a domain run in order and read only earlier entries; different domains can run concurrently. Libraries are separated by benchmark, executor, mode, budget, and seed.

Option	Behavior
--k 1	One Propose–Probe–Compose step, followed by reflection after adoption
--k K --mode parallel	Explore K candidates per scope and produce K compositions
--k K --mode sequential	Refine the harness through K successive steps
--workers N	Run N domains concurrently; tasks within each domain stay ordered
--substrate cx	Use Codex instead of the default OpenCode executor
--seed N	Shuffle the task stream reproducibly; does not control model sampling

Use the same benchmark, executor, subset, k, mode, and seed for refinement, judging, and reporting. For example:

export HARNESS2_RUNS_DIR=runs/lab-sequential-k3
harness2 --bench lab --k 3 --mode sequential \
  --only-domains antitrust-competition --max-tasks 1
harness2-judge --bench lab --k 3 --mode sequential
harness2-report --bench lab --k 3 --mode sequential

Use a separate output root when changing the budget or mode to keep experiments easy to distinguish.

Task selection and cost

Model-backed runs incur API costs for executor rollouts and improvement stages. The full LAB task stream contains 1,143 tasks. Start with a small selection and review model pricing before launching a full run.

• --only-domains restricts the run to the named domains.
• --max-tasks caps tasks per domain, not across the whole run.
• --dry-run previews selected tasks and available domains without model calls.

Each benchmark also has a script that runs checks, refinement, judging, and reporting in sequence. Set SUBSTRATE, K, MODE, and, for WorkBuddy, SUBSET; additional arguments are passed to the refinement command:

bash scripts/run_lab.sh --only-domains antitrust-competition --max-tasks 1
SUBSTRATE=cx K=3 MODE=sequential bash scripts/run_jobbench.sh --max-tasks 1
SUBSET=web bash scripts/run_workbuddy.sh --max-tasks 1

Reproducibility

When comparing budgets or modes, keep task selection, executor, improver, judge, and web access fixed. Use a distinct HARNESS2_RUNS_DIR when changing models or task selection; the saved configuration prevents incompatible resumes.

JobBench's dataset is resolved at download time rather than pinned to a commit. Setup records the downloaded Hugging Face revision in dependencies/job-bench-eval/dataset/main/.harness2-download.json. Report that revision alongside JobBench results.

Task partitions

To create a reproducible, domain-stratified partition of the installed tasks:

python scripts/split_tasks.py --bench lab --seed 42 --output splits/lab.json
harness2 --bench lab --split-file splits/lab.json --split test --mode parallel

The default train fraction is 0.5. Generated partitions are local files, not release assets, and the script refuses to overwrite an existing file.

Baseline evaluation

harness2-judge --bases scores a run namespace's reference executions, writing base_judged.json beside judged.json for the base-harness comparison. harness2-report covers the refined harness only.

Web access

All three benchmarks are public, and JobBench publishes each task's RUBRICS.json on Hugging Face. An executor with network access can therefore retrieve grading criteria. In a validation run of this release, a Codex rollout on JobBench downloaded its task's rubric without being instructed to.

The release does not restrict network access. After setup, block the following hosts for the executor to reduce access to benchmark data and grading criteria:

• huggingface.co, hf.co, and *.hf.co
• github.com and raw.githubusercontent.com, for benchmark repositories

Setup itself needs these hosts to download dependencies and data. Codex runs with its sandbox disabled, so apply restrictions at the network level, such as an egress firewall or DNS policy on the executor host, rather than relying on a tool setting. Report the restrictions used alongside results.

Results and resume

The default output roots are runs/lab/, runs/jb/, and runs/wb/<subset>/. They contain task summaries, execution artifacts, adopted harnesses, reflection notes, and graded results.

Repeat the same command to resume. Failed evaluations remain unscored rather than being counted as zero. Use a new HARNESS2_RUNS_DIR when changing models or task selection.

Refinement, judging, and reporting use harness2, harness2-judge, and harness2-report; each supports --help.