The project aims to measure whether the operational artifacts an LLM writes for a supercomputer (SLURM job scripts and Apptainer container recipes) are actually correct: verified by submission, execution and resource fit, not by textual similarity. Beyond writing scripts from scratch (T1), Anvil also measures whether a model can diagnose and repair a broken one (T2, see Diagnose-and-repair (T2)) and whether it can write a correct Apptainer recipe (T3).
- Any machine, to develop and to verify
- [Optional] An NVIDIA GPU or Apple Silicon, to generate scripts with a real model
- [Optional] A machine with SLURM installed, if you do not want to use Docker
- Python >= 3.10
- Docker Community Edition (recommended: it provides SLURM and GNU coreutils)
- [Optional] PyTorch and Transformers, only to generate with a model
- [Optional]
bitsandbytes, for 4-bit quantization on CUDA
The benchmark validates artifacts against a declared reference cluster, not against the hardware of the machine that runs it. Otherwise scores would depend on who runs the benchmark. The topology is defined by environment variables read by the container:
ANVIL_NODES=4 # virtual nodes
ANVIL_CPUS=16 # cores per node
ANVIL_MEM_MB=64000 # memory per node, in MB
ANVIL_GPUS=4 # GPUs per node
Changing the topology changes the results: it is part of the benchmark definition. See
docs/REFERENCE_CLUSTER.md.
Tasks live in tasks/t1_slurm.jsonl; their canonical solutions in tasks/t1_reference.jsonl.
Induced repair tasks (T2) live in tasks/t2_repair.jsonl, see
Diagnose-and-repair (T2).
To work on the benchmark, from a checkout:
make install
That builds .venv/ and installs into it, without touching the interpreter you invoked it with,
so the command lives at .venv/bin/anvil rather than on your PATH. Every anvil ... below is
written without a prefix; from a checkout it is .venv/bin/anvil ..., or activate the environment
once with source .venv/bin/activate and drop the prefix for good.
To generate scripts with a real model, also install the model extras:
make install-models
To use the verifier without a checkout, install the package:
pip install git+https://github.com/antoniorotundo2/anvil
That gives you the anvil command on the PATH of whatever environment you install it into,
with no dependencies beyond the standard library. The task
files travel with it, so anvil check job.sh --task t1_gpu_single works from any directory. A
checkout still reads tasks/ from the working directory, which is where every published number
was measured, and the packaged copy is only consulted when that path does not exist.
To run the project you can use two methods. The first one (recommended) uses Docker, which ships a
SLURM reference cluster and GNU coreutils, so every verification level is active. The second one is
the manual way, on your own machine, where submittability is skipped unless you install SLURM and
where functional runs under whatever bash and coreutils you happen to have.
make docker-run
Verify what the environment can actually check:
make doctor
The runtime is a variable, so the same targets run against Podman:
make RUNTIME=podman docker-build docker-test
The docker- prefix stays on the target names, since that is what CI and every note in this
repository already say. The scripts take the same variable:
RUNTIME=podman ./scripts/executor_ablation.sh results/<run>.
Verified on Podman 4.9.3, rootless, Ubuntu 24.04 under WSL2. docker-build and docker-test pass
there, 302 tests, and the short name ubuntu:24.04 resolves without help, so no fully-qualified
image is needed. Everything that grades under the bash executor works.
Real submission does not. docker-guards-sbatch builds the scheduler image and then stops:
==> WARNING: a placeable job did not become ready in 60s; submittability will be unreliable
slurmd has to create its step scope under /sys/fs/cgroup, and rootless Podman does not delegate
the controllers that needs, so no job is ever placeable. It stops rather than grading, which is the
point: without that check the run would have gone ahead and produced numbers with submittability
quietly unreliable. A rootful sudo podman is the obvious thing to try and has not been tried here,
since root keeps a separate image store and the image would be built again from scratch.
So: RUNTIME=podman for anything under the bash executor, Docker for the arms that need an
enforced allocation. On an SELinux host a bind mount may also need :z, which is not added here
because relabelling a checkout is not a side effect to introduce untested.
make run
The oracle returns canonical solutions and must score 1.0; the broken model returns deliberately faulty artifacts and must score 0.0. If the oracle drops, the benchmark is broken, not the model.
make guards
functional executes the script with bash in a sandbox by default. That sandbox has a memory
ceiling, ANVIL_SANDBOX_MEM_MB, 1024 by default and applied with ulimit -v, so it does nothing
on macOS. It exists to protect the machine, not to judge the artifact: it is deliberately not
derived from --mem, or a script under-requesting memory would be stopped by the sandbox and the
bash executor would start reporting the enforcement only real submission can measure. A script
stopped by it says so in its detail, and raising it is the right answer for a task whose payload
genuinely needs more.
--executor sbatch submits it to the scheduler for real instead, waits for the job and reads its
outcome from scontrol, so the walltime the script requested is enforced and the payload sees
every variable SLURM injects:
anvil run --model oracle --tasks tasks/t1_slurm.jsonl --executor sbatch -v
ANVIL_FUNCTIONAL_EXECUTOR=sbatch anvil verify --generations results/generations.jsonl
It is opt-in on purpose. Every published number was measured under bash, and switching the
default would make later ones incomparable with them; the executor is recorded in each result
file as functional_executor. It also needs a scheduler that genuinely runs jobs, not one that
merely accepts them: its own canary checks that first, and functional is skipped, never failed,
when the check does not hold. Same for a job the scheduler can never place, such as the dependency
task pointing at the held placeholder job.
make guards-sbatch
make docker-guards-sbatch
The container form builds its own image: the default one accepts jobs and never runs them, since
Ubuntu 24.04's SLURM has no accounting plugin other than slurmdbd and refuses each job with
Reason=InvalidAccount. make docker-build-sched adds slurmdbd and a local database, opt-in
like the apptainer image and on the same base, so the coreutils stay GNU. The run also needs
--privileged --cgroupns=host, which the Makefile supplies: slurmd creates its cgroup scope
under /sys/fs/cgroup, which a plain docker run mounts read-only. See
docs/REFERENCE_CLUSTER.md.
With slurmd running, the container also enforces the allocation through cgroups, so a job that
uses more memory than it requested is killed instead of finishing. Nothing in the eight T1 tasks
allocates enough to notice, so tasks/t1_exec.jsonl holds two tasks that do, neither of them
stating a memory number: the payload's real need is the ground truth. They under-spend for
different reasons, which is what keeps them from being one task measured twice. The first builds
64MB inside a command substitution, so the pipe and the variable are resident at once. The second
starts four workers at the same time, each holding 32MB, and --mem is the allocation for the whole
node, so a model that reasons correctly about a single worker still under-requests by a factor of
four.
The fault induced from them (F8, --mem cut to 16M) is well formed, within spec, accepted by the
scheduler, and passes every level under bash. It fails only here:
make docker-guards-enforcement
The set lives in its own file so that adding to it changes no digest: tasks/t1_slurm.jsonl and
tasks/t2_repair.jsonl are untouched, and every published number stays comparable.
Graded with five models it is where the executor earns its cost: 288 artifacts of 900 change
their strict verdict between the two arms, against 6 of 3900 on the sets whose requirements are
written in their prompts. Qwen2.5-Coder 7B, which scores 1.000 on resource_fit from scratch on
T1, has 29 of its 30 artifacts here promoted by the sandbox and every one of them killed by the
scheduler. See what the two arms disagree
about.
CPU and GPU binding remain outside this level: a job is confined to its cores, but no task asks what it was given.
Everything above measures a model. If instead you have a job script that an assistant wrote and
you want to know whether it will hold up, anvil check answers that with no task file, no model
and no benchmark run:
anvil check job.sh
syntax, safety and submittability need nothing but the script and, for the last one, a
scheduler to ask. resource_fit and functional compare against a spec, so without one they are
reported as not checked rather than passed. Give the script a task to be graded against and all
five run:
anvil check job.sh --task t1_mpi_multinode
The exit code is 0 when every level that ran is satisfied and 1 otherwise, which is what makes it
usable from a pre-submission hook or a CI step. --json prints the same verdict for a machine.
A cluster has rules that no task file knows: how long a job may run, how many nodes it may take,
which partitions exist, which directives are required. --policy checks a script against them:
anvil check job.sh --policy policies/reference_cluster.json
policy FAIL anvil reference cluster
nodes 9 exceeds the site maximum 4
--time 2880min exceeds the site maximum 1440min
--mem 131072MB exceeds the site maximum 64000MB
The comparison runs the opposite way from resource_fit, which is the distinction worth keeping
straight: a task fails a script that asks for too little, a policy fails one that asks for too
much. Ceilings apply to the effective request, so a script with no --nodes is judged as asking
for one node and one with no --ntasks as asking for one task per node. A missing --time is a
violation rather than a pass, because SLURM would apply a partition limit the file does not state
and the site cannot conclude the job fits.
The shipped example carries the declared reference cluster's own limits, and a test holds it to the bracket the rest of this project uses: it must accept every canonical solution, and it must still refuse a script that asks for too much. An example policy that rejected the benchmark's own reference answers would be defective rather than strict, which the first version of that file was.
The example travels with the package, so that command works from a checkout and from a
pip install alike. Your own file is a path like any other, and a path that exists always wins.
Every field is optional and an absent field is not a rule. A field that is not recognised is an
error rather than a silence: a misspelled max_mem_gb would otherwise read as a site with no
memory limit at all. See policies/reference_cluster.json.
make test
make lint
Generation needs the machine with the accelerator. Faithful verification needs the machine with the scheduler and GNU coreutils. They are rarely the same machine.
make generate MODEL=Qwen/Qwen2.5-Coder-1.5B-Instruct
make docker-verify
verify records bash, coreutils, base_image and functional_executor in its JSON output, so
every number carries the environment that produced it. This also enables the cross-distribution
ablation: generate once, verify against several base images.
Two machines also means two checkouts, so the report carries two digests beside that environment:
tasks_sha for the questions and verifier_sha for the rules that graded them. The generations
carry the first, and verify exits 2 rather than score answers to questions nobody asked:
[ERROR] these generations were produced against a different task file
(theirs: ['deadbeef1234'], current: bbc74707a030).
The second is stamped on the report, since it is only known once the grading happens. Nothing refuses at that point, because the run is internally consistent. The leaderboard is where it matters: a row whose digest differs from the rest of its column is printed marked stale rules rather than dropped, and marked means not comparable with the column, not merely older. The fix in both cases is to verify again from one checkout, which costs a verification and not a generation.
The container defaults to ubuntu:24.04, not the newest LTS: Ubuntu 26.04 replaces GNU coreutils
with uutils, which no HPC centre runs. Fidelity beats freshness.
docker build -t anvil:2604 --build-arg BASE_IMAGE=ubuntu:26.04 docker/
A task declares a natural-language prompt, the resource constraints to check, the directives that
must be written out explicitly, and the strings its output must contain. Every new task needs a
canonical solution in tasks/t1_reference.jsonl, or make guards will fail.
T1 asks a model to write a script from scratch. T2 hands it a broken one, from one of seven
fault classes anchored to failures observed on a real model (docs/OBSERVED_FAILURES.md, F1–F7),
and asks it to diagnose and fix it. A repair is correct if and only if it clears the exact same
verifier used to grade a from-scratch T1 solution: repair is not a softer notion of correctness.
tasks/t2_repair.jsonl is not hand-written: it is induced mechanically from the T1 canonical
solutions (anvil/inducer.py), and only variants that actually fail verification are kept.
Rebuild it after changing a T1 task or its reference solution:
make induce-t2
That target builds the image and runs inside it, and anvil induce refuses to run anywhere the
submittability level cannot be judged. The reason is that a variant is kept when the verifier
refuses it, and a skipped level is never a passed one: on a machine with no scheduler every variant
survives, including the ones that verify clean, and the file comes out larger while still carrying
a digest that makes it look authoritative. Which faults the benchmark contains is not something to
decide by accident.
Run a model against it and verify, same shape as T1:
make repair MODEL=Qwen/Qwen2.5-Coder-1.5B-Instruct
anvil repair --model oracle --repair-tasks tasks/t2_repair.jsonl --tasks tasks/t1_slurm.jsonl -v
anvil repair --model <hf-model-id> --repair-tasks tasks/t2_repair.jsonl --save-generations results/repair_generations.jsonl
anvil verify-repair --generations results/repair_generations.jsonl --repair-tasks tasks/t2_repair.jsonl -v
Both commands break the summary down per fault category (F1–F7) in addition to the overall
one, on screen and under "by_category" in --out's JSON: an aggregate pass@k can hide a category
a model never manages to repair.
The oracle repair (returns the T1 canonical solution, ignoring the diagnosis) must pass every
induced fault; a no-op "repair" that returns the broken script unchanged must pass none. If either
fails, t2_repair.jsonl or the repair verifier is broken, not a model.
make guards-t2
A third artifact type: a model writes an Apptainer definition file (.def) instead of a SLURM
script. Same shape as T1, different vocabulary: syntax, buildable (does apptainer build
succeed), functional, resource_fit (header and section set against the spec), safety.
Apptainer is opt-in and not part of the default image, since most anvil work never touches it:
make docker-build-apptainer
Run a model against tasks/t3_apptainer.jsonl and verify:
make recipe MODEL=Qwen/Qwen2.5-Coder-1.5B-Instruct
anvil recipe --model oracle --tasks tasks/t3_apptainer.jsonl -v
anvil recipe --model <hf-model-id> --tasks tasks/t3_apptainer.jsonl --save-generations results/recipe_generations.jsonl
anvil verify-recipe --generations results/recipe_generations.jsonl --tasks tasks/t3_apptainer.jsonl -v
Apptainer runs unprivileged inside the container, so no capability is granted: what it
needs is exemptions from Docker's confinement. make docker-guards-t3 applies them and
needs no argument. The same set works on every verified host, so there is nothing to
select per environment (see the Makefile's DOCKER_RUN_APPTAINER and docs/DESIGN.md
for what each one unlocks):
docker run --rm --security-opt seccomp=unconfined --security-opt apparmor=unconfined \
--security-opt systempaths=unconfined --device /dev/fuse \
-v "$PWD":/work -w /work anvil:apptainer ...
--privileged also works but grants far more than these actually need. Verified on
GitHub-hosted runners and on WSL2, where the strict bracket returns identical per-level
scores. On Docker Desktop for Mac, build succeeded but run failed
(exec ... failed: invalid argument), untested since the AppArmor findings.
buildable and functional both need a real apptainer binary, much less commonly available
than the bash that T1/T2's functional relies on. Without it, both are skipped, not failed:
make guards-t3
checks only what syntax/resource_fit/safety can prove. The full oracle-1.0/broken-0.0
bracket needs the opt-in image:
make docker-guards-t3
Three ways to prompt a model for T1: zero-shot (the default, no change from the rest of this
README), vector (TF-IDF similarity against tasks/retrieval_corpus.jsonl), vectorless (exact
tag match, no scoring). anvil run --retrieval selects the arm:
anvil run --model oracle --tasks tasks/t1_slurm.jsonl --retrieval vector -v
anvil run --model <hf-model-id> --tasks tasks/t1_slurm.jsonl --retrieval vectorless
OracleModel still recognises the task regardless of which arm is active (it matches on
prompt.startswith(task.prompt), since retrieved context is always appended after the original
prompt, never before it), so make guards stays valid for any --retrieval value.
Compare all three arms on the same model, seeds and tasks:
./scripts/retrieval_ablation.sh
MODEL=Qwen/Qwen2.5-Coder-1.5B-Instruct SEEDS="0 1 2" N=5 ./scripts/retrieval_ablation.sh
Verify one set of generations inside several base images and report where the verdicts
diverge, per sample and per level. It reads the *.generations.jsonl files that
scripts/run_experiments.sh saves beside every cell, so it inherits that run's seeds
without spending inference time again:
./scripts/crossdist_ablation.sh results/<run>
BASES="ubuntu:24.04 ubuntu:26.04" ./scripts/crossdist_ablation.sh results/<run>
Same shape, with the executor as the varying factor instead of the base image: it verifies each
cell twice, under bash and under real submission, inside the one image so nothing else changes.
make docker-build-sched
./scripts/executor_ablation.sh results/<run>
The number it reports is not either pass@k but the disagreement: how many scripts the sandbox
promotes and the scheduler stops, grouped by what stopped them (OUT_OF_MEMORY, TIMEOUT, a
refused submission, missing output). It also counts the samples real submission cannot judge, such
as the dependency task waiting on a job that never completes, which are skipped rather than
charged to the model.
How large that disagreement is turns out to be a property of the task set rather than of the method: 0.15% where the requirements are stated in the prompt, 32% where only the payload knows them. Run it against both before concluding anything about how much real submission is worth.
paper/anvil.tex is a preliminary manuscript describing the benchmark and the measurements
on this page. It is not posted anywhere yet.
make paper
make arxiv
make paper regenerates the figures' data from leaderboard/entries/ and compiles the PDF,
so the manuscript cannot quote a run that has since been re-imported; tests/test_paper.py
fails when the two disagree. make arxiv writes paper/anvil-arxiv.tar.gz, which is what an
arXiv submission wants: sources rather than a PDF. It also writes paper/abstract.txt, the
abstract flattened to plain text for a submission form that renders no markup.
The package carries anvil.bbl because
arXiv does not run BibTeX and renders whatever bibliography it is given, so a submission
without it builds with an empty References section and no error at all.
The build is byte-reproducible. SOURCE_DATE_EPOCH is pinned to the date the title page
carries, which is why \date is a fixed date and not \today: two compiles of identical
sources otherwise differ, and after make paper a git status that reports a modified
binary would say nothing about whether the paper changed.
docs/DESIGN.md: why execution-based verification, the five levels, the preflightdocs/REFERENCE_CLUSTER.md: the declared topology and its non-obvious detailsdocs/LEADERBOARD.md: where the measured models stand, generated fromleaderboard/entries/docs/RESULTS.md: every measured number, with what it is and what it is notdocs/DATASET.md: the task files, their schemas, and how to tell which copy you havedocs/OBSERVED_FAILURES.md: failure classes observed on a real modeldocs/HARDWARE.md: development machine vs experiment machine
Anvil is developed on one desktop with a single consumer GPU, which is what bounds the measurements: two model families at two sizes, three seeds, and a roadmap item that reads "more families, on borrowed hardware". Sponsorship goes to compute, so those arms stop being roadmap items. Sponsor this project.
MIT.