clusterctl v0.4.0

This release shows how far a long command has got, runs on many hosts at once what still went one host at a time, and decrypts Secret documents with the `sops` command, which takes the binary to a third of its size. It also removes the configuration keys nothing read and the `-o jsonpath` format, and gives the exit code of a command on many hosts one rule. Some of these changes refuse configuration, command lines or Secret documents that v0.3.0 accepted, which a minor release before 1.0.0 may do; [Upgrading from v0.3.0](#upgrading-from-v030) lists them.

### Progress

- **`--progress`**, or `CLUSTERCTL_PROGRESS`, chooses what standard error shows while a command runs. `auto`, the default, draws a live tree of the steps and the hosts under way when standard error is a terminal and standard output does not go into a pipe, and nothing otherwise; `tty` asks for the tree, `counter` for a single line, `plain` for lines a CI log can keep, and `none` for nothing. Standard output never carries progress, and without a terminal standard error holds what it held in v0.3.0, byte for byte, unless `plain` is asked for.
- A display that has run for a second or more ends with one summary line, such as `clusterctl: provision reinstall: failed in 18m03s: 478 ok, 2 failed`. The summary and the plain lines are written for people and may change; programs read the event log.
- **`--progress-log FILE`**, or `CLUSTERCTL_PROGRESS_LOG`, appends every event to a file as a line of JSON, in format version 1, and creates the file with mode 0600. A `TRACEPARENT` in the environment gives the log its trace; neither it nor `TRACESTATE` is handed on to the programs clusterctl runs.
- `--progress tty` or `--progress counter` where neither can be drawn, and a `--progress-log` that cannot be used, are usage errors (exit 2). The variables fail no command: they show nothing, or write no log, and say why in one line.
- The MCP server sends progress notifications for its tool calls, and its audit log records each call's trace.

The [Progress](https://gsi-hpc.github.io/clusterctl/v0.4/docs/guides/progress/) page of the manual describes the displays and the event log.

### Many hosts at once

- `dns lookup`, `provision status`, `secrets push`, `doctor --remote`, `bmc power` through the ipmitool backend, `fabric state`, and the MCP tools `describe_nodes` and `plan_change` now work on many hosts at once rather than one after another.
- Two new settings bound them: `services.dns.maxConcurrent`, 16 by default, and `bmc.ipmi.maxConcurrent`, 8. `--fanout` on the command line lowers these and the Redfish limit but never raises them, and is handed to ipmipower; `CLUSTERCTL_FANOUT` and `fanout.max` leave them alone.
- `doctor --remote` asks each role in one session. A role whose account logs in but runs nothing, such as one with a nologin shell, now fails.
- The MCP server runs at most two tool calls at once; the others wait, reported as `waiting for another tool call`.

### Secret documents are decrypted by the sops command

A workstation that reads a sops-encrypted Secret needs sops 3.10.0 or later, in `PATH` or named by `workstation.sopsBinary` or `CLUSTERCTL_SOPS_BINARY`; `clusterctl doctor` checks for it. Commands that use no secret do not need sops. The linux/amd64 binary shrinks from 50.5 MB to 17.3 MB, and fixes to sops and its cloud SDKs now come with your sops update rather than with a clusterctl release. clusterctl still makes every check it made before sops runs, and hands sops only the bytes it has checked. It now also refuses sops metadata with a field it does not know, and master keys both inside and outside `key_groups`, where sops would ignore the groups.

### Exit codes

A command on many hosts now exits with the worst of its hosts' codes, in one order: 130, then 3, then 2, then 1.

- `secrets push` with one node refused and one unreachable exits 3 (was 1).
- `exec`, `copy`, `cinc`, `fabric hca` and `node hardware` exit 2 (was 1) when a node's configuration is at fault.
- `cinc` with one node interrupted and one unreachable exits 130 (was 3).
- `hostkey verify`, `hostkey refresh`, `bmc ping`, `dns lookup` and `dns aliases` keep rules of their own, which the exit code reference now lists.
- `secrets push` refuses two secrets for one target (exit 2), and exits 3 when a node that failed could not be reached for any of its secrets.
- A remote command is also stopped locally, with exit 3, once its timeout, the grace period and the time it may take to reach the host have passed, so a host that hangs without dropping the connection no longer holds a command up.
- A command on an infrastructure host whose output was cut off at the limit exits 3 (was 1), as Slurm's commands already did.
- An `ageFile` credential whose file or identities cannot be read exits 2 (was 1), as an `ageFile` secret does.

### Configuration

- **Keys nothing read are gone**, and refused on load as unknown: `bmc.ipmi.passwordTransport`, `bmc.pdu.credential`, `fanout.connectTimeout`, `services.cinc.archivePath`, `services.cinc.baseCookbook`, `services.cinc.rolesPath`, `services.fabric.guidFormat`, `services.http.root`, `services.http.baseUrl`, `services.mail.*` and `workstation.pager`. `CLUSTERCTL_PAGER` goes with the last.
- An explicit `""` or `0` for a key the defaults set, such as `services.dhcp.configPath` or `bmc.redfish.maxConcurrent`, is refused; it used to be read as the default. `services.tftp.root` and `services.tftp.logPath` may no longer be empty.
- The context, tunnel and PDU users must be portable POSIX user names. `alice@EXAMPLE.ORG` and `svc$` now fail `config validate`, with their line, where they used to validate and then fail every connection, and `config init --user` refuses them too. `.alice` is now accepted, as connecting already did.
- A ProxyJump host that is not a role is checked as a host name, so `gw_1.example.org` is refused.

### Boot

- New: `boot grub log` shows what the TFTP server, `in.tftpd`, `tftpd`, `atftpd` or `dnsmasq-tftp`, wrote into `services.tftp.logPath`.
- `boot grub set` refuses a target that is not a file under `services.tftp.root` (exit 2, with `--dry-run` too), and writes the link relative, so that a TFTP server confined to its root, such as `in.tftpd -s`, finds it.

### Output

- `-o jsonpath=…` is gone: it exits 2, as an unknown format. `-o jq=…` does what its templates did, and the [output guide](https://gsi-hpc.github.io/clusterctl/v0.4/docs/guides/output/) gives the jq for each.
- `boot log`, `boot sync`, `dhcp log` and `fabric counters` honour `-o json` and `-o yaml`, as a list of lines.
- Table columns that hold CJK characters or emoji line up.
- A failed remote command reads `<program> on <host> exited N: <what it said>` everywhere.
- `slurm job summary -o json` rows have the key order of the MCP tool's, and credential names and ties in the job summary come in a fixed order.
- `fabric state` prints the ports that answered when the fabric stops early, and asks the fabric in a dry run too.

### Other fixes

- A Ctrl-C at a password prompt ends the reading of credentials: `provision status` used to ask again for every node, and could leave the terminal without echo. A credential lookup that failed is remembered rather than tried again for every node.
- A worker's panic is written after a password prompt another node asks, never into it.
- Host key scans stop once interrupted.
- `cinc run` gets its 30 minutes, or `fanout.commandTimeout` when that is longer.
- Idle Redfish connections time out, and are closed after a fan-out.
- scp's progress meter is shown only for a single transfer.
- ipmitool lines longer than 400 bytes are cut and end in `…`.
- `tunnel start`, `tunnel stop` and `config use-context` complete their names when `--config` is given.
- `mcp serve --set` is parsed as every other command's is.
- A Redfish client or an IPMI backend printed in a debug line no longer shows its password.
- `describe_nodes` no longer stops reading groups at the first node that fails.
- A path such as `~bob/x` is no longer read as `$HOME` followed by `bob/x`.
- A cached remote file with a modification time in the future is no longer trusted for ever. The cache format changed, so old entries read as misses once.

### For Go programs that import nodeset

`nodeset.Resolver` now holds only what parsing calls, `Resolve` and `All`. `List` and `DefaultSource` moved to the new, optional `Lister` interface, which `MapResolver` implements. A resolver written for v0.3.0 still satisfies `Resolver`; code that called `List` or `DefaultSource` through a `Resolver` asserts `Lister` now. Folded output and error messages are unchanged. The package is to move to a module of its own, `github.com/GSI-HPC/go-nodeset`; a later release will say when.

### Upgrading from v0.3.0

The command line gains `--progress`, `--progress-log` and `boot grub log`, and loses `-o jsonpath`; the schema is still `clusterctl/v1alpha1`. Check these before you upgrade:

- **sops on every workstation that reads a Secret document**, 3.10.0 or later, in `PATH` or named by `workstation.sopsBinary`.
- **Every workstation before the site's configuration.** v0.3.0 refuses a key it does not know, so upgrade everyone before a site sets `services.dns.maxConcurrent` or `bmc.ipmi.maxConcurrent`.
- **The removed keys**, in every layer, and **explicit empty values** for keys the defaults set.
- **User names and jump hosts** that the stricter rules refuse.
- **Scripts that read exit codes or use `-o jsonpath`.** See [Exit codes](#exit-codes) and [Output](#output).
- **`xargs` on the IPMI gateway and the fabric host.** The ipmitool backend and `fabric state` now run `xargs -0 -P` there, which a BusyBox built without those options lacks.
- **GRUB targets** outside `services.tftp.root`, which `boot grub set` refuses.
- **Secret documents** whose sops metadata has a field clusterctl does not know, or master keys both inside and outside `key_groups`.

`clusterctl config validate` reports the configuration changes, with file and line; `clusterctl doctor` finds a missing sops, and `clusterctl doctor --remote` an `xargs` the hosts lack.

mise offers a release only once it is 24 hours old. To take this one sooner, name it:

```console
$ mise use -g github:GSI-HPC/clusterctl@0.4.0
```

clusterctl v0.3.0

This release makes clusterctl more careful. Exit codes now mean what the manual says they mean, text from nodes, BMCs and infrastructure hosts can no longer take over your terminal, the MCP server holds an agent to the site and the plan it showed, and the Redfish client and the tunnel stop trusting what they should not. No command or flag is new. Some of these fixes change exit codes or refuse configuration that v0.2.0 accepted, which a minor release before 1.0.0 may do; [Upgrading from v0.2.0](#upgrading-from-v020) lists them.

### Exit codes and interrupts

- **Usage errors exit 2.** An unknown flag, a bad flag value, a wrong number of arguments or an unknown command used to exit 1, which reads as "some nodes are unhealthy". A misspelt subcommand, such as `clusterctl slurm node drian`, printed help and exited 0; it is now refused, with a suggestion. `help` and `completion` follow the same rule.
- **An interrupt exits 130**, whichever command it stopped. The first Ctrl-C now also ends a confirmation or password prompt, and a second one ends the process. ssh is sent SIGTERM rather than killed, so it can restore the terminal, and no further node is started once a fan-out is cancelled.
- **A remote exit 255 is no longer read as an unreachable host.** A remote command that exits 255 is reported as 254, so 255 comes only from ssh itself.
- **A command that fails on an infrastructure host exits 1, not 3**, and the error quotes the first line the host wrote.
- **A panic in one fan-out target** is that target's failure; the others finish and are reported.

### Terminal safety

Everything a node, a BMC, Slurm, a PXE host, a DHCP file or a group source prints now goes through one escaper, and so do error messages. Bidirectional controls, line and paragraph separators and bytes that are not UTF-8 are shown as escapes, so output cannot be reordered, forged as another node's line, or write to your clipboard. `-o json` is unchanged.

### MCP server

- A plan is bound to the cluster, hosts and commands it showed. `apply_plan` refuses it if the configuration has since changed, and the question is built from what applying would do now, with the server's own summary last, next to the answer.
- Every refusal is written to the audit log. A plan that cannot be recorded is not offered, and an apply that cannot be recorded sends nothing.
- `read_command` reaches only the site's hosts, ignores `CLUSTERCTL_NODES`, pins `--fanout`, may only shorten `--timeout`, stops at its output limit, and no longer offers `help` or `completion`.
- ssh and password helpers never prompt without a terminal (`BatchMode=yes`), so a call cannot hang on a password prompt.
- A panicking tool becomes a failed call instead of ending the server.

### BMCs and Redfish

- `hostkey` and `dns lookup --bmc`, `describe_nodes` and `bmc web` now reach the `bmcAddress` the inventory records, like every other BMC command.
- Resets are checked against the vendor profile's `resetTypes`, and against `@Redfish.ActionInfo` where the firmware lists its reset types only there.
- Redirects are refused rather than followed, so credentials cannot be sent to another scheme or port.
- A first-use certificate pin is recorded only once, even when several connections race.

### Other fixes

- `tunnel` runs sshuttle with the site's generated ssh configuration, host keys and account, stops only the process that is the profile's tunnel, and refuses an exclude that expands to nothing.
- `--force` names every protected or unknown host it lets through, also with `-y`.
- `boot set --dry-run`, `provision reinstall --dry-run` and `doctor --remote --dry-run` run the same read-only checks as the real run.
- `node groups NODE` and `bmc web NODE` select their argument like every other node argument.
- The output kept of a remote command is bounded at 32 MiB per stream; a result that was cut off is a failure.
- The group list of an exec source is cached like its memberships.

### Upgrading from v0.2.0

The command line gains nothing and loses nothing, and the schema is still `clusterctl/v1alpha1`. Check these before you upgrade:

- **Scripts that read exit codes.** A usage error is now 2 rather than 1 (or 0, for a misspelt subcommand); an interrupt is 130; a remote exit of 255 arrives as 254; a command failing on an infrastructure host is 1 rather than 3.
- **Node names with capitals** in a `NodeInventory` are refused on load. Write them in lower case; host names are not case sensitive, and such nodes were never found before.
- **`fanout.max` of 0 or less**, in any layer, `CLUSTERCTL_FANOUT`, `--set` or `--fanout`, is refused. It used to become 16.
- **A `Cluster` without `inventories`** now uses only its own site's inventories. With several sites loaded it is refused and told to list them.
- **A tunnel profile** that sets `--ssh-cmd`, `--remote`, `--pidfile` or `--daemon` in its options is refused, and one whose exclude expands to nothing is refused too.

`clusterctl config validate` reports the configuration changes, with file and line.

mise offers a release only once it is 24 hours old. To take this one sooner, name it:

```console
$ mise use -g github:GSI-HPC/clusterctl@0.3.0
```

clusterctl v0.2.0

clusterctl now writes a first configuration for you, and the manual describes the release you install rather than what is on main.

### What is new

- **`clusterctl config init [DIR]`** writes the least configuration that resolves: a `Config` with one context, a `Site` with a login node, a `Cluster` and an empty `NodeInventory`, one document to a file, with comments that say what to fill in. `--site`, `--cluster`, `--domain`, `--login` and `--user` set the names. Without `DIR` the files go where configuration is read from: the directory `--config` or `CLUSTERCTL_CONFIG` names, or else your own configuration directory, usually `~/.config/clusterctl`. It writes only into an empty directory and never overwrites a file, and `--dry-run` lists the files without writing them. When no configuration is found, the error now points to it. `clusterctl mcp` does not offer it to an agent.
- **A manual for every release.** The [manual](https://gsi-hpc.github.io/clusterctl/) at the root now describes the latest release, not main. Each minor line keeps its own manual under `/vX.Y/`, main's is under `/dev/`, and a switcher beside the title moves between them. The JSON Schemas that the `$schema` line of a configuration file names now describe the latest release too.

### Upgrading from v0.1.0

Nothing needs to change. No command, flag, exit code or configuration field was changed or removed, the schema is still `clusterctl/v1alpha1`, and a configuration written for v0.1.0 loads as before. `config init` is for starting a new configuration: it refuses a directory that already holds one.

mise offers a release only once it is 24 hours old. To take this one sooner, name it:

```console
$ mise use -g github:GSI-HPC/clusterctl@0.2.0
```

clusterctl v0.1.0

The first release of clusterctl: one static binary in place of the 27 Bash scripts of the cluster toolkit. It reaches the hosts of a site, selects nodes with ClusterShell node set syntax, runs commands on them in parallel, drives their service processors, reinstalls them and administers Slurm, for as many clusters as the configuration describes.

### What is in it

- **Reach everything**: one host key file kept in version control, a generated ssh configuration, and sshuttle profiles for the networks behind a gateway.
- **Select and fan out**: ClusterShell compatible node sets, groups from node attributes, tables or the workload manager, and bounded parallel execution that reports every node.
- **Operate hardware**: Redfish with a pinned certificate, FreeIPMI and ipmitool, rack power units and the InfiniBand fabric.
- **Reinstall nodes**: DHCP inspection, per-node PXE and GRUB boot paths, age or sops encrypted secrets streamed to the node, and the configuration management client.
- **Administer Slurm**: nodes and their drain reasons, the queue and the accounting database, accounts, users and fair share.
- **Work with an agent**: `clusterctl mcp` lets an AI agent read the cluster and plan changes, and nothing is sent until you confirm it.
- **Ask before changing anything**: a command that changes something shows what it is about to do and asks first, `--dry-run` sends nothing, and a protected host needs `--force`.
- **The node set engine as a library**: `github.com/GSI-HPC/clusterctl/nodeset`, for other Go programs.

### Coming from the shell toolkit

There is no compatibility layer. [`doc/migration.md`](https://github.com/GSI-HPC/clusterctl/blob/v0.1.0/doc/migration.md) maps every command and setting to its counterpart, and `examples/site/` in each archive is a complete configuration to start from.

### Before you rely on it

- This is a 0.x release. The configuration schema is `clusterctl/v1alpha1`, and a minor release may still change a command, a flag or a field; its notes will say so.
- What needs real hardware or a real cluster (IPMI against a service processor, fabric diagnostics, a Slurm controller, a PXE boot) is tested up to the command handed to the transport, not against the hardware. Run it with `--dry-run` on your site first.

### Platforms and verification

Linux and macOS, on amd64 and arm64, statically linked. Every archive is listed in `checksums.txt` and carries a build provenance attestation:

```console
$ gh attestation verify clusterctl_linux_amd64.tar.gz --repo GSI-HPC/clusterctl
```

With [mise](https://mise.jdx.dev), `mise use -g github:GSI-HPC/clusterctl` installs it and checks both.