Add per-run resource limits, priority, and resource usage by crmne · Pull Request #1499 · microsoft/mxc · GitHub
Skip to content

Add per-run resource limits, priority, and resource usage - #1499

Draft
Carmine Paolino (crmne) wants to merge 5 commits into
microsoft:mainfrom
crmne:resource-limits
Draft

Carmine Paolino (crmne) wants to merge 5 commits into
microsoft:mainfrom
crmne:resource-limits

Conversation

@crmne

@crmne Carmine Paolino (crmne) commented Oct 14, 2026 •

Copy link
Copy Markdown

📖 Description

MXC confines what a workload can read, write, and reach, but not how much it can consume, and its result does not say how much it used. This PR adds an optional resources section to the development contract (1.1.0-alpha), enforces it on Bubblewrap, Seatbelt, and ProcessContainer, and reports the workload's usage in the output metadata.

{
  "version": "1.1.0-alpha",
  "process": { "commandLine": "python3 build.py" },
  "resources": { "cpuTimeMs": 120000, "memoryMb": 2048, "maxProcesses": 64, "priority": "low" }
}

Supplying resources, even as {}, adds resourceUsage to the output metadata (ExecutionMetadata.resource_usage in the Rust SDK):

{ "userCpuTimeUs": 3410000, "systemCpuTimeUs": 12000, "peakMemoryBytes": 391553024, "limitExceeded": "cpuTime" }

Requests without resources are unchanged: no new arguments, no new metadata, and the same policy hash.

Design

Contract. resources is a top-level, recursively closed object in the one-shot development contract, following the existing flow: mxc_contract::dev::Resources → wire::Resources in CommonRequestIR → ContainerPolicy.resources: Option<ResourcePolicy>. Published contracts keep rejecting it (version-boundary tests), state-aware requests do not carry it, and the typed V1 SDK (exact 1.0.0) does not expose it yet; it is reachable through "MXC request JSON". The schema and TypeScript wire types are regenerated.

Field Meaning
cpuTimeMs CPU time each process in the workload may use
memoryMb Memory each process may commit
maxProcesses Processes the workload may run at once
priority normal (default), low, or idle

Limits are per process on every backend. That is what rlimits can express, and the Windows job object uses its per-process limits to match.

Fail closed. ResourcePolicySupport declares, per backend, which fields it enforces, and validate_common rejects any other field before a sandbox is created (FailurePhase::Rejected). Backends that enforce none reject any resources section, so usage is never silently missing either.

Field Bubblewrap Seatbelt ProcessContainer Others
cpuTimeMs RLIMIT_CPU RLIMIT_CPU + watchdog JOB_OBJECT_LIMIT_PROCESS_TIME rejected
memoryMb RLIMIT_DATA rejected JOB_OBJECT_LIMIT_PROCESS_MEMORY rejected
maxProcesses rejected rejected JOB_OBJECT_LIMIT_ACTIVE_PROCESS rejected
priority nice 10 / 19 nice 10 / 19 BELOW_NORMAL / IDLE class rejected unless normal

Bubblewrap. A new mxc_unix_resources core module (shared by the two Unix backends, like mxc_pty) sets RLIMIT_CPU (rounded up to whole seconds, soft equal to hard), RLIMIT_DATA, and the nice value on bwrap between fork and exec, so every process in the sandbox inherits them.

  • Memory uses RLIMIT_DATA, not RLIMIT_AS. An address-space limit also counts reservations that are never committed. Measured on Linux 7.2 with LibreOffice 26.8: soffice --headless --convert-to pdf on a small pptx and docx crashes (exit 139) under a 2 GB RLIMIT_AS, and is flaky even at 8 GB (thread creation fails), but converts under a 2 GB and even a 256 MB RLIMIT_DATA, which still makes a Python script allocating 3 GB fail with MemoryError. The same holds through MXC itself.
  • Usage is read with waitid(P_PID, WEXITED | WNOHANG | WNOWAIT) and its rusage argument before std::process::Child reaps the child, so Child keeps owning the reap and its pid-reuse guarantees. ChildAccounting wraps every reap path for this.
  • bwrap's PID 1 reaper reports the workload's exit through an eventfd and is never waited for, so its rusage (the whole workload) never reaches bwrap's parent. A run with resources therefore adds --as-pid-1. The namespace still dies with the workload. The trade-off, documented in the backend guide: the workload reaps its own orphans and, as PID 1, ignores signals from inside the sandbox it has no handler for.
  • limitExceeded: cpuTime is reported when the workload was killed by SIGXCPU, or by SIGKILL after using the limit (within 100 ms: the kernel's sampled CPU clock and the rusage that reaches the parent differ by a few milliseconds). A memory limit only makes allocations fail, and how a program reacts is its own business, so Unix backends never guess memory.
  • maxProcesses is rejected: a per-workload count needs a cgroup (pids.max), and RLIMIT_NPROC counts every process of the user. MXC has no cgroup plumbing today, so cgroup v2 support (memory.max, pids.max, cpu.stat) is left as a follow-up.

Seatbelt. The same rlimit and nice code runs before sandbox_init. Two macOS specifics, both measured on macOS 27 (arm64):

  • setrlimit rejects RLIMIT_DATA, RLIMIT_AS, and RLIMIT_RSS with EINVAL, so memoryMb is rejected.
  • macOS answers RLIMIT_CPU with a single SIGXCPU and never escalates: a program that ignores it runs on indefinitely. A watchdog thread checks the workload's session (or, with inherited stdio, its process group, which a CPU limit now creates as a timeout already does) every 100 ms with proc_pid_rusage and kills any member whose own CPU time reached the limit, through the same audit-token signalling as the session sweep. Like the timeout's tree kill, it does not reach a process that leaves the session.
  • Usage comes from proc_pid_rusage on the exited, unreaped process (user, system, and waited-for children's times, converted from Mach units). macOS reports only the process's own peak footprint, so peakMemoryBytes is omitted.
  • resources with the legacy launchMethod: "open" is rejected: the workload runs inside Terminal.app, not as MXC's child.

ProcessContainer. Both tiers already assign the suspended workload to a job object before resuming it. UiJobObject::apply_resource_limits adds the limits and priority class there (keeping KILL_ON_JOB_CLOSE) before assignment, failing closed, and associates an I/O completion port. UiJobObject::resource_usage reports TotalUserTime / TotalKernelTime, PeakProcessMemoryUsed, and the first END_OF_PROCESS_TIME, PROCESS_MEMORY_LIMIT, or ACTIVE_PROCESS_LIMIT notification. The usage is merged into the output metadata after teardown, next to any captureDenials output. Windows counts the CPU limit in user mode only and checks it every few seconds.

Results. SandboxOutputMetadata gains resource_usage: Option<ResourceUsage>; ResourceUsage and ResourceLimit are re-exported from mxc_sdk::v1 and documented in the Rust V1 API reference. A run MXC ended by timeout or kill reports usage but no limitExceeded.

Compatibility

  • Adding a field to ExecutionMetadata, which has public fields, breaks code that constructs it with a struct literal. Only wxc's tests did in this repository.
  • Policy hashes of existing requests are unchanged (resources is skipped when unset).

Follow-ups

  • A cgroup v2 path for Bubblewrap: per-workload memory, process count, and CPU accounting.
  • Node and .NET SDK types for resources and resourceUsage, and the typed V1 Rust request, once the field is promoted to a published contract.
  • Printing resourceUsage from the executors (wxc-exec, lxc-exec, mxc-exec-mac) the way captureDenials prints its pointer, and resource usage in ETW telemetry.
  • resources on state-aware exec requests.

🔗 References

Resolves #1498.

🔍 Validation

  • Linux (Linux 7.2, x86_64, bubblewrap 0.12.0): cargo fmt --all -- --check; cargo clippy -p mxc-sdk -p lxc -p mxc_schema_gen --all-targets -- -D warnings; cargo test -p mxc-sdk --lib (2,041 pass; 3 fail identically on unmodified main on this host: sdk_v1_conformance::invalid_documents_are_rejected, mxc_pty::tests::native_bridge_cancellation_does_not_inject_eof, and, under load only, provider_monitor::tests::a_monitor_over_an_already_reaped_sandbox_still_settles); the exact-contract, version-boundary, and streaming_bubblewrap integration tests; check-contract-codegen.js, check-schema-versions.js, validate-configs.js.
  • macOS 27 (arm64): cargo fmt --check, cargo clippy -p mxc-sdk --all-targets -- -D warnings, the mxc_unix_resources, validator, and Seatbelt unit tests, and the streaming integration tests (29 pass).
  • Windows 11 Home 26200 (x64): cargo fmt --check, cargo clippy -p mxc-sdk --all-targets -- -D warnings, the job object and validator unit tests (49 pass), and the ignored ProcessContainer streaming tests with --ignored on an elevated host (6 pass).
  • GitHub Actions Build workflow on the fork, all 23 jobs green (lint, build and test on Linux, macOS, and Windows x64 and arm64, versioning, SDK unit and integration tests): https://github.com/crmne/microsoft-mxc/actions/runs/38609552228
  • End to end through the mxc Ruby gem, which builds on this branch, on all three platforms.

✅ Checklist

📋 Issue Type

  • Bug fix
  • Feature
  • Task
Microsoft Reviewers: Open in CodeFlow

MXC confines what a workload can read, write, and reach, but not how
much CPU and memory it can consume, and its result does not say how
much it used (microsoft#1498).

Add an optional `resources` section to the one-shot development
contract: `cpuTimeMs`, `memoryMb`, `maxProcesses`, and `priority`.
Supplying it, even empty, also asks the backend to report the
workload's resource usage in the output metadata, so existing requests
keep their current output.

Details

- Add `Resources` and `ResourcePriority` to the 1.1.0-alpha contract
  and adapt them into `CommonRequestIR` and `ContainerPolicy.resources`.
  Published contracts keep rejecting the field.
- Add `ResourceUsage` and `ResourceLimit` to `SandboxOutputMetadata`
  (`ExecutionMetadata` in the SDK).
- Add `ResourcePolicySupport`, a per-backend declaration checked by
  `validate_common`, so every backend rejects a limit it cannot
  enforce before it creates a sandbox. No backend enforces anything
  yet; the backends opt in with their implementations.
- Skip `resources` when serializing an unset policy so existing policy
  hashes do not change.
- Regenerate the development schema and TypeScript wire types, and
  document the section in docs/schema.md.

Tests

- Contract, adapter, normalization, version-boundary, and
  per-backend validation tests.
Bubblewrap runs accept `resources` but enforce nothing yet. Apply the
limits as rlimits and a nice value on `bwrap`, which every process in
the sandbox inherits, and report the workload's usage.

Details

- Add `mxc_unix_resources`, shared by the Unix backends that launch the
  workload as a direct child: it applies `RLIMIT_CPU` (rounded up to
  whole seconds, soft equal to hard), `RLIMIT_DATA`, and a nice value
  between fork and exec, and peeks at the exited child's usage with
  `waitid(WNOWAIT)` before `std::process::Child` reaps it.
- Limit memory with `RLIMIT_DATA`, not `RLIMIT_AS`: a 2 GB address-space
  limit crashes `soffice --convert-to pdf`, while a 2 GB data limit lets
  it convert and still stops a script allocating 3 GB.
- Start the workload as PID 1 (`--as-pid-1`) when `resources` is
  supplied. bwrap's own PID 1 reaper reports the exit through an eventfd
  and is never waited for, so the workload's usage would not reach MXC.
- Report `limitExceeded: cpuTime` when the workload was killed by
  SIGXCPU, or by SIGKILL after using at least the limit.
- Reject `maxProcesses`: a per-workload count needs a cgroup, and
  RLIMIT_NPROC counts every process of the user.

Tests

- Unit tests for the rlimit translation and limit detection, and a
  live child that inherits the limits and is peeked before the reap.
- Streaming Bubblewrap tests for usage, descendants, CPU time, memory,
  priority, and the maxProcesses rejection.
Seatbelt runs accept `resources` but enforce nothing yet. Apply the CPU
time limit and priority to the process MXC launches, which every
process in the sandbox inherits, and report the workload's usage.

Details

- Set `RLIMIT_CPU` and a nice value between fork and exec, before
  `sandbox_init`, through the shared `mxc_unix_resources`.
- macOS answers `RLIMIT_CPU` with a single SIGXCPU, which a program can
  catch or ignore, and never escalates. Add a watchdog that kills any
  member of the workload's session (or, with inherited stdio, its
  process group) whose own CPU time reached the limit, and report
  `limitExceeded: cpuTime` when it fires. Inherited stdio with a CPU
  limit gets its own process group, as a timeout already does.
- Read usage with `waitid(WNOWAIT)` and `proc_pid_rusage` on the exited
  process before it is reaped. macOS reports only the process's own
  peak footprint, so `peakMemoryBytes` is omitted.
- Reject `memoryMb` (macOS rejects RLIMIT_DATA, RLIMIT_AS, and
  RLIMIT_RSS with EINVAL), `maxProcesses`, and `resources` with the
  legacy `launchMethod: "open"`, whose workload is not MXC's child.

Tests

- Streaming Seatbelt tests for usage with descendants, CPU time
  (including a workload that ignores SIGXCPU), priority, and the
  rejected limits; unit tests for the open-launch rejection. Run on
  macOS 27 (arm64).
- A Bubblewrap test that a workload ignoring SIGXCPU is still stopped.
ProcessContainer runs accept `resources` but enforce nothing yet. Both
the BaseContainer and AppContainer tiers already assign the suspended
workload to a job object before resuming it, so set the limits there,
where they cover every descendant.

Details

- `UiJobObject::apply_resource_limits` sets the per-process user-mode
  time limit, the per-process committed-memory limit, the active
  process limit, and the priority class, keeping
  KILL_ON_JOB_CLOSE, and associates an I/O completion port for the
  job's limit notifications. Both runners call it before assignment and
  fail closed when it fails.
- `UiJobObject::resource_usage` reports the job's total user and kernel
  time, the peak committed memory of its largest process, and the first
  limit notification (END_OF_PROCESS_TIME, PROCESS_MEMORY_LIMIT, or
  ACTIVE_PROCESS_LIMIT). A run MXC ended by timeout or kill reports no
  limit.
- The usage is published in the output metadata after teardown, next
  to any captureDenials output.

Tests

- Job object tests for every limit, the priority class, and the
  notifications, run on Windows 11 (x64).
- Ignored ProcessContainer streaming tests for usage, CPU time, and the
  process limit, run on an elevated host.
`ExecutionMetadata.resource_usage` is public, so its `ResourceUsage`
and `ResourceLimit` types belong in the versioned `mxc_sdk::v1`
namespace rather than only in the hidden runtime models.

Details

- Re-export `ResourceUsage` and `ResourceLimit` from `mxc_sdk::v1`.
- Document both types and the new `ExecutionMetadata` field in the Rust
  V1 API reference, including that the typed V1 request (exact 1.0.0)
  cannot carry `resources` yet.
- Use the public paths in the streaming tests.
@crmne
Carmine Paolino (crmne) requested review from a team and a balanced review from Copilot October 14, 2026 18:24
@crmne
Carmine Paolino (crmne) requested a review from a team as a code owner October 14, 2026 18:24

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Carmine Paolino (crmne) added a commit to crmne/mxc that referenced this pull request Oct 14, 2026
The limits are now submitted to Microsoft as microsoft/mxc#1499, so
the notes point readers at the pull request as well as the issue.
@bbonaby

Copy link
Copy Markdown
Collaborator

Hi Carmine Paolino (@crmne) thanks for creating the Feature request and the PR! That said, since this would affect MXC policy, the team will need to discuss this in more detail. In the future, since this is a feature request it would be better if we first communicate in the issue where it was created so that we can confirm the design, implementation and think about any future consequences of adding the feature. We'll get back to you on this issue once we've discussed. Thanks again!

@bbonaby

Copy link
Copy Markdown
Collaborator

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature request: per-run resource limits, priority, and CPU accounting

3 participants