{{ message }}
feat(compiler): Compilation memory ceiling#3786
Open
RafaelGranza wants to merge 6 commits into
Open
Conversation
Contributor
Author
|
Wanted to be sure the 2 GB ceiling doesn't kill honest compiles.
Max is ~1.66 GB, about 19% under the 2048 MB limit. It doesn't grow with concurrency either (address space is a per-process reservation, not shared budget), so no honest compile is killed by the limit on either architecture. (arm64 is Docker on Apple Silicon, amd64 a native 16-core host. On macOS |
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sets two memory defaults for Sierra compilation, from measurements.
What changes
--max-compilation-memory--node-memory-reservePlus: the compile child process is pinned to
GOMAXPROCS=2.Real vs virtual memory
Each compilation runs in its own child process, with two very different memory numbers:
They differ this much because Go reserves far more virtual memory than it uses.
Compilation memory ceiling (2 GB)
--max-compilation-memoryis anRLIMIT_ASlimit, so it caps virtual memory. A compile maps ~1.6 GB of it, so the limit has to be around that, not the ~200 MB of real usage. Capping real memory is not reliable, only trhorughcgroup, which needs special permision in containers.Measured peak virtual memory of the heaviest honest mainnet contract:
2 GB sits just above that peak. This flag also caps how many compile at once, so lowering it from 4 GB doubles concurrency (a 16 GB node goes from 3 to 6 slots) and halves the worst-case ceiling per compile.
Pinning the compile child to GOMAXPROCS=2
A compilation is one Rust FFI call, so extra Go threads do not make it faster; they only add per-thread stacks that grow the virtual memory peak. Inheriting a large host GOMAXPROCS pushes the memory peak up (1.93 GB at 128 cores, near the limit). Pinning keeps it host-independent (~1.58 GB) at no time cost.
Node memory reserve (4 GB)
Measured on a real synced mainnet node via
process_resident_memory_bytes, which counts the node's whole footprint together: Go heap, pebble (DB) cache, state, contract execution and RPC.4 GB covers the peak with headroom.
Validation
Checked derivation, fallback and virtual peak across Docker (arm64), native macOS (arm64) and a native amd64 host, from 1 to 128 cores. Build, tests and lint green.