{{ message }}
tests/run-perfbench.py: Skip large tests based on bm_params.#19128
Open
dpgeorge wants to merge 1 commit intomicropython:masterfrom
Open
tests/run-perfbench.py: Skip large tests based on bm_params.#19128dpgeorge wants to merge 1 commit intomicropython:masterfrom
dpgeorge wants to merge 1 commit intomicropython:masterfrom
Conversation
A large benchmark test cannot run on small targets if the target doesn't have enough RAM to load the test, or to run it. Prior to the change in this commit, `run-perfbench.py` had an explicit list of tests which were too large for small targets to load, and `bm_params` took care of deciding if the target had enough RAM to run the test. Having an explicit list for large test scripts is not very general. This commit improves the situation by using `bm_params` to also decide if the target will be able to load the test. It does this by using a regex to search for `bm_params` and extracting the first pair of N/M values. The M is the minimum memory the target needs in order to run the test, and the test will be skipped entirely (not even loaded) if the target has less than that minimum. This means that tests that are too big to load on the target will be properly skipped, instead of attempting to download to the target and fail with a `MemoryError`. Note: it's not really possible to exec the test script on the host to extract `bm_params` because some tests cannot run under CPython. Signed-off-by: Damien George <damien@micropython.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A large benchmark test cannot run on small targets if the target doesn't have enough RAM to load the test, or to run it. Prior to the change in this commit,
run-perfbench.pyhad an explicit list of tests which were too large for small targets to load, andbm_paramstook care of deciding if the target had enough RAM to run the test.Having an explicit list for large test scripts is not very general. This commit improves the situation by using
bm_paramsto also decide if the target will be able to load the test. It does this by using a regex to search forbm_paramsand extracting the first pair of N/M values. The M is the minimum memory the target needs in order to run the test, and the test will be skipped entirely (not even loaded) if the target has less than that minimum.This means that tests that are too big to load on the target will be properly skipped, instead of attempting to download to the target and fail with a
MemoryError.Note: it's not really possible to exec the test script on the host to extract
bm_paramsbecause some tests cannot run under CPython.Testing
Tested on PYBV10, PYBD_SF6 and ESP32_GENERIC_C3. All tests still run (none are skipped).
Tested on ADAFRUIT_ITSYBITSY_M0_EXPRESS using N=48 and M=20. 8 tests are now skipped because they are too large and the test run completes without failure.
Trade-offs and Alternatives
Instead of regex I thought to
exec()the code on the host to extractbm_params, but as mentioned above that can't work because CPython cannot execute some of the tests (eg due to@micropython.viper).Generative AI
I did not use generative AI tools when creating this PR.