iframe-proxy

projectgus · 2024-11-27T06:56:56Z

Summary

This PR is attempting to bring some CircuitPython changes upstream to MicroPython. From these PRs:

The changes:

Implement overflow checks for buffer conversions, in a similar but more general way to py/objint: Fix int.to_bytes() buffer size checks. #13087.
Add optional int.to_bytes(signed=bool) parameter. This removes the CPython incompatibility where currently int.to_bytes in MicroPython behaves as if signed=self < 0 when converting.

Note that this may require MicroPython code changes where it previously behaved differently to CPython:

Anywhere that int.to_bytes() was being called on a negative number will now raise OverflowError. The fix is to add signed=True.
Anywhere that int.to_bytes() was silently truncating the output will now raise OverflowError. The fix is either to ensure the length is sufficient for the value, or mask the value before converting it (i.e. (x & 0xFFFF).to_bytes(2).

After cherry-picking commits directly from CircuitPython and resolving conflicts, some additional changes were needed (and in the end I've squashed most of them down into one):

Update existing MicroPython tests to pass. Many tests are now simpler, as behaviour now closer to CPython. Some tests needed to be converted to unittest to verify both current and MP V2 behaviour is correct.
Restore optional passing of arguments for byteorder and length, as per py/objint: Make byteorder argument optional in byte-conversion methods. #14387 and CPython >=3.11.
Do not raise OverflowError from array constructors if a value is out of range, truncate the value instead. This is a documented difference between CPython and MicroPython, and after discussing with @dpgeorge decided changing it here has the potential to break existing MP code. The overflow checks are kept if MICROPY_PREVIEW_VERSION_2 is set, so that we can change behaviour to match CPython (and CircuitPython) in MicroPython 2. I guess Python code size may become a littler bigger if masking needs to be added, but it also eliminates a potential silent bug.
Additional refactoring commit to merge similar code paths and bring the code size back down closer to current master.
Double the allowed stack size on the Windows port. This seems necessary to allow debug builds to pass the new unit tests in this PR.
Remove C function mp_binary_set_val_array_from_int - after the above refactors this was no longer used except in stm32/adc.c and a coverage test. Removing it reduces the stm32 port code size substantially.

Testing

Unit tests passing on unix and rp2 ports.
Unit tests passing on Zephyr port (uses long long implementation for bigint).

Trade-offs and Alternatives

We already have some of this bounds checking functionality in master now, but the approach taken by CircuitPython is more general and has potential to apply the same checks in multiple places (i.e. array constructors) with no further impact on code size.

codecov · 2024-11-27T07:02:39Z

github-actions · 2024-11-27T07:09:45Z

Code size report:

Reference:  Revert "rp2: Build with nano.specs for newlib-nano.". [b67ac73]
Comparison: py/binary,objint: Add overflow checks and int.to_bytes(signed=True). [merge of 5ecf096]
  mpy-cross:  +256 +0.067% [incl +32(data)]
   bare-arm:  +100 +0.177% 
minimal x86:  +366 +0.196% [incl +32(data)]
   unix x64:  +256 +0.030% standard[incl +32(data)]
      stm32:  +104 +0.026% PYBV10
      esp32:   -76 -0.004% ESP32_GENERIC[incl +80(data)]
     mimxrt:   +88 +0.022% TEENSY40
        rp2:   +96 +0.010% RPI_PICO_W
       samd:  +120 +0.043% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:   -10 -0.002% VIRT_RV32

projectgus · 2025-02-26T06:31:02Z

Only two things left here...

Project coverage has somehow gone down even though the patch coverage is 100%. I've added some additional cove in coverage.c but it's still not quite enough.
The Windows x64 Debug builds are failing in the uctypes_load_store test with RuntimeError: maximum recursion depth exceeded. I don't know if this is an actual stack usage problem, a bug triggered when a unittest assert fails, or something else...

projectgus · 2025-02-26T06:32:56Z

The Windows x64 Debug builds are failing in the uctypes_load_store test with RuntimeError: maximum recursion depth exceeded

Ah OK, there's a whole section about stack usage in the windows port README so I'm guessing we're just running out of stack.

projectgus · 2025-03-04T05:10:57Z

      stm32:  +156 +0.040% PYBV10

Had a quick look into why stm32 got bigger while all the other bare metal ports got smaller. Didn't find anything unusual: no significant new symbols are being linked compared to master. Several functions shrunk as they now call a common helper. I think maybe they just didn't shrink as much as on the other bare metal ports...

Diff of linked symbol sizes

--- ./build-PYBV10-master/firmware.syms 2025-03-04 15:59:56.145991980 +1100
+++ ./build-PYBV10/firmware.syms        2025-03-04 16:00:08.656797386 +1100
@@ -5948,0 +5949 @@
+00000018 t allowed_args.0
@@ -6199 +6200 @@
-00000264 t compile_scope_inline_asm
+00000260 t compile_scope_inline_asm
@@ -6503 +6504 @@
-00000970 t emit_inline_thumb_op
+00000978 t emit_inline_thumb_op
@@ -7326 +7327 @@
-000000d8 t int_to_bytes
+0000005c t int_to_bytes
@@ -7770,3 +7771,3 @@
-0000002a T mp_binary_set_int
-00000114 T mp_binary_set_val
-0000008c T mp_binary_set_val_array
+0000005a T mp_binary_set_int
+000000f2 T mp_binary_set_val
+0000006e T mp_binary_set_val_array
@@ -8329 +8330 @@
-00000042 T mp_obj_exception_attr
+00000040 T mp_obj_exception_attr
@@ -8373 +8374 @@
-00000016 T mp_obj_int_to_bytes_impl
+00000114 T mp_obj_int_to_bytes
@@ -8521 +8522 @@
-0000091c T mp_qstr_const_hashes
+00000922 T mp_qstr_const_hashes
@@ -8523 +8524 @@
-0000048e T mp_qstr_const_lengths
+00000491 T mp_qstr_const_lengths
@@ -8525 +8526 @@
-00001250 T mp_qstr_const_pool
+0000125c T mp_qstr_const_pool
@@ -8891 +8892 @@
-000000d2 T mpz_as_bytes
+0000009e T mpz_as_bytes
@@ -9768 +9769 @@
-00000084 t set_aligned
+00000080 t set_aligned
@@ -10078 +10079 @@
-0000007c t task_attr
+00000078 t task_attr
@@ -10216 +10217 @@
-00000200 t uctypes_struct_attr_op
+00000204 t uctypes_struct_attr_op

(Files generated with nm --print-size firmware.elf | cut -d' ' -f2- | tee firmware.syms)

I don't see much we can do to improve this. Interestingly, the firmware size increase on PYBV10 is about the same if LTO is enabled (although the firmwares are overall smaller.)

dpgeorge · 2025-03-04T05:16:45Z

Had a quick look into why stm32 got bigger while all the other bare metal ports got smaller. Didn't find anything unusual

What about:

-00000016 T mp_obj_int_to_bytes_impl
+00000114 T mp_obj_int_to_bytes

That's 254 extra bytes. Is that similar on other ports that did decrease?

projectgus · 2025-03-04T06:15:08Z

That's 254 extra bytes. Is that similar on other ports that did decrease?

I think so, this is the common function that the others now call. They grow by basically the same amount on both samd and rp2 ports. Here's samd:

@@ -6342 +6338 @@
-00000016 T mp_obj_int_to_bytes_impl
+00000114 T mp_obj_int_to_bytes

Full size diffs

--- build-ADAFRUIT_ITSYBITSY_M4_EXPRESS-master/firmware.syms    2025-03-04 17:06:29.044966391 +1100
+++ build-ADAFRUIT_ITSYBITSY_M4_EXPRESS/firmware.syms   2025-03-04 17:07:24.983870265 +1100
@@ -993 +992,0 @@
-t $d
@@ -4381,3 +4379,0 @@
-t $t
-t $t
-t $t
@@ -4404,0 +4401 @@
+00000018 t allowed_args.0
@@ -4620 +4617 @@
-00000260 t compile_scope_inline_asm
+00000264 t compile_scope_inline_asm
@@ -4822 +4819 @@
-00000968 t emit_inline_thumb_op
+00000970 t emit_inline_thumb_op
@@ -5312 +5309 @@
-000000d8 t int_to_bytes
+0000005c t int_to_bytes
@@ -5758,4 +5755,3 @@
-0000002a T mp_binary_set_int
-0000016c T mp_binary_set_val
-0000008c T mp_binary_set_val_array
-000000f2 T mp_binary_set_val_array_from_int
+0000005a T mp_binary_set_int
+00000150 T mp_binary_set_val
+0000006e T mp_binary_set_val_array
@@ -5777 +5773 @@
-0000008c t mp_builtin_compile
+00000088 t mp_builtin_compile
@@ -6300 +6296 @@
-0000003a T mp_obj_exception_attr
+00000038 T mp_obj_exception_attr
@@ -6342 +6338 @@
-00000016 T mp_obj_int_to_bytes_impl
+00000114 T mp_obj_int_to_bytes
@@ -6477 +6473 @@
-00000542 T mp_qstr_const_hashes
+00000536 T mp_qstr_const_hashes
@@ -6479 +6475 @@
-000002a1 T mp_qstr_const_lengths
+0000029b T mp_qstr_const_lengths
@@ -6481 +6477 @@
-00000a9c T mp_qstr_const_pool
+00000a84 T mp_qstr_const_pool
@@ -6802 +6798 @@
-000000d2 T mpz_as_bytes
+0000009e T mpz_as_bytes
@@ -7172 +7168 @@
-00000084 t set_aligned
+00000080 t set_aligned
@@ -7245 +7241 @@
-00000034 t slice_attr
+00000030 t slice_attr
@@ -7402 +7398 @@
-00000070 t task_attr
+0000007c t task_attr
@@ -7553 +7549 @@
-00000200 t uctypes_struct_attr_op
+00000204 t uctypes_struct_attr_op
@@ -7599 +7595 @@
-00000038 t usb_device_attr
+00000034 t usb_device_attr

--- build-RPI_PICO-master/firmware.syms        2025-03-04 17:01:17.898379768 +1100
+++ build-RPI_PICO/firmware.syms 2025-03-04 17:01:28.760234428 +1100
@@ -31,4 +30,0 @@
-t $d
-t $d
-t $d
-t $d
@@ -244 +239,0 @@
-t $d
@@ -630,0 +626,2 @@
+t $d
+r $d
@@ -1202,0 +1200 @@
+t $d
@@ -1536,0 +1535 @@
+t $d
@@ -5462,3 +5460,0 @@
-t $t
-t $t
-t $t
@@ -5503,0 +5500 @@
+00000018 r allowed_args.0
@@ -5826 +5823 @@
-00000278 t compile_scope_inline_asm
+00000274 t compile_scope_inline_asm
@@ -6144 +6141 @@
-00000498 t emit_inline_thumb_op
+00000494 t emit_inline_thumb_op
@@ -6781 +6778 @@
-000000d4 t int_to_bytes
+00000060 t int_to_bytes
@@ -6786 +6783 @@
-00000044 t iobase_ioctl
+00000040 t iobase_ioctl
@@ -7352,4 +7349,3 @@
-0000002a T mp_binary_set_int
-00000154 T mp_binary_set_val
-00000084 T mp_binary_set_val_array
-00000084 T mp_binary_set_val_array_from_int
+00000056 T mp_binary_set_int
+0000012e T mp_binary_set_val
+00000064 T mp_binary_set_val_array
@@ -7370 +7366 @@
-00000064 t mp_builtin_compile
+00000068 t mp_builtin_compile
@@ -7885 +7881 @@
-0000005c T mp_native_type_from_qstr
+00000064 T mp_native_type_from_qstr
@@ -7962 +7958 @@
-00000016 T mp_obj_int_to_bytes_impl
+00000110 T mp_obj_int_to_bytes
@@ -8109 +8105 @@
-000002ee R mp_qstr_const_hashes
+000002f1 R mp_qstr_const_hashes
@@ -8111 +8107 @@
-000002ee R mp_qstr_const_lengths
+000002f1 R mp_qstr_const_lengths
@@ -8113 +8109 @@
-00000bd0 R mp_qstr_const_pool
+00000bdc R mp_qstr_const_pool
@@ -8355 +8351 @@
-00000064 T mp_vfs_blockdev_init
+00000060 T mp_vfs_blockdev_init
@@ -8365 +8361 @@
-00000068 T mp_vfs_getcwd
+0000006c T mp_vfs_getcwd
@@ -8367,2 +8363,2 @@
-00000060 T mp_vfs_ilistdir
-00000070 t mp_vfs_ilistdir_it_iternext
+00000064 T mp_vfs_ilistdir
+00000074 t mp_vfs_ilistdir_it_iternext
@@ -8370 +8366 @@
-0000007c T mp_vfs_import_stat
+0000007a T mp_vfs_import_stat
@@ -8433 +8429 @@
-0000002e T mp_vfs_rename
+00000030 T mp_vfs_rename
@@ -8435 +8431 @@
-00000016 T mp_vfs_rmdir
+00000018 T mp_vfs_rmdir
@@ -8437 +8433 @@
-00000040 T mp_vfs_stat
+0000003c T mp_vfs_stat
@@ -8439 +8435 @@
-00000054 T mp_vfs_statvfs
+00000058 T mp_vfs_statvfs
@@ -8441 +8437 @@
-0000009c T mp_vfs_umount
+000000a0 T mp_vfs_umount
@@ -8450 +8446 @@
-00000140 T mpz_as_bytes
+000000ea T mpz_as_bytes
@@ -9025 +9021 @@
-00000084 t set_aligned
+0000007c t set_aligned
@@ -9102 +9098 @@
-00000038 t slice_attr
+00000034 t slice_attr

Ah! But the very big difference on these ports is that, unlike stm32, the mp_binary_set_val_array_from_int function is no longer linked in this PR! That's about 127 bytes.

That's the function which is still called from stm32/adc.c . So if we can find another way to implement that, the code would go down on stm32 (and we also wouldn't need the changes in #16858).

projectgus · 2025-03-04T06:15:52Z

EDIT: Accidentally buried the useful info in the <details> tag above.

projectgus · 2025-03-04T06:25:49Z

That's the function which is still called from stm32/adc.c . So if we can find another way to implement that, the code would go down on stm32 (and we also wouldn't need the changes in #16858).

Specifically pyb.ADC.read_timed and read_timed_multi are documented as supporting either an array of 8/16-bit integers or a bytearray.

mp_binary_set_val_array_from_int supports any numeric typecode. So I think there's potentially a much simpler version we can embed into adc.c and remove the other function entirely. But probably best to do in a separate PR.

Support signed param: result = int.from_bytes(bytearray(), order='big'|'little', signed=False|True) Add `length`, `byteorder`, `signed` according to the micropython#16311. Signed-off-by: Ihor Nehrutsa <Ihor.Nehrutsa@gmail.com>

projectgus · 2026-06-23T23:37:40Z

Yes. Although the error message is different to CPython, the exception type is the same and the behaviour is the same.

Hmm. This exception is the most visible "breaking change" for MicroPython code which worked before this PR, so maybe we can add an explanatory error message similar to CPython (as this one is a bit obscure).

dpgeorge · 2026-06-24T01:41:28Z

This exception is the most visible "breaking change" for MicroPython code which worked before this PR,

I don't follow. In current master we get:

>>> x=-2
>>> x.to_bytes(signed=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: function doesn't take keyword arguments

So this never worked. Or do you mean that x.to_bytes() works on current master and this PR makes that not work (now matching CPython)?

projectgus · 2026-06-24T05:01:26Z

Or do you mean that x.to_bytes() works on current master and this PR makes that not work (now matching CPython)?

That's the one. (-2).to_bytes() will start raising an exception with this change.

This seems like it's only really a problem on Debug builds, but I think can't hurt to increase it on all windows builds. This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <angus@redyak.com.au>

This removes the last usage of mp_binary_set_val_array_from_int(). It will be a little slower, but shouldn't be measurably so compared to the ADC sampling. This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <angus@redyak.com.au>

If >1 unittest-enabled module is included, the results of the merged module won't match (as it runs some previously registered tests again). Signed-off-by: Angus Gratton <angus@redyak.com.au>

dpgeorge · 2026-06-24T05:47:31Z

-                    mp_binary_set_int(GET_SCALAR_SIZE(val_type & 7), self->flags == LAYOUT_BIG_ENDIAN,
-                        self->addr + offset, val);
+                    size_t item_size = GET_SCALAR_SIZE(val_type & 7);
+                    mp_binary_set_int(item_size, self->addr + offset, item_size, val, self->flags == LAYOUT_BIG_ENDIAN);


As I understand it, this is a breaking change in behaviour because it can now raise if val doesn't fit.

Is it worth retaining existing behaviour simply by masking val as appropriate so that it always fits? (Then the uctypes_array_load_store.py test does not need to change.)

Not quite, mp_binary_set_int() doesn't ever raise - the API just changed a little so it could be called from more places.

The changes in uctypes_array_load_store.py are to account for bytearray setters gaining overflow checks in MicroPython V2.0 (because moductypes creates bytearrays internally).

Not quite, mp_binary_set_int() doesn't ever raise - the API just changed a little so it could be called from more places.

Ah, I see.

The changes in uctypes_array_load_store.py are to account for bytearray setters gaining overflow checks in MicroPython V2.0 (because moductypes creates bytearrays internally).

Hmm, I don't see the original version of that test storing into a bytearray? It creates a bytearray only as a backing buffer for the uctypes struct.

Hmm, yeah... I would classify my confidence about moductypes internals as "very low" but my understanding is that the relevant code path is here:
https://github.com/micropython/micropython/blob/master/extmod/moductypes.c#L502

Can see it by doing something like this:

>>> b = bytearray(5) >>> desc = {"arr": (uctypes.ARRAY | 0, uctypes.UINT8 | 5)} >>> s = uctypes.struct(uctypes.addressof(b), desc, uctypes.LITTLE_ENDIAN) >>> type(s.arr) <class 'bytearray'> >>> desc = {"arr": (uctypes.ARRAY | 0, uctypes.INT8 | 5)} >>> s = uctypes.struct(uctypes.addressof(b), desc, uctypes.LITTLE_ENDIAN) >>> type(s.arr) <class 'struct'>

Which is why the is_v2 differences in the test are all gated behind if is_v2 and item_sz == 1. (I'm not actually sure why the test passes here for INT8 when expecting the OverflowError, maybe all of the test values are in-range for INT8 but not UINT8...?)

This is still a weird change I admit, because other kinds of moductypes arrays won't add overflow checks in V2 but UINT8 will...

So can think of two possible ways forward:

Change the test to explicitly check if isinstance(s.arr, bytearray) which will make the reason for the special case in the test more obvious.

Remove the moductypes optimisation of treating UINT8 specially as a bytearray, so you always get back a struct object of some kind. I'm not sure what the other trade-offs of this might be.

Any preference?

Interesting. In your snip above the second one (INT8) is not a bytearray, so that's why the check is not done.

If you modify extmod/uctypes_array_load_store.py to add print(s.arr) just before the inner for loop, you'll see that everything except UINT8 are internal uctypes struct's, but the UINT8 case is a bytearray.

Hmm...

2. Remove the moductypes optimisation of treating UINT8 specially as a bytearray, so you always get back a struct object of some kind. I'm not sure what the other trade-offs of this might be.

That would make things cleaner/simpler, and not give any regressions wrt overflow checks.

But, it would probably give regressions in other areas, eg uctypes.struct can't be sliced, iterated, etc. Whereas bytearray can be.

Could we return a memoryview instead of bytearray? Or does memoryview also do overflow checking with this PR?

Change the test to explicitly check if isinstance(s.arr, bytearray) which will make the reason for the special case in the test more obvious.

I think at the very least do this option. Then it's clear what's going on.

Interesting. In your snip above the second one (INT8) is not a bytearray, so that's why the check is not done.

Yes, sorry that's what I meant - the part I didn't get is why item_sz == 1 is a sufficient check given it matches both INT8 and UINT8 and they're different types.

I get it now, it's because the range of values used in each test is [-2, -1, 0, 1, 2]. So they're always in-range for all signed types, meaning that the full check if is_v2 and item_sz == 1 and (n < item_min or n > item_max): only hits the special-case for UINT8.

So, I can definitely rewrite the test to make the behaviour clearer. But that still depends on keeping the behaviour that an array of UINT8 is a bytearray type not the same struct type as the other arrays.

To reword the options I can think of:

Change the test to explicitly check if isinstance(s.arr, bytearray) which will make the reason for the special case in the test more obvious. MicroPython V2 will have a breaking change for moductypes code that uses arrays of UINT8 and relies on implicit truncation, same as any other code which uses bytearray and relies on implicit truncation.

Remove the V2 bounds check for bytearray setters (making it different to the array module). Don't recommend this for the reasons given in the PR description (possibility for silent bugs, difference to CPython).

Remove the moductypes optimisation of having a UINT8 array be a bytearray, so the arr element always gives back a struct object of some kind. This will keep all moductypes behaviour the same in MicroPython V2. I'm not sure what the other trade-offs of not returning a bytearray in this case would be.

Oops, our comments overlapped. I'll do option 1 for now, we can always rethink the V2 behaviour change later.

Could we return a memoryview instead of bytearray? Or does memoryview also do overflow checking with this PR?

With V2 it will, yes. Same code path.

Have pushed a new test which makes the reason for the behaviour more explicit.

dpgeorge · 2026-06-24T06:08:52Z

Following up on octoprobe failures: I ran the current version of this PR on ADAFRUIT_ITSYBITSY_M0_EXPRESS and there are 3 new failures introduced by this PR.

$ ./run-tests.py -t a0 --via-mpy --emit native

FAIL `basics/array_int_repr.py`

Traceback (most recent call last):
  File "<stdin>", line 37, in <module>
  File "unittest/__init__.py", line 70, in <module>
  File "unittest/__init__.py", line 205, in TestCase
MemoryError: memory allocation failed, allocating 232 bytes

FAIL `basics/array_limits_intbig.py`

Traceback (most recent call last):
  File "<stdin>", line 37, in <module>
MemoryError: memory allocation failed, allocating 160 bytes

FAIL `extmod/uctypes_array_load_store.py`

Traceback (most recent call last):
  File "<stdin>", line 37, in <module>
MemoryError: memory allocation failed, allocating 302 bytes

projectgus · 2026-06-24T06:30:01Z

Following up on octoprobe failures: I ran the current version of this PR on ADAFRUIT_ITSYBITSY_M0_EXPRESS and there are 3 new failures introduced by this PR.

Thanks @dpgeorge. So if I understand correctly these are all running out of memory after the test module starts executing but before any test case runs?

dpgeorge · 2026-06-24T06:35:39Z

So if I understand correctly these are all running out of memory after the test module starts executing but before any test case runs?

Yes, I think that's the case. Although I'm not 100% certain because it's hard to tell with the native emitter due to limited traceback info. I'll see if I can dig deeper to see exactly which allocation is failing.

dpgeorge · 2026-06-24T13:48:01Z

I'll see if I can dig deeper to see exactly which allocation is failing.

I looked deeper into the above 3 failing tests. They all fail trying to import unittest.

I added a gc.collect() and micropython.mem_info() to all 3 tests just before they import unittest. And then printed a message after import unittest. I get (using #19329):

$ ./run-tests.py -t a0 --via-mpy --emit native --trace-output basics/array_int_repr.py basics/array_limits_intbig.py extmod/uctypes_array_load_store.py
platform=samd arch=armv6m inlineasm=thumb float=32-bit
TRACE: tests/basics/array_int_repr.py
START TEST
stack: 1276 out of 7168
GC: total: 21248, used: 10272, free: 10976
 No. of 1-blocks: 57, 2-blocks: 11, max blk sz: 229, max free sz: 324
Traceback (most recent call last):
  File "<stdin>", line 37, in <module>
  File "unittest/__init__.py", line 70, in <module>
  File "unittest/__init__.py", line 169, in TestCase
MemoryError: memory allocation failed, allocating 184 bytes
FAIL  tests/basics/array_int_repr.py
NOTE: tests/basics/array_int_repr.py may be a unittest that doesn't run unittest.main()
TRACE: tests/basics/array_limits_intbig.py
START TEST
stack: 1276 out of 7168
GC: total: 21248, used: 16080, free: 5168
 No. of 1-blocks: 73, 2-blocks: 18, max blk sz: 395, max free sz: 34
Traceback (most recent call last):
  File "<stdin>", line 37, in <module>
MemoryError: memory allocation failed, allocating 80 bytes
FAIL  tests/basics/array_limits_intbig.py
NOTE: tests/basics/array_limits_intbig.py may be a unittest that doesn't run unittest.main()
TRACE: tests/extmod/uctypes_array_load_store.py
START TEST
stack: 1276 out of 7168
GC: total: 21248, used: 12400, free: 8848
 No. of 1-blocks: 57, 2-blocks: 14, max blk sz: 295, max free sz: 152
Traceback (most recent call last):
  File "<stdin>", line 37, in <module>
MemoryError: memory allocation failed, allocating 302 bytes
FAIL  tests/extmod/uctypes_array_load_store.py
3 tests performed (34 individual testcases)
0 tests passed
3 tests failed: tests/basics/array_int_repr.py tests/basics/array_limits_intbig.py tests/extmod/uctypes_array_load_store.py

The message after import unittest never prints, so they all fail on that import.

unittest itself requires just a little over 10k to import, and at the point of import the above tests all have 10k or less free. So that's why the import -- and hence the test -- fails.

There are some small optimisations we can make to unittest to get it down in size, but that'll probably only help basics/array_int_repr.py which is just on the edge of having enough heap memory. The other 2 tests just cannot run on this device with 20k total heap.

projectgus · 2026-06-25T00:36:15Z

The message after import unittest never prints, so they all fail on that import.

In this case, how about:

try:
    import unittest
except MemoryError:
    print("SKIP")
    raise SystemExit

... at least, for now. It's more boilerplate than I prefer, and there's a potential gap for a test which can import unittest but then can't run the test (although maybe the peak memory usage will still happen during the import on a lot of tests?) However, sounds like it will at least catch these particular failures.

dpgeorge · 2026-06-25T01:05:47Z

In this case, how about:

I tried that (although used "SKIP-TOO-LARGE" instead for more detailed reporting) and it does skip correctly on all 3 tests with the native emitter (on ADAFRUIT_ITSYBITSY_M0_EXPRESS).

BUT, running just --via-mpy (bytecode emitter, not native emitter) on the basics/array_limits_intbig.py it fails with MemoryError part way through (as you predicted!):

test_bigint_overflow (__main__.Test) ... ok
test_unsigned_negative (__main__.Test) ... ok
test_zero (__main__.Test) ... ok
test_one (__main__.Test) ... ok
test_array (__main__.Test) ... ok
test_setter_overflow_Q (__main__.Test) ...Traceback (most recent call last):
  File "<stdin>", line 37, in <module>
  File "tests/basics/array_limits_intbig.py", line 118, in <module>
  File "unittest/__init__.py", line 464, in main
  File "unittest/__init__.py", line 269, in run
  File "unittest/__init__.py", line 254, in run
  File "unittest/__init__.py", line 432, in _run_suite
  File "unittest/__init__.py", line 407, in run_one
  File "unittest/__init__.py", line 350, in _handle_test_exception
  File "unittest/__init__.py", line 339, in _capture_exc
MemoryError: memory allocation failed, allocating 136 bytes
CRASH

Maybe we can live with that for now (so, include that try-import-unittest-except logic), and I can make a few optimisations to unittest to bring its size down.

projectgus · 2026-06-25T03:40:36Z

@dpgeorge Thanks for working through these. I've updated the tests according to the discussion.

dpgeorge · 2026-06-25T04:36:51Z

I made a PR to optimise unittest micropython/micropython-lib#1125 . Using that version of unittest the tests above can all run or skip on ADAFRUIT_ITSYBITSY_M0_EXPRESS.

This PR was originally a cherry-pick of CircuitPython commits 095c844, d103ac1, c592bd6 and 8664a65. However, substantial additional changes so the implementation has diverged a lot from CircuitPython's: - Keep CPython >= 3.11 defaults for int.to_bytes(), consistent with 80c5e76 and 0b432b3. - Refactors to reduce the code size impact of this change. Various code paths are now funneled into new function mp_obj_int_to_bytes() and existing function mp_binary_set_int(), except where a simple assignment is used for performance reasons (i.e. array, moductypes). - Keep the MicroPython behaviour of not overflow checking assignments to arrays, bytearrays, etc. - but enable overflow checks as part of MicroPython V2.0. - Update tests to work with the new behaviour (similar to CPython) and add coverage for new code and some corner cases. Some tests are converted to unittest so we can easily verify both current and V2.0 overflow. This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <angus@redyak.com.au>

projectgus · 2026-06-25T04:45:48Z

I made a PR to optimise unittest micropython/micropython-lib#1125 . Using that version of unittest the tests above can all run or skip on ADAFRUIT_ITSYBITSY_M0_EXPRESS.

That's great! Do you want to keep the try/except guards in these tests, or pull them out?

dpgeorge · 2026-06-25T04:57:27Z

dpgeorge

Thanks for persisting with this PR, I know it's taken a while to get to this stage with a lot of back and forth.

It looks really good now. Nice to finally have int.to_bytes(signed) support.

Let's merge it!

projectgus added the py-core Relates to py/ directory in source label Nov 27, 2024

projectgus force-pushed the feature/int_tobytes_signed branch from 77160cc to 0fb050a Compare November 27, 2024 06:59

projectgus force-pushed the feature/int_tobytes_signed branch 3 times, most recently from c84ab58 to 88b516a Compare December 4, 2024 07:27

projectgus force-pushed the feature/int_tobytes_signed branch from 88b516a to 140c2c9 Compare December 11, 2024 05:14

projectgus mentioned this pull request Dec 17, 2024

extmod/uctypes: fix addressof when returning large values #16440

Merged

projectgus mentioned this pull request Feb 24, 2025

py/objint.c: Add signed param to int.from_bytes(). #16798

Open

projectgus force-pushed the feature/int_tobytes_signed branch 4 times, most recently from 4faf739 to b3b3d4e Compare February 25, 2025 07:14

projectgus commented Feb 25, 2025

View reviewed changes

Comment thread ports/unix/coverage.c Outdated

projectgus mentioned this pull request Mar 4, 2025

unix/coverage: Increase coverage of mp_binary_set_val_array_from_int. #16858

Closed

projectgus force-pushed the feature/int_tobytes_signed branch from 2c3f941 to 6f3c7e0 Compare March 4, 2025 05:11

projectgus commented Mar 4, 2025

View reviewed changes

Comment thread py/binary.c

projectgus commented Mar 4, 2025

View reviewed changes

Comment thread py/binary.c Outdated

dpgeorge reviewed Jun 23, 2026

View reviewed changes

Comment thread py/objint.c Outdated

Comment thread py/objint.c Outdated

projectgus added 3 commits June 24, 2026 15:25

windows: Double the stack size allowance on windows port.

c9bee3c

This seems like it's only really a problem on Debug builds, but I think can't hurt to increase it on all windows builds. This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <angus@redyak.com.au>

tools/ci: Don't include unittest tests in the merged .mpy.

4a83dbe

If >1 unittest-enabled module is included, the results of the merged module won't match (as it runs some previously registered tests again). Signed-off-by: Angus Gratton <angus@redyak.com.au>

projectgus force-pushed the feature/int_tobytes_signed branch from 4342ec3 to 3a5beb4 Compare June 24, 2026 05:31

dpgeorge reviewed Jun 24, 2026

View reviewed changes

projectgus force-pushed the feature/int_tobytes_signed branch from 3a5beb4 to 9a8d0ba Compare June 24, 2026 07:04

dpgeorge reviewed Jun 24, 2026

View reviewed changes

Comment thread tests/extmod/uctypes_array_load_store.py Outdated

projectgus force-pushed the feature/int_tobytes_signed branch from 9a8d0ba to fa992c4 Compare June 25, 2026 00:33

projectgus force-pushed the feature/int_tobytes_signed branch from fa992c4 to f6d4441 Compare June 25, 2026 03:40

dpgeorge reviewed Jun 25, 2026

View reviewed changes

Comment thread tests/basics/array_limits_intbig.py Outdated

dpgeorge mentioned this pull request Jun 25, 2026

unittest: optimise for code size and memory use micropython/micropython-lib#1125

Open

projectgus force-pushed the feature/int_tobytes_signed branch from f6d4441 to 5ecf096 Compare June 25, 2026 04:45

dpgeorge approved these changes Jun 25, 2026

View reviewed changes

dpgeorge merged commit e00daa3 into micropython:master Jun 25, 2026
76 checks passed

Sunbelt Computer Software

PL/B Language Development and Support

Uh oh!

Uh oh!

Conversation

projectgus commented Nov 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing

Trade-offs and Alternatives

Uh oh!

codecov Bot commented Nov 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions Bot commented Nov 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

projectgus commented Feb 26, 2025

Uh oh!

projectgus commented Feb 26, 2025

Uh oh!

projectgus commented Mar 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dpgeorge commented Mar 4, 2025

Uh oh!

projectgus commented Mar 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

projectgus commented Mar 4, 2025

Uh oh!

projectgus commented Mar 4, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

projectgus commented Jun 23, 2026

Uh oh!

dpgeorge commented Jun 24, 2026

Uh oh!

projectgus commented Jun 24, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

projectgus Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dpgeorge commented Jun 24, 2026

FAIL basics/array_int_repr.py

FAIL basics/array_limits_intbig.py

FAIL extmod/uctypes_array_load_store.py

Uh oh!

projectgus commented Jun 24, 2026

Uh oh!

dpgeorge commented Jun 24, 2026

Uh oh!

dpgeorge commented Jun 24, 2026

Uh oh!

Uh oh!

projectgus commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

projectgus commented Nov 27, 2024 •

edited

Loading

codecov Bot commented Nov 27, 2024 •

edited

Loading

github-actions Bot commented Nov 27, 2024 •

edited

Loading

projectgus commented Mar 4, 2025 •

edited

Loading

projectgus commented Mar 4, 2025 •

edited

Loading

projectgus Jun 24, 2026 •

edited

Loading

FAIL `basics/array_int_repr.py`

FAIL `basics/array_limits_intbig.py`

FAIL `extmod/uctypes_array_load_store.py`

projectgus commented Jun 25, 2026 •

edited

Loading