iframe-proxy

m-u-xyz · 2022-06-20T14:51:06Z

This backports 3.11's TaskGroup class to uasyncio.

I don't care much about the except * stuff, not in µPy context anyway, but sane error recovery is crucial and taskgroups make this job a whole lot easier, not to mention much less bug prone. (Writing from a lot of Trio and anyio experience here.)

Bottom line: I refuse to write async code without using taskgroups. So here you are.

~~TODO: Write a couple of tests.~~

Closes #8508.

m-u-xyz · 2022-06-20T20:14:12Z

dpgeorge · 2022-06-21T00:17:35Z

Wow, thanks for this! It's very nice to see that it can be implemented in pure Python. I don't know anything about TaskGroup so will need to learn how it works.

dpgeorge · 2022-06-21T00:18:05Z

It's not my commit message that's failing the test!

You're right. That CI check will need fixing so it allows "Revert" commit messages.

dpgeorge · 2022-06-21T04:10:16Z

It's not my commit message that's failing the test!

You're right. That CI check will need fixing so it allows "Revert" commit messages.

Actually the CI check is working OK, it's only supposed to check the commit messages that are unique to the PR.

If you rebase (and force push) on latest master it will hopefully get the CI green.

m-u-xyz · 2022-06-21T06:38:25Z

Pushed. Writing tests, found a significant bug (related to cancellation of course). Investigating.

m-u-xyz · 2022-06-22T19:44:14Z

Huh. I have no idea where those test failures are coming from.

dlech · 2022-06-22T19:56:59Z

The CI logs say error is:

+micropython-coverage: ../../py/emitnative.c:2806: emit_native_raise_varargs: Assertion `n_args == 1' failed.

It looks like the bytecode emitter allows 2 args but the native emitter only allows 1.

m-u-xyz · 2022-06-22T20:16:31Z

It looks like the bytecode emitter allows 2 args but the native emitter only allows 1.

Well, yeah, I saw that line, but what has that got to do with implementing __hash__ and/or how do I fix that? I didn't touch no emitters … so either I'm doing something really dumb here, or I found a bug which I'm not qualified to fix, esp. since the code works on my system(s).

dlech · 2022-06-23T19:45:04Z

Assuming you are using the unix port, to reproduce locally, try make -C ports/unix test_full or run micropython with -X emit=native.

There are a number of tests that are disabled when emit=native because of raise_varargs already, basics/try_reraise.py, basics/try_reraise2 and basics/try_finally_return2.py to name a few (search for raise_varargs in tests/run-tests.py).

I suspect you have written some Python code similar to one of these that causes the same error.

peterhinch · 2022-06-25T17:22:29Z

I'm running this on a Pyboard 1.1 - TaskGroups are a very nice feature.

Something Google seems loath to divulge is how to access return values from the members of a group (assuming they terminate normally).

m-u-xyz · 2022-06-25T17:35:40Z

@peterhinch Glad that my code is of use. I'll try to fix up the CI errors shortly; battling with Covid aftereffects, so if somebody else wants to fix up this PR, feel free.

You need to return any values explicitly, e.g. by setting an object attribute to it or queuing the value or whatever.

peterhinch · 2022-06-26T10:19:48Z

Sorry you're unwell - I hope you feel better soon.

Writing tests, found a significant bug (related to cancellation of course). Investigating.

Is this still outstanding or is the code ready for review?

I very much hope this is implemented: in many applications it's a big improvement over gather.

m-u-xyz · 2022-06-26T11:16:04Z

Is this still outstanding or is the code ready for review?

No, that's done. The workaround I implemented admittedly isn't particularly clean, but it gets the job done and doesn't alter non-taskgroup code (as that might introduce bugs or slowdowns).

peterhinch · 2022-06-27T07:55:00Z

@dpgeorge @jimmo This is to advocate for TaskGroup with a real world example where they fit the bill perfectly.

Consider a communication link between two peers using an unreliable medium such as WiFi. This may be near the limit of range and the link can fail in a variety of ways. The simplest way to recover is the "belt and braces" approach: if a peer encounters an unrecoverable error, it takes the link down for a period long enough for the other peer also to suffer an unrecoverable error. That way both peers start from a "power up" state.

A TaskGroup might then comprise:

Task trying to keep the WiFi available through brief outages.
Task waiting on received messages.
Task pinging the other peer and checking a response.
Intermittently a task is added to the group. It sends a message, awaits an ack, and terminates.

If any of these experience an error that cannot be handled locally, the exception is passed up to be handled by the task that created the group. When this occurs, every task in the group terminates in an orderly way, running cleanup code in finally clauses or via context managers.

The TaskGroup is ideal for this - and in my testing works fine :) I rest my case...

m-u-xyz · 2022-06-27T13:21:14Z

@peterhinch Exactly. Unstructured tasks (like asyncio's baseline tasks) mean that you have to keep track of what's still running and what might have to be cancelled and/or restarted on your own. Code doing that tends to contain a heap of bugs you can't test for, and basically doesn't scale.

In contrast, when a taskgroup's async context manager has ended you know that there's no dangling tasks, unprocessed finally clauses, unreported exceptions, or similar nonsense around; you can proceed with confidence. When (not if) the framework ensures that whole classes of errors can't happen, you no longer have to test for them, or even think about them.

The latter point is much more powerful than you'd assume at first glance.

This patch adds a mostly-compatible backport of CPython 3.11's `TaskGroup` class to uasyncio. Also, there is a new `run_server` method in uasyncio/stream.py which supports task groups and doesn't run in the background (no need). TODO: Write some more tests. Closes micropython#8508. Signed-off-by: Matthias Urlichs <matthias@urlichs.de>

The C implementation (_uasyncio.Task) was not. Signed-off-by: Matthias Urlichs <matthias@urlichs.de>

Tasks can be cancelled before they have an opportunity to run. Handling this situation is non-trivial in MicroPython's context. Signed-off-by: Matthias Urlichs <matthias@urlichs.de>

If two tasks try to start sleeping at the same time, the SingletonGenerator will be in use, thus we need to allocate a new temporary handler on the heap. As the singleton isn't that single any more, it gets renamed. Found by taskgroup test micropython#12. Signed-off-by: Matthias Urlichs <matthias@urlichs.de>

This fixes some edge cases with cancellation. Signed-off-by: Matthias Urlichs <matthias@urlichs.de>

still failing, though Signed-off-by: Matthias Urlichs <matthias@urlichs.de>

Rudimentary but works. Signed-off-by: Matthias Urlichs <matthias@urlichs.de>

If there's an exception (or more) *and* a cancellation in a taskgroup, the cancellation must be ignored. Signed-off-by: Matthias Urlichs <matthias@urlichs.de>

This mirrors Python change 594c369. Signed-off-by: Matthias Urlichs <matthias@urlichs.de>

Cancelling a taskgroup that's no longer active should not do anything. Signed-off-by: Matthias Urlichs <matthias@urlichs.de>

Signed-off-by: Matthias Urlichs <matthias@urlichs.de>

`run_server` is not in CPython. We like to avoid that. Signed-off-by: Matthias Urlichs <matthias@urlichs.de>

Signed-off-by: Matthias Urlichs <matthias@urlichs.de>

dpgeorge · 2026-03-22T11:55:28Z

I'm sorry but this keeps getting bumped to the next release. I need to set aside some quality time to understand task groups and review this PR.

m-u-xyz · 2026-04-25T07:59:04Z

A patch that adds taskgroup.cancel(), with basically the same semantics as this PR's, has been merged into CPython.

projectgus

Hi @smurfix,

Sorry I'm coming in very late to this! Thanks for keeping the PR open and up to date for so long.

I have some comments but they're mostly docs and CPython differences. The implementation looks very clean and well commented, and the test coverage is comprehensive which is great.

Apologies if anything I've commented on here has previously discussed, I skimmed the comment history but I haven't read every comment - sorry.

Certainly I'm not asking for any major additional work, don't want perfect to become the enemy of good here as this looks really useful to have in MicroPython! Are you still interested in working on it?

projectgus · 2026-05-06T00:35:55Z


+class TaskGroup
+---------------
+


At the moment the link at the top of this page is to CPython 3.8's asyncio reference, which doesn't have TaskGroups. Suggest starting this section with a short explanation of this being a port of the CPython 3.11 feature, and a link to https://docs.python.org/3.11/library/asyncio-task.html#task-groups

projectgus · 2026-05-06T00:38:21Z

+See Nathaniel J. Smith's `essay on Structured Concurrency
+<https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/>`_
+for an introduction why you should use taskgroups instead of starting
+"naked" tasks.
+
+.. note::
+    His "nursery" objects are called "taskgroup" in asyncio; the
+    equivalent of a "go statement" is `Loop.create_task`.


This is interesting context and motivation, but I'm a bit torn on whether this is suitable for the MicroPython asyncio reference or not. (i.e. the equivalent CPython docs provide a single sentence summary of what TaskGroups do and that feels more appropriate for reference documentation.)

projectgus · 2026-05-06T00:41:10Z

+    This object is an async context managed holding a group of tasks.
+    Tasks can be added to the group using `TaskGroup.create_task`.


Suggested change

This object is an async context managed holding a group of tasks.

Tasks can be added to the group using `TaskGroup.create_task`.

An asynchronous context manager holding a group of tasks. Tasks can be

added to the group using `TaskGroup.create_task`. All tasks are awaited when

the context manager exits.

(Originally this was just a suggestion to fix the typo, but changed it to copying the CPython doc summary exactly!)

projectgus · 2026-05-06T00:57:08Z


    This is a coroutine.

+.. function:: run_server(callback, host, port, backlog=5, taskgroup=None)


I have two concerns here:

This function doesn't exist in CPython 3.11 asyncio, and we're trying not to add any more MicroPython-only names in CPython-compatible modules..

This function doesn't appear to exist in MicroPython asyncio module either, at least in this version of the PR it's in the examples/run_server.py file. Ergonomically this is a bit rough for someone who wants to use it.

Is the comment above about how start_server doesn't work well with TaskGroups exclusive to MicroPython? If so, is there any way that we can change MicroPython's implementation of start_server to be closer to CPython's in this regard?

(Otherwise, if start_server also doesn't work well with TaskGroups on CPython then a good path forward might be to add run_server as a micropython-lib library that can be separately mip install-ed, and mention it in the docs for start_server here.)

projectgus · 2026-05-06T00:59:52Z


    Create a task from the given *coro* and return the new `Task` object.

+    You should not call this function when you're using taskgroups.


Suggested change

You should not call this function when you're using taskgroups.

.. warning: Due to MicroPython internal limitations, this function can't be used at the same time as TaskGroups.

projectgus · 2026-05-06T01:32:11Z

+from .taskgroup import TaskGroup

-__version__ = (3, 0, 0)
+__version__ = (3, 0, 1)


Suggested change

__version__ = (3, 0, 1)

__version__ = (3, 1, 0)

Nitpicky, but adding a whole new feature is at least worth a minor version bump!

projectgus · 2026-05-06T01:32:54Z

@@ -0,0 +1,841 @@
+# Test Lock class


Suggested change

# Test Lock class

# Test TaskGroups

projectgus · 2026-05-06T01:39:51Z

+    If multiple subtasks raise exceptions in parallel, it's unclear which
+    of them should be propagated. Thus an `ExceptionGroup` exception
+    collects them and is raised instead.
+


Suggested change

.. note:: In CPython ExceptionGroup is a built-in exception, but in MicroPython it is

part of the asyncio module.

MicroPython does not support CPython 3.11's ``except*`` syntax for handling

exception groups automatically.

projectgus · 2026-05-06T01:40:21Z

+    MicroPython does not support CPython 3.11's syntax for filtering handling
+    exception groups.
+


Suggested change

MicroPython does not support CPython 3.11's syntax for filtering handling

exception groups.

Suggest removing this from here and covering it in the class documentation above.

projectgus · 2026-05-06T01:43:57Z

+    if sys.implementation.name != "micropython":
+        print("Not handled: divide by zero")


Looks like some of these tests can run on CPython (as the top of the file checks if ExceptionGroup is a built-in), but not all of them (hence the .exp file and these checks)?

Not asking you to necessarily change anything, but how much work would it be to have tests which run on CPython & MicroPython instead?

m-u-xyz force-pushed the tg branch 2 times, most recently from 5a46a80 to b5723ea Compare June 20, 2022 20:08

dpgeorge added the extmod Relates to extmod/ directory in source label Jun 21, 2022

dpgeorge reviewed Jun 21, 2022

View reviewed changes

Comment thread extmod/uasyncio/taskgroup.py Outdated

m-u-xyz force-pushed the tg branch from b5723ea to ed736ed Compare June 21, 2022 05:00

m-u-xyz force-pushed the tg branch 8 times, most recently from 1a0dffd to 7d20532 Compare June 22, 2022 19:29

peterhinch mentioned this pull request Jul 4, 2022

Disconnect stability improvements peterhinch/micropython-mqtt#77

Merged

m-u-xyz force-pushed the tg branch from 7d20532 to fddbe65 Compare February 13, 2023 17:37

dpgeorge modified the milestones: release-1.27.0, release-1.28.0 Dec 1, 2025

m-u-xyz added 18 commits December 12, 2025 13:25

extmod/uasyncio: Task needs to be hashable.

17e7a71

The C implementation (_uasyncio.Task) was not. Signed-off-by: Matthias Urlichs <matthias@urlichs.de>

extmod/uasyncio/taskgroup: Rewrite state machine.

450e79d

Tasks can be cancelled before they have an opportunity to run. Handling this situation is non-trivial in MicroPython's context. Signed-off-by: Matthias Urlichs <matthias@urlichs.de>

extmod/asyncio/taskgroup.py: State machine update.

d068fd1

This fixes some edge cases with cancellation. Signed-off-by: Matthias Urlichs <matthias@urlichs.de>

tests/extmod/asyncio_taskgroup: Improve test output.

e754c6e

still failing, though Signed-off-by: Matthias Urlichs <matthias@urlichs.de>

extmod/asyncio/core: Implement BaseExceptionGroup.split().

dbe0f85

Rudimentary but works. Signed-off-by: Matthias Urlichs <matthias@urlichs.de>

extmod/asyncio/taskgroup: Colliding cancel vs. exception.

b925b89

If there's an exception (or more) *and* a cancellation in a taskgroup, the cancellation must be ignored. Signed-off-by: Matthias Urlichs <matthias@urlichs.de>

extmod/asyncio/taskgroup: Exiting vs. aborting taskgroups.

ce9e5a9

This mirrors Python change 594c369. Signed-off-by: Matthias Urlichs <matthias@urlichs.de>

extmod/asyncio/taskgroup: NO-OP cancelling an inactive taskgroup.

7ac1801

Cancelling a taskgroup that's no longer active should not do anything. Signed-off-by: Matthias Urlichs <matthias@urlichs.de>

docs/library/asyncio: Add a section on taskgroups.

4a46e73

Signed-off-by: Matthias Urlichs <matthias@urlichs.de>

extmod/asyncio/taskgroup: do the more common test first

6873116

Signed-off-by: Matthias Urlichs <matthias@urlichs.de>

extmod/asyncio/taskgroup: handle self-cancellation

13b463b

Signed-off-by: Matthias Urlichs <matthias@urlichs.de>

extmod/asyncio/taskgroup: print errors to stderr

5327d8d

Signed-off-by: Matthias Urlichs <matthias@urlichs.de>

docs/asyncio: Fix capitalization error.

8c22f91

Signed-off-by: Matthias Urlichs <matthias@urlichs.de>

asyncio: move run_server to examples.

6eb3254

`run_server` is not in CPython. We like to avoid that. Signed-off-by: Matthias Urlichs <matthias@urlichs.de>

webassembly/asyncio: Add support for taskgroups.

fc950a2

Signed-off-by: Matthias Urlichs <matthias@urlichs.de>

ports/webassembly: Add exception groups to its asyncio module.

3bca4bd

Signed-off-by: Matthias Urlichs <matthias@urlichs.de>

m-u-xyz force-pushed the tg branch from 32608f0 to 3bca4bd Compare December 12, 2025 16:03

m-u-xyz mentioned this pull request Dec 12, 2025

Crash/segfault in async code #18559

Open

projectgus self-requested a review January 28, 2026 23:44

dpgeorge modified the milestones: release-1.28.0, release-1.29.0 Mar 22, 2026

projectgus requested changes May 6, 2026

View reviewed changes

dpgeorge modified the milestones: release-1.29.0, release-1.30 Aug 13, 2026

		This object is an async context managed holding a group of tasks.
		Tasks can be added to the group using `TaskGroup.create_task`.

-    This object is an async context managed holding a group of tasks.
-    Tasks can be added to the group using `TaskGroup.create_task`.
+    An asynchronous context manager holding a group of tasks. Tasks can be
+    added to the group using `TaskGroup.create_task`. All tasks are awaited when
+    the context manager exits.


		This is a coroutine.

		.. function:: run_server(callback, host, port, backlog=5, taskgroup=None)


		Create a task from the given coro and return the new `Task` object.

		You should not call this function when you're using taskgroups.

	You should not call this function when you're using taskgroups.
	.. warning: Due to MicroPython internal limitations, this function can't be used at the same time as TaskGroups.

+    .. note:: In CPython ExceptionGroup is a built-in exception, but in MicroPython it is
+       part of the asyncio module.
+       MicroPython does not support CPython 3.11's ``except*`` syntax for handling
+       exception groups automatically.

		MicroPython does not support CPython 3.11's syntax for filtering handling
		exception groups.

		if sys.implementation.name != "micropython":
		print("Not handled: divide by zero")

Sunbelt Computer Software

PL/B Language Development and Support

Uh oh!

Uh oh!

Conversation

m-u-xyz commented Jun 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

m-u-xyz commented Jun 20, 2022

Uh oh!

dpgeorge commented Jun 21, 2022

Uh oh!

dpgeorge commented Jun 21, 2022

Uh oh!

Uh oh!

dpgeorge commented Jun 21, 2022

Uh oh!

m-u-xyz commented Jun 21, 2022

Uh oh!

m-u-xyz commented Jun 22, 2022

Uh oh!

dlech commented Jun 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

m-u-xyz commented Jun 22, 2022

Uh oh!

dlech commented Jun 23, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

peterhinch commented Jun 25, 2022

Uh oh!

m-u-xyz commented Jun 25, 2022

Uh oh!

peterhinch commented Jun 26, 2022

Uh oh!

m-u-xyz commented Jun 26, 2022

Uh oh!

peterhinch commented Jun 27, 2022

Uh oh!

m-u-xyz commented Jun 27, 2022

Uh oh!

dpgeorge commented Mar 22, 2026

Uh oh!

m-u-xyz commented Apr 25, 2026

Uh oh!

projectgus left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

projectgus May 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

m-u-xyz commented Jun 20, 2022 •

edited

Loading

dlech commented Jun 22, 2022 •

edited

Loading

dlech commented Jun 23, 2022 •

edited

Loading

projectgus May 6, 2026 •

edited

Loading