gh-66435: Allow nested event loops

[davidbrochart]

This PR is basically a copy/paste of @erdewit's nest_asyncio library. It allows the event loop to be re-entrant, by making it possible to call asyncio.run() inside an already running event loop.
This feature request was rejected in the past, but @1st1 recently mentioned that he could reconsider this decision.
Here is an example:

import asyncio

async def task(name):
    for i in range(10):
        await asyncio.sleep(0.1)
        print(f"from {name}: {i}")

async def bar():
    asyncio.create_task(task("bar"))
    await asyncio.sleep(1.1)
    print("bar done")

def foo():
    # asyncio.run inside an already running event loop
    # pre-empts the execution of any other task in the event loop
    asyncio.run(bar())

async def main():
    t = asyncio.create_task(task("main"))  # not executed until foo() is done
    foo()
    await asyncio.sleep(1.1)  # t resumes execution

asyncio.run(main())

This will print:

from bar: 0
from bar: 1
from bar: 2
from bar: 3
from bar: 4
from bar: 5
from bar: 6
from bar: 7
from bar: 8
from bar: 9
bar done
from main: 0
from main: 1
from main: 2
from main: 3
from main: 4
from main: 5
from main: 6
from main: 7
from main: 8
from main: 9

• Issue: asyncio: nested event loop #66435

[cpython-cla-bot]

All commit authors signed the Contributor License Agreement.

[bedevere-bot]

Most changes to Python require a NEWS entry.

Please add it using the blurb_it web app or the blurb command-line tool.

[AA-Turner]

Please update the title to gh-66435: Allow nested event loops (although closed, it is the most appropriate issue to link against). You also need a NEWS entry.

A

[maartenbreddels]

This is different from nest_asyncio right?

This PR only allows 'nested event' loops, which I think is really great (I vaex we do this using this decorator https://github.com/vaexio/vaex/blob/633970528cb5091ef376dbca2e4721cd42525419/packages/vaex-core/vaex/asyncio.py#L7 but it's using the asyncio.events._set_running_loop private API).

'nest_asyncio' will also run the tasks of the parent/previous event loop, i.e. it will give an output like this:

from main: 0
from bar: 0
from main: 1
from bar: 1
from main: 2
from bar: 2
from main: 3
from bar: 3
from main: 4
from bar: 4
from main: 5
from bar: 5
from main: 6
from bar: 6
from main: 7
from bar: 7
from main: 8
from bar: 8
from main: 9
from bar: 9
bar done

Which basically leads to a sync_call_that_can_resume_the_async_task_I_created_on_the_previous_line() which can be very dangerous (although sometimes very useful).

I think with this PR it is not possible to resume a task from the previous event_loop, is that right?

To make it more concrete, this example works with nest_asyncio:

 import asyncio
 
+import nest_asyncio
+
+nest_asyncio.apply()
+
+t = None
+
 
 async def task(name):
     for i in range(10):
@@ -8,6 +14,7 @@
 
 
 async def bar():
+    await t
     asyncio.create_task(task("bar"))
     await asyncio.sleep(1.1)
     print("bar done")
@@ -20,6 +27,7 @@
 
 
 async def main():
+    global t
     t = asyncio.create_task(task("main"))  # not executed until foo() is done
     foo()
     await asyncio.sleep(1.1)  # t resumes execution

output:

from main: 0
from main: 1
from main: 2
from main: 3
from main: 4
from main: 5
from main: 6
from main: 7
from main: 8
from main: 9
from bar: 0
from bar: 1
from bar: 2
from bar: 3
from bar: 4
from bar: 5
from bar: 6
from bar: 7
from bar: 8
from bar: 9
bar done

Does this PR give an exception, does it run it, or does it deadlock? I think the appropriate behavior should be an exception (complaining the awaited task t was not created with the running event loop).

[davidbrochart]

@maartenbreddels indeed, I didn't copy exactly the code from nest_asyncio and I must have forgotten something. I'll look at it again, but anyway as @erdewit mentioned, this code would probably need to be written by core developers of CPython, and this PR is just a POC.
It raises this question though: do we want asyncio.run() to block the execution of other tasks, as it is the case at the current stage of this PR, or do we want them to also execute concurrently, as it is the case in nest_asyncio? In the former case, maybe your solution in Vaex is enough. Correct me if I'm wrong, but it's "just" allowing to run another event loop while already in a running event loop, right?

[maartenbreddels]

do we want asyncio.run() to block the execution of other tasks

If we don't, then it is not cooperative multitasking anymore. But I can't find in the Python docs that asyncio claims it is actually. In any case, it will probably break existing code.

[davidbrochart]

then it is not cooperative multitasking anymore.

I think it would be less dangerous, as you mentioned, that asyncio.run() always blocks any other task. Otherwise, having other tasks running while it doesn't have await would really be confusing and break the most basic rule of asyncio.
I think it's still different than what you do in Vaex, since it allows to re-use the same event loop, while in Vaex you run in a new event loop. In jupyter_client for instance, we need to re-use the same event loop as pyzmq.

In any case, it will probably break existing code.

Yes, that's why we would need to opt-in for this behavior.

[graingert]

@davidbrochart would it be possible for jupyter_client to spawn a python subprocess for running user code in instead?

[davidbrochart]

A Jupyter kernel already runs in its own process, and jupyter-client communicates with it through ZMQ sockets.
When we added the async API to jupyter-client, we also tried implementing the blocking API by running the async functions in a separate thread with its own event loop (similarly to what anyio's to_thread.run_sync does), but we ran into issues related to pyzmq's async context running in a different event loop.
I still think allowing nested event loops would be useful (as an opt-in behavior), but I'd like to know if this PR could get a chance to get in before investing more time in it.

[davidbrochart]

I've simplified the changes to what I think is the minimum to get the behavior described in the top comment. Now the nested call to asyncio.run should explicitly pass running_ok=True in order to re-enter the event loop.

asyncio.run(bar(), running_ok=True)

[willingc]

@davidbrochart @maartenbreddels If you have time to update this PR, we can discuss it at JupyterCon.

[davidbrochart]

Thanks for the heads up @willingc, I rebased on main.
Here is an example usage:

import asyncio

async def task(name):
    for i in range(10):
        await asyncio.sleep(0.1)
        print(f"from {name}: {i}")

async def bar():
    asyncio.create_task(task("bar"))
    await asyncio.sleep(1.1)
    print("bar done")

def foo():
    # asyncio.run inside an already running event loop
    # pre-empts the execution of any other task in the event loop
    asyncio.run(bar(), running_ok=True)

async def main():
    t = asyncio.create_task(task("main"))  # not executed until foo() is done
    foo()
    await asyncio.sleep(1.1)  # t resumes execution

asyncio.run(main())

That will print:

from bar: 0
from bar: 1
from bar: 2
from bar: 3
from bar: 4
from bar: 5
from bar: 6
from bar: 7
from bar: 8
from bar: 9
bar done
from main: 0
from main: 1
from main: 2
from main: 3
from main: 4
from main: 5
from main: 6
from main: 7
from main: 8
from main: 9

If we wanted the nested call to asyncio.run not block the top-level one, i.e. have:

from main: 0
from bar: 0
from main: 1
from bar: 1
from main: 2
from bar: 2
from main: 3
from bar: 3
from main: 4
from bar: 4
from main: 5
from bar: 5
from main: 6
from bar: 6
from main: 7
from bar: 7
from main: 8
from bar: 8
from main: 9
from bar: 9
bar done

We would not really need this PR, we would just check if a loop is already running, and if yes then call asyncio.create_task, otherwise call asyncio.run.

[arhadthedev]

@davidbrochart Could you sign the CLA, please? #93338 (comment)

[davidbrochart]

@davidbrochart Could you sign the CLA, please? #93338 (comment)

Done.

[gvanrossum]

Whoa. A process issue: discussions about whether a particular feature should be accepted should happen on an Issue, not in a PR. The best thing to do is to reopen gh-66435, which I will do. See you there! (I do appreciate the discussion and the clear examples of two different proposed behaviors.)

[willingc]

@davidbrochart I'm closing this PR since we decided to not add nested asyncio to the library. #66435 (comment) Thanks for your effort here and your suggestion on the issue for a third party library.