Polars use nest_asyncio

[douglas-raillard-arm]

Checks

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of Polars.

Reproducible example

No polars example at hand since I only realized polars used nest_asyncio from the changelog. The general problem looks like that:

import uvloop
import nest_asyncio

async def main():
    print('hello world')
    nest_asyncio.apply()

uvloop.run(main())

Log output

Traceback (most recent call last):
  File "/work/projects/lisa/testuvloop.py", line 8, in <module>
    uvloop.run(main())
  File "venv/lib/python3.12/site-packages/uvloop/__init__.py", line 109, in run
    return __asyncio.run(
           ^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "uvloop/loop.pyx", line 1517, in uvloop.loop.Loop.run_until_complete
  File "venv/lib/python3.12/site-packages/uvloop/__init__.py", line 61, in wrapper
    return await main
           ^^^^^^^^^^
  File "/work/projects/lisa/testuvloop.py", line 6, in main
    nest_asyncio.apply()
  File "venv/lib/python3.12/site-packages/nest_asyncio.py", line 19, in apply
    _patch_loop(loop)
  File "venv/lib/python3.12/site-packages/nest_asyncio.py", line 193, in _patch_loop
    raise ValueError('Can\'t patch loop of type %s' % type(loop))
ValueError: Can't patch loop of type <class 'uvloop.Loop'>

Issue description

Polars currently relies on nest_asyncio:

polars/py-polars/polars/io/database/_utils.py

Line 39 in f73937a

nest_asyncio.apply()

I would suggest to avoid doing so, as nest_asyncio only works with event loop implementations shipped with the standard library (as per its documentation). On top of that, we observed that failed nest_asyncio.apply() calls were leaving behind some broken state that would force to restart the process [1].

The library is currently not maintained anymore (github repo is archived and the original author and maintainer has sadly left us), so there is little hope of these issues fixed shortly.

We faced the aforementioned issues when running under uvloop (FastAPI server with uvicorn), which lead us to implement an equivalent based on greenlets (see devlib.utils.asyn.run() for an equivalent of asyncio.run() that nests). greenback does more-or-less the same thing but seemed to be using unsafe code (access of stdlib C structs using ctypes from pure-Python) [2].

[1] I can't remember the details, something related to the event loop policy staying patched to modify event loops to use nest_asyncio's equivalent to the original asyncio.run())

[2] AFAIR greenback uses ctypes to patch the current running asyncio.Task in await greenback.ensure_portal(), which we found unnecessary in devlib. Instead we use loop.set_task_factory to interpose our coroutine shim in the top-level . If we happen to run in an non-instrumented asyncio.Task, we just delegate the call to another thread with another event loop as a fallback.

Expected behavior

Polars should work with any asyncio event loop implementation.

Installed versions

N/A