fix(python): Replacing nest-asyncio with greenlet in database dependencies

[phi-friday]

The nest-asyncio has been deprecated.
I replaced it with using threads instead, and in the case of sqlalchemy, using greenlet.

I specified greenlet as an optional dependency,
but it is also an implicit dependency of sqlalchemy when using asynchronous sessions.

[alexander-beedie]

Looks like very nice work! I was not looking forward to updating this 😄

Should close #17334? Can you confirm that it plays nice with Notebooks, and inside uvloop? (I'll start checking the PR in more detail later today or tomorrow, but if you've already taken a look at these cases -or could try them out?- that would be great).

[codecov]

Codecov Report

Attention: Patch coverage is 82.50000% with 7 lines in your changes missing coverage. Please review.

Project coverage is 80.37%. Comparing base (ca6d46c) to head (35113ae).
Report is 1 commits behind head on main.

Files	Patch %	Lines
py-polars/polars/io/database/_utils.py	81.57%	5 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #17811      +/-   ##
==========================================
- Coverage   80.39%   80.37%   -0.02%     
==========================================
  Files        1496     1496              
  Lines      197480   197507      +27     
  Branches     2820     2822       +2     
==========================================
- Hits       158764   158750      -14     
- Misses      38194    38234      +40     
- Partials      522      523       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[phi-friday]

Looks like very nice work! I was not looking forward to updating this 😄

Should close #17334? Can you confirm that it plays nice with Notebooks, and inside uvloop? (I'll start checking the PR in more detail later today or tomorrow, but if you've already taken a look at these cases -or could try them out?- that would be great).

it works in local with uvloop

from __future__ import annotations

import sqlite3
import tempfile
from datetime import date
from pathlib import Path
from typing import Any

import uvloop
from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine

import polars as pl
from polars.testing import assert_frame_equal


def tmp_sqlite_db(test_db: Path) -> Path:
    def convert_date(val: bytes) -> date:
        return date.fromisoformat(val.decode())

    sqlite3.register_converter("date", convert_date)
    conn = sqlite3.connect(test_db)

    conn.executescript(
        """
        CREATE TABLE IF NOT EXISTS test_data (
            id    INTEGER PRIMARY KEY,
            name  TEXT NOT NULL,
            value FLOAT,
            date  DATE
        );
        REPLACE INTO test_data(name,value,date)
          VALUES ('misc',100.0,'2020-01-01'),
                 ('other',-99.5,'2021-12-31');
        """
    )
    conn.close()
    return test_db


def test_read_async(tmp_sqlite_db: Path) -> None:
    async_engine = create_async_engine(f"sqlite+aiosqlite:///{tmp_sqlite_db}")
    async_connection = async_engine.connect()
    async_session = async_sessionmaker(async_engine)
    async_session_inst = async_session()

    expected_frame = pl.DataFrame(
        {"id": [2, 1], "name": ["other", "misc"], "value": [-99.5, 100.0]}
    )
    async_conn: Any
    for async_conn in (
        async_engine,
        async_connection,
        async_session,
        async_session_inst,
    ):
        if async_conn in (async_session, async_session_inst):
            constraint, execute_opts = "", {}
        else:
            constraint = "WHERE value > :n"
            execute_opts = {"parameters": {"n": -1000}}

        df = pl.read_database(
            query=f"""
                SELECT id, name, value
                FROM test_data {constraint}
                ORDER BY id DESC
            """,
            connection=async_conn,
            execute_options=execute_opts,
        )
        assert_frame_equal(expected_frame, df)


async def main() -> None:
    with tempfile.TemporaryDirectory() as temp_dir:
        sqlite_db = Path(temp_dir) / "test.db"
        tmp_sqlite_db(sqlite_db)
        test_read_async(sqlite_db)

    print("done")


if __name__ == "__main__":
    uvloop.run(main())

output:

❯ rye run python ./test_uvloop.py
done

[phi-friday]

also, in notebook, it works.

[alexander-beedie]

Nice! I'll experiment using some other non-alchemy async drivers shortly (such as SurrealDB), but if it's working everywhere else than I'm optimistic ;)

[ritchie46]

@alexander-beedie can you take a look at this one?

[alexander-beedie]

@alexander-beedie can you take a look at this one?

Yup; have just been trying to create a stripped-down test case of an example I've found where the new code hangs indefinitely 🤔