Using task_tags to report an error

[WTSRUVF]

Bug summary

When I used task_tags to limit the total concurrency, an error was thrown.

code

    @task(tags=["small_instance"])
    def run(self, data):

config

21:26:52.218 | ERROR   | uvicorn.error - Exception in ASGI application
Traceback (most recent call last):
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
    self.dialect.do_execute(
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 941, in do_execute
    cursor.execute(statement, parameters)
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/sqlalchemy/dialects/sqlite/aiosqlite.py", line 147, in execute
    self._adapt_connection._handle_exception(error)
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/sqlalchemy/dialects/sqlite/aiosqlite.py", line 298, in _handle_exception
    raise error
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/sqlalchemy/dialects/sqlite/aiosqlite.py", line 129, in execute
    self.await_(_cursor.execute(operation, parameters))
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 132, in await_only
    return current.parent.switch(awaitable)  # type: ignore[no-any-return,attr-defined] # noqa: E501
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 196, in greenlet_spawn
    value = await result
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/aiosqlite/cursor.py", line 48, in execute
    await self._execute(self._cursor.execute, sql, parameters)
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/aiosqlite/cursor.py", line 40, in _execute
    return await self._conn._execute(fn, *args, **kwargs)
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/aiosqlite/core.py", line 132, in _execute
    return await future
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/aiosqlite/core.py", line 115, in run
    result = function()
sqlite3.OperationalError: database is locked

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    return await self.app(scope, receive, send)
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/prefect/server/api/server.py", line 150, in __call__
    await self.app(scope, receive, send)
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/starlette/routing.py", line 460, in handle
    await self.app(scope, receive, send)
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/starlette/middleware/gzip.py", line 20, in __call__
    await responder(scope, receive, send)
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/starlette/middleware/gzip.py", line 39, in __call__
    await self.app(scope, receive, self.send_with_gzip)
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/starlette/routing.py", line 73, in app
    response = await f(request)
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/prefect/server/utilities/server.py", line 47, in handle_response_scoped_depends
    response = await default_handler(request)
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/fastapi/routing.py", line 212, in run_endpoint_function
    return await dependant.call(**values)
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/prefect/server/api/concurrency_limits.py", line 277, in decrement_concurrency_limits_v1
    cl.active_slots = list(active_slots)
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/contextlib.py", line 188, in __aexit__
    await self.gen.__anext__()
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/prefect/server/database/interface.py", line 136, in session_context
    yield session
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/contextlib.py", line 188, in __aexit__
    await self.gen.__anext__()
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/prefect/server/database/configurations.py", line 471, in begin_transaction
    yield transaction
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/sqlalchemy/ext/asyncio/session.py", line 1876, in __aexit__
    await greenlet_spawn(
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 203, in greenlet_spawn
    result = context.switch(value)
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/sqlalchemy/engine/util.py", line 149, in __exit__
    self.rollback()
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/sqlalchemy/engine/util.py", line 145, in __exit__
    self.commit()
  File "<string>", line 2, in commit
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
    ret_value = fn(self, *arg, **kw)
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1313, in commit
    self._prepare_impl()
  File "<string>", line 2, in _prepare_impl
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
    ret_value = fn(self, *arg, **kw)
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1288, in _prepare_impl
    self.session.flush()
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 4352, in flush
    self._flush(objects)
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 4488, in _flush
    transaction.rollback(_capture_exception=True)
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 4448, in _flush
    flush_context.execute()
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 466, in execute
    rec.execute(self)
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 642, in execute
    util.preloaded.orm_persistence.save_obj(
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 85, in save_obj
    _emit_update_statements(
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 885, in _emit_update_statements
    c = connection.execute(
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1418, in execute
    return meth(
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 515, in _execute_on_connection
    return connection._execute_clauseelement(
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1640, in _execute_clauseelement
    ret = self._execute_context(
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context
    return self._exec_single_context(
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1986, in _exec_single_context
    self._handle_dbapi_exception(
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2355, in _handle_dbapi_exception
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
    self.dialect.do_execute(
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 941, in do_execute
    cursor.execute(statement, parameters)
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/sqlalchemy/dialects/sqlite/aiosqlite.py", line 147, in execute
    self._adapt_connection._handle_exception(error)
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/sqlalchemy/dialects/sqlite/aiosqlite.py", line 298, in _handle_exception
    raise error
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/sqlalchemy/dialects/sqlite/aiosqlite.py", line 129, in execute
    self.await_(_cursor.execute(operation, parameters))
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 132, in await_only
    return current.parent.switch(awaitable)  # type: ignore[no-any-return,attr-defined] # noqa: E501
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 196, in greenlet_spawn
    value = await result
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/aiosqlite/cursor.py", line 48, in execute
    await self._execute(self._cursor.execute, sql, parameters)
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/aiosqlite/cursor.py", line 40, in _execute
    return await self._conn._execute(fn, *args, **kwargs)
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/aiosqlite/core.py", line 132, in _execute
    return await future
  File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/aiosqlite/core.py", line 115, in run
    result = function()
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked
[SQL: UPDATE concurrency_limit SET active_slots=:active_slots, updated=strftime('%Y-%m-%d %H:%M:%f000', 'now') WHERE concurrency_limit.id = :concurrency_limit_id RETURNING updated]
[parameters: {'active_slots': '["c154aa6c-4a8d-4a49-9fdf-ee29c9ac3619", "c2505180-27e0-4bda-841e-db08c405d46f", "cb7824fd-6083-4329-9bd8-e6308f2f3f80", "fcb096fc-bf1b-40a7-b44c-3d17c41aae6e"]', 'concurrency_limit_id': '49f3a22b-b7e3-4a65-8c02-ebb13c4a2818'}]
(Background on this error at: https://sqlalche.me/e/20/e3q8)

Version info

Version:             3.1.10
API version:         0.8.4
Python version:      3.9.16
Git commit:          b11b56fc
Built:               Tue, Dec 24, 2024 10:58 PM
OS/Arch:             linux/x86_64
Profile:             local
Server type:         ephemeral
Pydantic version:    2.10.4
Server:
  Database:          sqlite
  SQLite version:    3.45.1
Integrations:
  prefect-dask:      0.3.2

Additional context

Package                   Version
------------------------- -----------
aiosqlite                 0.20.0
alembic                   1.14.0
amqp                      5.3.1
annotated-types           0.7.0
anyio                     4.7.0
apprise                   1.9.1
APScheduler               3.11.0
asgi-lifespan             2.1.0
async-timeout             5.0.1
asyncpg                   0.30.0
attrs                     24.3.0
bce-python-sdk            0.9.25
beautifulsoup4            4.12.3
billiard                  4.2.1
blinker                   1.9.0
bs4                       0.0.2
cachetools                5.5.0
celery                    5.4.0
certifi                   2024.12.14
cffi                      1.17.1
charset-normalizer        3.4.1
click                     8.1.8
click-didyoumean          0.3.1
click-plugins             1.1.1
click-repl                0.3.0
cloudpickle               3.1.0
colorama                  0.4.6
coolname                  2.2.0
croniter                  3.0.4
cryptography              44.0.0
dask                      2024.8.0
dateparser                1.2.0
Deprecated                1.2.15
distributed               2024.8.0
docker                    7.1.0
exceptiongroup            1.2.2
fastapi                   0.115.6
Flask                     3.1.0
Flask-SQLAlchemy          3.1.1
fsspec                    2024.12.0
future                    1.0.0
graphviz                  0.20.3
greenlet                  3.1.1
griffe                    1.5.1
h11                       0.14.0
h2                        4.1.0
hpack                     4.0.0
httpcore                  1.0.7
httpx                     0.28.1
humanize                  4.11.0
hyperframe                6.0.1
idna                      3.10
importlib_metadata        8.5.0
itsdangerous              2.2.0
jieba                     0.42.1
Jinja2                    3.1.5
jinja2-humanize-extension 0.4.0
joblib                    1.4.2
jsonpatch                 1.33
jsonpointer               3.0.0
jsonschema                4.23.0
jsonschema-specifications 2024.10.1
kombu                     5.4.2
Levenshtein               0.26.1
locket                    1.0.0
lxml                      5.2.2
Mako                      1.3.8
Markdown                  3.7
markdown-it-py            3.0.0
MarkupSafe                3.0.2
mdurl                     0.1.2
msgpack                   1.1.0
nltk                      3.9.1
numpy                     2.0.2
oauthlib                  3.2.2
opentelemetry-api         1.29.0
orjson                    3.10.12
packaging                 24.2
pandas                    2.2.3
partd                     1.4.2
pathspec                  0.12.1
pendulum                  3.0.0
pip                       22.0.4
prefect                   3.1.10
prefect-dask              0.3.2
prometheus_client         0.21.1
prompt_toolkit            3.0.48
psutil                    6.1.1
py4j                      0.10.9.7
pycparser                 2.22
pycrypto                  2.6.1
pycryptodome              3.21.0
pydantic                  2.10.4
pydantic_core             2.27.2
pydantic-extra-types      2.10.1
pydantic-settings         2.7.0
Pygments                  2.18.0
PyMySQL                   1.1.1
pyspark                   3.5.4
python-dateutil           2.9.0.post0
python-dotenv             1.0.1
python-Levenshtein        0.26.1
python-slugify            8.0.4
python-socks              2.5.3
pytz                      2024.2
PyYAML                    6.0.2
RapidFuzz                 3.11.0
readchar                  4.2.1
redis                     5.2.1
referencing               0.35.1
regex                     2024.11.6
requests                  2.32.3
requests-oauthlib         2.0.0
rfc3339-validator         0.1.4
rich                      13.9.4
rpds-py                   0.22.3
ruamel.yaml               0.18.6
ruamel.yaml.clib          0.2.12
setuptools                58.1.0
shellingham               1.5.4
six                       1.17.0
sniffio                   1.3.1
sortedcontainers          2.4.0
soupsieve                 2.6
SQLAlchemy                2.0.36
starlette                 0.41.3
tblib                     3.0.0
text-unidecode            1.3
time-machine              2.16.0
timeout-decorator         0.5.0
toml                      0.10.2
toolz                     1.0.0
tornado                   6.4.2
tqdm                      4.67.1
typer                     0.12.5
typing_extensions         4.12.2
tzdata                    2024.2
tzlocal                   5.2
ujson                     5.10.0
urllib3                   2.3.0
uvicorn                   0.34.0
vine                      5.1.0
wcwidth                   0.2.13
websockets                13.1
Werkzeug                  3.1.3
wrapt                     1.17.0
zhon                      2.1.1
zict                      3.0.0
zipp                      3.21.0

[zzstoatzz]

hi @WTSRUVF - thanks for the issue! I don't think this locking is caused by your use of concurrency tags here, ultimately I suspect this is the same sqlite locking problem as #10188 and #16299, so unless you're noticing the concurrency not actually working, I will close this in favor of those

[WTSRUVF]

hi @WTSRUVF - thanks for the issue! I don't think this locking is caused by your use of concurrency tags here, ultimately I suspect this is the same sqlite locking problem as #10188 and #16299, so unless you're noticing the concurrency not actually working, I will close this in favor of those

Thank you for your reply. As you said, this is not dangerous, just like the above question, but I still have two questions.

1. According to the reply in 10188, I executed export PREFECT_API_SERVICES_FLOW_RUN_NOTIFICATIONS_ENABLED=false in the shell, but when I executed prefect start, it told me that there was no start command. The log will still be reported when the program is executed again.
2. Is the log in this question also not dangerous? I didn't find any related information from other questions.
   Error message when program is closed：sqlalchemy.pool.impl.NullPool  #16504

[zzstoatzz]

@WTSRUVF

1. what were you trying when you did prefect start? to start a server you can run prefect server start or you can explore any level of the CLI with prefect --help or prefect server --help
2. the database is locked error reflects a performance issue specific to sqlite, which may or may not be related to the NullPool issue you linked. I think that issue is fine as a standalone issue for now, but I think more information about how that arose for you would be helpful

for now I think I'll close this issue and track the locking issues on the other two issues i mentioned earlier

please feel free to let me know if you think there's a standalone reason to keep this open!

[WTSRUVF]

@WTSRUVF

1. what were you trying when you did prefect start? to start a server you can run prefect server start or you can explore any level of the CLI with prefect --help or prefect server --help
2. the database is locked error reflects a performance issue specific to sqlite, which may or may not be related to the NullPool issue you linked. I think that issue is fine as a standalone issue for now, but I think more information about how that arose for you would be helpful

for now I think I'll close this issue and track the locking issues on the other two issues i mentioned earlier

please feel free to let me know if you think there's a standalone reason to keep this open!

I followed this reply to block the log information. After executing the first command, I executed my python program and the error log still existed. (The second command does not exist)