Error on Moving from Sequential to LocalExecutor

[sheharbano96]

I just updated my airflow setup to use LocalExecutor but it started giving this error:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/spawn.py", line 126, in _main
    self = reduction.pickle.load(from_parent)
AttributeError: 'QueuedLocalWorker' object has no attribute 'wrapper'

It works fine with Sequential Executor but keeps queuing tasks with this error on the scheduler.

[Shubhisnottaken]

have you resolved this?

[keichi]

Thanks for you advice, I am using macOS so 3. might be the issue. 1. and 2. should be fine since I installed Python via Homebrew and Airflow via PyPI using the constraints file. I'll try on a Linux system.

[potiuk]

Thanks for you advice, I am using macOS so 3. might be the issue. 1. and 2. should be fine since I installed Python via Homebrew and Airflow via PyPI using the constraints file. I'll try on a Linux system.

Looking at the paths - you ARE using the built-in Python

/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/spawn.py"

This - very likely - is the system Python.

I guess even if you have installed Airflow with brew it might not be used by default. On top of it you are not using virtualenv - which is generaly a bad idea, - installng application in "python" without creating venv is generally very strongly discourage by the pip team and anyone from packaging team. It causes a LOT of problems when you mix system dependencies and pip installed packages. Latest version of pip even does not allow you to do it and you need to specigy --allow-breaking-system-packages or smth like that flag when you try to do it.

I strongly recommend using uv including uv python and looking at setting up venv with it. It's lightning fast (comparing to brew managed python) and it has nice workflow with venv management

[keichi]

I think you are confusing me with @sheharbano96

[potiuk]

Ah, Might be. That's always a risk when you comment on other's thread and continue conversation with your case. Sometimes it's worth to start new discussion with "relate to #` to avoid it. When you have 10-15 messages in a discussion and as maintainer you handle 100s of them a day, clear separation helps, so that you do not have to re read the whole thread every time you come back to it.

Just for the future :)

[sheharbano96]

@potiuk Yes, I am using macOS but I have installed Airflow in the python's virtual environment. I'll try updating the interpreter and reinstalling airflow with constraints and see if the issue persists.
I will be running it on Linux for production but testing it on my local which is macOS. The SequentialExecutor doesn't work for sending slack updates so I was trying to update it to LocalExecutor to test slack updates.
Thank you for your response.