Support for LocalExecutor

[bigdataguineapig]

Is there a way for users to run with LocalExecutor rather than SequentialExecutor?

This would require moving away from sqllite in order to support multiple connections to the database. Maybe it would be possible to spin up a second container with something like postgres or mariadb?

[rishi-aga]

Here is a workaround on mac

1. Install MySQL locally, start server and create a database

   > brew install MySQL
   > mysql.server start
   > mysql -u root
   
   mysql> CREATE DATABASE airflow_db CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
   Query OK, 1 row affected (0.00 sec)
   
   mysql> CREATE USER 'airflow_user' IDENTIFIED BY 'airflow_pass';
   Query OK, 0 rows affected (0.00 sec)
   
   mysql> GRANT ALL PRIVILEGES ON airflow_db.* TO 'airflow_user';
   Query OK, 0 rows affected (0.00 sec)
   
   mysql> exit

2. Edit composer-local-dev/composer_local_dev/constants.py, search NOT_MODIFIABLE_ENVIRONMENT_VARIABLES and delete both elements in the set to make it empty.
3. Create your local env, then add below entries to variables.env

AIRFLOW__CORE__EXECUTOR=LocalExecutor
AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=mysql+mysqldb://airflow_user:[email protected]/airflow_db

4. Start your local env.

[bigdataguineapig]

Thanks, ideally I am looking for a way to isolate the db in case I want to have multiple composer environments. This would be common if testing a migration to a more recent version of airflow.

Ideally, would support a multi container approach for the db. Celery would be nice but not required, for example the file shown here. https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html#fetching-docker-compose-yaml

[matiasmicen]

@bigdataguineapig first of all thanks for the steps, it worked!

I upgraded the composer to 2.6.5 with airflow 2.7.3 and now I got this message when I start the composer locally:

2024-04-05T10:19:42 WARNING composer_local_dev.environment:'AIRFLOW__CORE__EXECUTOR' environment variable cannot be set and will be ignored.

I checked the Configuration on the Admin menu and got the core executore in SequentialExecutor :(

Do you know what could be the issue?
Thanks!

[fabiosangregorio]

+1, testing concurrency with SequentialExecutor is not ideal
PS: thanks @rishi-aga, worked like a charm!

[molcay]

Hello all (@bigdataguineapig, @rishi-aga, @matiasmicen, @fabiosangregorio)

I started to work on this issue to add support for LocalExecutor while I was trying to use this workaround explained by @rishi-aga. I realized one point and would like to confirm with you:
@matiasmicen did you install the composer-dev via pip install composer-dev? For me, installing the composer-dev via pip install composer-dev did not work because we are changing the source code of composer-dev. So, we need to install the composer-dev in editable mode using pip install -e .. Otherwise, the changes in the composer-local-dev/composer_local_dev/constants.py file cannot effect the composer-dev executable.
This might be the case for you also.