Documentation: Warning for _AIRFLOW_PATCH_GEVENT

[ppenelon]

Hi,

I recently had a huge pain point with Airflow running on ECS via the official airflow Docker image (apache/airflow:2.9.2).

For the configuration I have a scheduler (4 vCPU, 8 Gib), some workers (4 vCPU, 8 Gib) and a webserver (2 vCPU, 4 Gib).

Every containers are sharing the same environment variables to apply the same configuration anywhere.

The problem was that at some point, the scheduler were non-stop crashing with the following error:

File ""/home/airflow/.local/lib/python3.12/site-packages/airflow/jobs/scheduler_job_runner.py"", line 1738, in _find_zombies
 self.job.executor.send_callback(request)
File ""/home/airflow/.local/lib/python3.12/site-packages/airflow/executors/base_executor.py"", line 499, in send_callback
 self.callback_sink.send(request)
File ""/home/airflow/.local/lib/python3.12/site-packages/airflow/callbacks/pipe_callback_sink.py"", line 47, in send
 self._get_sink_pipe().send(callback)
File ""/usr/local/lib/python3.12/multiprocessing/connection.py"", line 206, in send
 self._send_bytes(_ForkingPickler.dumps(obj))
File ""/usr/local/lib/python3.12/multiprocessing/connection.py"", line 427, in _send_bytes
 self._send(header + buf)
File ""/usr/local/lib/python3.12/multiprocessing/connection.py"", line 384, in _send
  n = write(self._handle, buf)
     ^^^^^^^^^^^^^^^^^^^^^^^^
BlockingIOError: [Errno 11] Resource temporarily unavailable

The first attempt was to increase the maximum file descriptor for the airflow user from 1000 (default) to 65535 (root limit) which works some time but after some days the problem came back.

After many (many) unsuccessful attempts, I saw some content related with this error and the gevent patching.

The webserver is using AIRFLOW__WEBSERVER__WORKER_CLASS=gevent so it naturally came with _AIRFLOW_PATCH_GEVENT=1.

As the environment variables are shared between all containers, it means that the _AIRFLOW_PATCH_GEVENT environment variable was applied on the webserver but also on the scheduler.

I'm not a Python expert but the conclusion was that the gevent patching was affecting the multiprocessing module via the duplex sockets then causing troubles on the scheduler internal communications.

As it was a painful problem to solve I'm proposing this documentation upgrade to warn anyone to apply this setup.

I'm not sure if it's the right way to do it but feel free to take/modify/close this merge request the way you want, as it can already serve anyone searching for this error via Google.

Thanks

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

• Pay attention to the quality of your code (ruff, mypy and type annotations). Our pre-commits will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
• Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: [email protected]
  Slack: https://s.apache.org/airflow-slack

[potiuk]

The compat error already fixed in main.

[boring-cyborg]

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

[ppenelon]

Thanks @potiuk, out of curiosity I would love to learn more about the fix if you have a commit, pull request or anything.

[potiuk]

#40690

[ppenelon]

I'm not sure how this is related, but thanks for your time @potiuk

[potiuk]

I'm not sure how this is related, but thanks for your time @potiuk

Your PR failed (in compatibility tests) because of a main issue that this PR fixed - maybe that was not clear :). It was only CI issue - maybe It was too much brevity in the description :)

[ppenelon]

Oh thanks for the precision, I misunderstood your first message, have a great day :)