Add support for custom celery configs

[arorasachin9]

closes: #45037

Description:
Currently Airflow support limited celery options only. This PR adds the support for the additional celery config for celery workers.

1. Changes are completely backward compatible
2. No test cases found for these changes
3. If the config is not available then default value of {} is taken which will be same as earlier only
4. If config is available then it will be added to the celery config and applied to celery workers.

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

• Pay attention to the quality of your code (ruff, mypy and type annotations). Our pre-commits will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
• Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: [email protected]
  Slack: https://s.apache.org/airflow-slack

[jscheffl]

Thanks for the PR - the contribution looks really small and is efficient. Thanks for taking care for the backwards compatibility.

I assume the fallback handling in default_celery.py:72is not needed if the config entry is correctly registered as a default in providers/src/airflow/providers/celery/provider.yaml - There all configs for the provider should be registered. This also would add missing documentation.

[jedcunningham]

Can you also add a test for this?

[arorasachin9]

Addressed the comments. CI is failing not able to debug why CI pipeline is failing.

[jscheffl]

Except some minor comment I think it is good, let me check what is broken on CI. Nothing related to your code.
Ignore that it is "red" atm, will be better once you push the next change on the PR.

Then I think... LGTM!

[jscheffl]

Somthing is wrong with this PR and CI - I don't see the reason but I am very sure it is not related to the changes. I am on it...

[jscheffl]

Okay, I also did not know but defaults seem to be Format-String formatted and the string "{}" is an invalid sequence. So to effectively have an {} as default we need to use {{}} here to quote the brackets.

Suggested change

	default: "{}"
	default: "{{}}"

[jscheffl]

Okay ... interesting, this also resolves the CI problem - it seems the constrains also can not be built as the templating fails there. I did a PR with the fixes applied as test and this is turning green: #45160
As with the other proposal a check is being made that the settings are a dict, you can even lave an empty string as default...

Suggested change

	default: "{}"
	default: ""

[jscheffl]

As CI did not get to this stage (yet) as failing earlier, there will be a problem reported by mypy for providers like:

providers/src/airflow/providers/celery/executors/default_celery.py:72: error:
Incompatible types in assignment (expression has type
"Union[dict[Any, Any], list[Any], str, int, float, None]", variable has type
"dict[Any, Any]")  [assignment]
    extra_celery_config: dict = conf.getjson("celery", "extra_celery_confi...
                                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~...
Found 1 error in 1 file (checked 1 source file)
Error 1 returned

As the JSON result can be other values than a dict (irrespective of the type declaration above, you need to use:

Suggested change

	**extra_celery_config,
	**(extra_celery_config if isinstance(extra_celery_config, dict) else {}),