Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixed the endless sensor reschedule #45224

Conversation

morooshka
Copy link
Contributor

Fixed the endless sensor reschedule

Rescheduling mode. In case of unhandled exception occurred during the sensor's first try run the rescheduling attempt is not saved into DB because of the full transaction rollback. the "first try" here can be also the first one after clearing the task. In that case the start moment for calculating the task running duration should be taken as:

  • In case of first retry after exception: current system time
  • The first successfully saved to DB rescheduled tries reschedule date for the following rescheduled runs

closes: #45050

@boring-cyborg boring-cyborg bot added the area:core-operators Operators, Sensors and hooks within Core Airflow label Dec 26, 2024
@morooshka morooshka force-pushed the fix/issue-45050-endless-reschedule-if-first-try-runtime-error branch from 22581c1 to 4080eff Compare December 26, 2024 22:19
Copy link
Contributor

@jscheffl jscheffl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix. Sounds meaningful. Special thanks for the extensions and intensive test added as well.

I'd assume it also makes sense to port this back to Airflow 2.10-line? Does anything speaks against this?

airflow/sensors/base.py Show resolved Hide resolved
@jscheffl jscheffl added the backport-to-v2-10-test Mark PR with this label to backport to v2-10-test branch label Dec 26, 2024
@jscheffl jscheffl added this to the Airflow 2.10.5 milestone Dec 26, 2024
@jscheffl jscheffl merged commit 7f2b8ef into apache:main Dec 27, 2024
50 checks passed
Copy link

Backport failed to create: v2-10-test. View the failure log Run details

Status Branch Result
v2-10-test Commit Link

You can attempt to backport this manually by running:

cherry_picker 7f2b8ef v2-10-test

This should apply the commit to the v2-10-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

jscheffl pushed a commit to jscheffl/airflow that referenced this pull request Dec 27, 2024
(cherry picked from commit 7f2b8ef)

Co-authored-by: morooshka <[email protected]>
@jscheffl jscheffl added the type:bug-fix Changelog: Bug Fixes label Dec 27, 2024
jscheffl added a commit that referenced this pull request Dec 27, 2024
(cherry picked from commit 7f2b8ef)

Co-authored-by: morooshka <[email protected]>
jason810496 pushed a commit to jason810496/airflow that referenced this pull request Dec 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:core-operators Operators, Sensors and hooks within Core Airflow backport-to-v2-10-test Mark PR with this label to backport to v2-10-test branch type:bug-fix Changelog: Bug Fixes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Endless sensor rescheduling if the first sensor run failed to be saved in DB
2 participants