Manual Interventions

[dylanbstorey]

Description

A "manual intervention" operator as a core component of airflow. As an analogy, you commonly see these as last stages of a CI/CD process to release and distribute artifacts after successful builds.

Use case/motivation

Sometimes it is useful/necessary to keep a "human in the loop" for some data processing tasks. In a few occurrences I've had the need allow for an external (non "data") party to be able to release the last stage of a data pipeline as a task - doing this in airflow required us creating an external service + sensor for managing the checkpoint. The general flow looked like this :

• task registers lock with external service + sends message to user that it is ready
• sensor begins polling for lock release
• user clicks link to "clear" the lock
• sensor releases downstream tasks.

I could see this being useful to other users of airflow, but as a core feature. With the introduction of dataset awareness + deferrable operators, i believe it will also be able to be designed in a better way than back in Airflow 1.

Related issues

Not to my knowledge.

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[o-nikolas]

Hey @dylanbstorey

Thanks for the interesting feature request!
Manual steps that I've used in build/release pipelines in the past often have some kind of UI element as well (some things you're suggested to verify, often a text box to submit a "reason" or "justification" for your manual approval or override, a button to click to approve, etc). What do you imagine from that point of view for such an operator or first class feature?

This would need much more investigation and discussion with the community (and you should definitely do that!). Speaking of which, would you like me to assign this task to you, I see you've checked that you're willing to submit a PR 😃

[potiuk]

I think this is far more than PR - it's more of an AIP. It's been raised by a number of people and there are similar discussions. For example we had a discussion #27325 about the very thing. Seems super useful, but IMHO this really needs a deliberate discussion and it needs not a "feature" but Airflow Improvement Proposal - https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals - we should discuss the approach, users, permissions, whether to make it plugs into current Sensors, Defferables, how to handle asyc/io approach, how we shoudl communicate between the "sensor" and webserver, how to handle cancellations, whether we should introduce separate state (this was proposed in #27325 . how it should be surfaced in the UI, etc. etc.

I think at this moment there are more questions than answers and since it is crossing all layers in airflow, I think this is quite a big Proposal to make, not a feature.

So rather than openinig a PR (or even opening an issue) I think if someone wants to implement it, starting with discussing it in devlist and writing an Airflow Improvement Proposal with detailed discussion about the design is a good idea (and it is very welcome - seems that this is highly requested feature).

Maybe @dylanbstorey or @cdabella from the other thread would like to pick it up ? Or maybe someone else will soon ? Hopefully we will get it soon.

For now - I am converting it into a discussion, it's far too big and not defined to be a feature. (but again it would be a welcome addition).

[dylanbstorey]

About where i expected this to end up. We'll start drafting an AIP.

[janaki-sasidhar]

Use prefect?

[aswinselva03]

Any progress on this?