Airflow DAG Updates Not Reflecting in UI Without File Rename or Reserialization (v2.7.3 on Kubernetes)

[galeyal2]

I’m facing a problem with Airflow (v2.7.3 on Kubernetes) where changes made to the function code in a DAG aren’t reflected in the Airflow UI consistently. Specifically:

When I modify the function code in a separate file used by the DAG and push the updates to Git, the UI doesn’t show these changes unless I rename the file.

Running airflow dags reserialize temporarily displays the updated DAG in the UI.

After approximately one hour, the UI reverts back to showing the original DAG version, even though no further code changes or restarts have occurred.

This behavior is puzzling because it seems like Airflow is caching an old version of the DAG. I expected that pushing updates to Git would automatically reflect in the DAG on the UI without requiring a file rename or manual reserialization.

Here’s what I tried so far:

Pushing Changes to Git: I modified the function code in a separate file used by the DAG and pushed these updates to Git, expecting the Airflow UI to automatically display the updated DAG version.

Running airflow dags reserialize: Since the UI did not reflect the changes, I tried running airflow dags reserialize, which did make the updated DAG visible in the UI. However, I was hoping this would be a permanent fix.

Waiting for Persistence of Changes: After running airflow dags reserialize, the DAG updates appeared temporarily, but after approximately an hour, the UI reverted to the previous version. My expectation was that once the changes were reflected in the UI, they would remain stable without further manual intervention.

Renaming the Function File: To force an update, I tried renaming the function file, which did reflect the changes in the UI. This approach seems impractical, as I’d prefer not to rename files every time a code change is made.

• I also tried restarting all Airflow services, but this did not make any difference.

Any solutions or explanation would be greatly appreciated.

Here’s a simplified view of the main DAG and the function file:

Main DAG File (perception-enrichments-regression-prototype.py):

with DAG('perception-enrichments-regression-prototype',
         default_args=DEFAULT_ARGS,
         schedule_interval="@daily",
         description='perception-enrichments-prototype',
         max_active_runs=3,
         catchup=False,
         tags=["perception-enrichments"],
         is_paused_upon_creation=True,
         ) as dag:
    set_alert_email_in_dag(dag)

    init_dag_variables = PythonOperator(
        task_id="init-dag-variables3",
        python_callable=init_dag_variables_callable,
        provide_context=True,
        trigger_rule=TriggerRule.NONE_FAILED,
    )

    start = EmptyOperator(task_id='perception-enrichments-prototype-start')
    done = EmptyOperator(task_id='perception-enrichments-prototype-done')

    start >> init_dag_variables >> get_ada_enrich_prototype("{{ ti.xcom_pull(key='pit') }}", dag) >> done

Function File (groups/annotation_group/ada_enrich_prototype.py):

from typing import Optional
from airflow.operators.bash import BashOperator
from airflow.operators.empty import EmptyOperator
from airflow.operators.python import PythonOperator
from airflow.utils.task_group import TaskGroup

def get_ada_enrich_prototype(dag, pit: Optional[str] = None):
    with TaskGroup(group_id='ada-enrich-prototype') as ada_enrich_prototype:
        start = EmptyOperator(task_id='start-enrich-prototype123')
        start2 = EmptyOperator(task_id='start2-enrich-prototype123')
        start3 = BashOperator(task_id='start3-enrich-prototype123', bash_command='echo "Hello World"')
        done = EmptyOperator(task_id='done-enrich-prototype', trigger_rule="all_done")
        
        start >> start2 >> start3 >> done
    return ada_enrich_prototype

[potiuk]

I think this is some kind of deployment issue - where either you have some old version of files being re-parsed by some running instance of scheduler or maybe bytecode files are still present in your DAG folder. Similar cases happened where people had an old version of airlfow scheduler still running and using old version of files (And using the same database). Likely something like that happens to you