Running default Dagster Databricks Pipes yield a Engine Event Error

[srggrs]

What's the issue?

Running the example as provided in the documentation will yield an error in the execution that does not prevent the asset to be running but it is unclear if the logs are passed back to dagster or not or if it fails to do some message passing task.

What did you expect to happen?

No error at all

How to reproduce?

Run the Databricks Dagster Pipes examples as reported in the online documentation

Dagster version

1.9.0

Deployment type

Local

Deployment details

Run using python=3.10
and
dagster-databricks==0.25.0
dagster-dbt==0.25.0

Additional information

Stacktrace of the error

[pipes] framework exception occurred in PipesDbfsMessageReader logs thread
KeyError: 'extras'

Stack Trace:
  File "/<path to py env>/lib/python3.10/site-packages/dagster/_core/pipes/utils.py", line 463, in _logs_thread
    if unstarted_log_readers[key].target_is_readable(params):
,  File "/<path to py env>/lib/python3.10/site-packages/dagster_databricks/pipes.py", line 428, in target_is_readable
    return self._get_log_path(params) is not None
,  File "/<path to py env>/lib/python3.10/site-packages/dagster_databricks/pipes.py", line 457, in _get_log_path
    cluster_driver_log_root = params["extras"].get("cluster_driver_log_root")

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.
By submitting this issue, you agree to follow Dagster's Code of Conduct.