Dagster-Slack: Slack message decorator for the successful materialisation of a Pandas DataFrame. #23726
CHRISCARLON
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Summary
The existing methods in the Dagster-Slack package weren't sufficient for my use case.
What I needed to do:
Send a formatted message containing a basic overview of a Pandas DataFrame to a specified Slack channel upon the successful materialisation of an asset.
The formatted message needs to look like this...
Proposed Feature
Is there a way to create a decorator function to achieve the following:
Send a formatted message containing a basic overview of a Pandas DataFrame to a specified Slack channel upon the successful materialisation of an asset.
I think I need to access the AssetExecutionContext directly and couldn't seem to find a way to access the data I needed with existing HookContext methods.
As a result, I created a workaround that works pretty well so far.
Would it be possible to formally integrate something like this into the Dagster-Slack codebase?
Something that's better than my hacky workaround.
Workaround:
Benefits of the Proposed Feature
In my case, the Pandas DataFrame ends up being processed into a DeltaLake table in a Silver Data Bucket (Medallion Data Lake Structure) for use in downstream data products.
I think integrating a feature like this would be great for Data Engineers who are looking for better/improved ways to communicate with Data Analysts - in particular for communicating what data Analysts can expect to receive further downstream for dashboard data products etc.
Many thanks,
Chris
Beta Was this translation helpful? Give feedback.
All reactions