[Core feature] Neptune Integration at the workflow level

[tomtou-bspace]

Motivation: Why do you think this is important?

I have a machine learning workflow that logs metadata and metrics to neptune. each task logs different metrics but they should all log to the same neptune run. right now each task opens a new neptune run unless an existing run_id is explicitly fed to the decorator, which isn't really useful in my case.

Goal: What should the final outcome look like, ideally?

there should be an option where the workflow creates a neptune run and pushes it to the context of all tasks.

Describe alternatives you've considered

tasks check if a neptune run exists in the workflow context, if it doesnt exist, it creates a run and pushes it to the common context

Propose: Link/Inline OR Additional context

a dynamic workflow neptune decorator that eventually creates a neptune run when executed , then pushes the run_id to the context of tasks in that workflow. possibly allowing for specific run params as input when executing workflow programmaticaly e.g custom_run_id (could be the execution name)
another Idea could be that tasks can check in the workflow context if a run id already exists i.e if another task already created the run, if not the task should create the run and update the state of the common context so that other tasks use that.

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[eapolinario]

right now each task opens a new neptune run unless an existing run_id is explicitly fed to the decorator, which isn't really useful in my case.

@tomtou-bspace , can you expand on that?