Run workflow from step N reusing inputs from another run

[EvgeniiTitov]

Hi,

I am currently working on a Flyte POC. I am migrating one of our KF pipelines and I am trying to understand if I can run a workflow from step N and then provide another workflow run id to reuse the inputs from.

I've been reading the docs and I can barely find any info about it. ChatGPT suggested using the recover function, which doesn't seem to exist in flytekit.

Could anyone give me a hint how that could be achieved in Flyte, please? Our use case is not necessarily just recovering a failed run, but rather a rapid prototyping when DS people could run a heavy workflow end to end, and then, say, rerun only certain parts of the workflow later (model training with different hyperparams etc) reusing artifacts from the previous complete run (heave SQL + feature engineering). So, ideally they say I want to run from step ModelTraining reusing features generated in another run id 123. I want to do it programmatically from python/CLI

Thank you!

ET

[Future-Outlier]

You can use create a cache task and reference it!
cache: https://docs.flyte.org/en/latest/user_guide/development_lifecycle/cache_serializing.html#cache-serializing
reference: https://docs.flyte.org/en/latest/user_guide/productionizing/reference_tasks.html#reference-tasks