Where Parts of Flyte Code Get Run

[sayvazov]

We had been running into some issues with Flyte workflows and tasks that are intended to run in different containers. Our project is academia-adjacent, so we have to use a lot of legacy code and rarely maintained libraries. This also means that the versions of even well supported products, like Numpy, are basically locked into when a particular project was developed. Our solution to this has been to push for actual coding standards and to be very aggressive about containerizing everything. Flyte provides great support for running particular tasks in specific containers, but it isn't always clear where particular code is run, so we had been running into issues with imports. Yee has been very kind in clearing up a lot of our questions and misconceptions, so a minimal example and summary is below.

Assume that we have two tasks, A and B, that run in containerA and containerB respectively. We make an update to a local version of a code file, and try to build a workflow that runs both tasks, and register it to a running cloud instance of Flyte. This means we actually have 4 distinct environments:

1. The environment that docker and flytectl are run in
2. The environment that Flyte is running in
3. containerA
4. containerB

When you tell Flyte that task A is to be run in containerA, Flyte will only try to execute taskA in containerA. But in order to register the workflow with the remote Flyte instance, you need to build the containers and then generate the protobuf, which happens in environment 1. In order to do this, you must import and run the python files where these tasks are defined. This means that while the task isn't run in environment 1, the task must be defined in environment 1. This in particular means that all of the imports must succeed, which causes issues if, say, certain functions exist only in a particular version of a library you're using. You could run flytectl out of containerA, but then taskB's dependencies also need to be in containerA. Essentially, environment 1 must be able to define, if not run, all tasks.

The options to solving this issue are to either gate imports to library version (as you can for python2/3 imports), or to move the imports in question into the task function bodies. Because the functions are never run, you don't have to worry about the imports failing in environment 1 any more. The long term solution is that you actually want a third Docker image that you use exclusively for building workflows out of. This image needs to contain all the libraries and code that you use, but doesn't need to actually be able to run anything. In fact, you could write up some stub libraries to replace the real ones, and install them only in this image.

[sayvazov]

@kumare3 for proofreading and visibility.

[georgesnelling]

@sayvazov, thank you for taking the time to write this up so clearly.