[flytepropeller][flyteadmin] Streaming Decks V2 #6053

Future-Outlier · 2024-11-27T15:38:59Z

Tracking issue

Why are the changes needed?

To enhance user visibility into Flyte Decks at different stages of workflow execution (running, failing, and succeeding), enabling better debugging and analysis.

What changes were proposed in this pull request?

Concept:

propeller will turn node info to NodeExecutionEvent, and send it to admin.

flyte/flytepropeller/pkg/controller/nodes/executor.go

Lines 1251 to 1261 in b3330ba

    
           nev, err := ToNodeExecutionEvent( 
        
           	nCtx.NodeExecutionMetadata().GetNodeExecutionID(), 
        
           	p, 
        
           	nCtx.InputReader().GetInputPath().String(), 
        
           	nCtx.NodeStatus(), 
        
           	nCtx.ExecutionContext().GetEventVersion(), 
        
           	nCtx.ExecutionContext().GetParentInfo(), nCtx.Node(), 
        
           	c.clusterID, 
        
           	nCtx.NodeStateReader().GetDynamicNodeState().Phase, 
        
           	c.eventConfig, 
        
           	targetEntity)

Life Cycle:

use new flytekit > 1.14.0

summary:

NO HEAD request to be called. (save resource)
use config from task template to know whether enable deck or not

details:

propeller keep adding DeckURI when the task is running if FLYTE_ENABLE_DECK=true in the task template.
propeller will put DeckURI to node info, and turn it to NodeExecutionEvent to flyte admin.
flyte admin will add DeckURI to Closure

flyte console will get DeckURI by sending request to admin.

flyte/flyteadmin/dataproxy/service.go

Lines 175 to 189 in b3330ba

    
           		nativeURL = node.GetClosure().GetDeckUri() 
        
           	} 
        
           } else { 
        
           	return nil, errors.NewFlyteAdminErrorf(codes.InvalidArgument, "unsupported source [%v]", reflect.TypeOf(req.GetSource())) 
        
           } 
        
           if len(nativeURL) == 0 { 
        
           	return nil, errors.NewFlyteAdminErrorf(codes.Internal, "no deckUrl found for request [%+v]", req) 
        
           } 
        
           ref := storage.DataReference(nativeURL) 
        
           meta, err := s.dataStore.Head(ctx, ref) 
        
           if err != nil { 
        
           	return nil, errors.NewFlyteAdminErrorf(codes.Internal, "failed to head object before signing url. Error: %v", err) 
        
           }

if flyte console can't get the DeckURI from the node Closure, it will not show the Flyte Deck button.

old flytekit <= 1.14.0

summary:

we keep the backward compatible (show deck when succeed)

details:

In the terminal state, use a HEAD request to know if the Deck URI exists or not.
if exist, then put it to the node info.

How was this patch tested?

unit test and remote execution.

python code:

from flytekit import ImageSpec, task, workflow
from flytekit.deck import Deck

flytekit_hash = "6b55930d0a77efc3594ebaac056f2c75024e61b5"
flytekit = f"git+https://github.com/flyteorg/flytekit.git@{flytekit_hash}"

# Define custom image for the task
custom_image = ImageSpec(packages=[flytekit],
                            apt_packages=["git"],
                            registry="localhost:30000",
                            env={"FLYTE_SDK_LOGGING_LEVEL": 10},
                         )

@task(enable_deck=False, container_image=custom_image)
def t_no_deck():
    # Deck.publish()
    print("No Deck")

@task(enable_deck=True, container_image=custom_image)
def t_deck():
    import time
    """
    1st deck only show timeline deck
    2nd will show
    """
    for i in range(3):
        Deck.publish()
        time.sleep(1)

@task(enable_deck=True, container_image=custom_image)
def t_fail_deck():
    import time

    for i in range(3):
        Deck.publish()
        time.sleep(3)
    time.sleep(10)
    raise ValueError("Failed Deck")

@workflow
def wf():
    t_no_deck()
    t_deck()
    t_fail_deck()

if __name__ == "__main__":
    from flytekit.clis.sdk_in_container import pyflyte
    from click.testing import CliRunner
    import os

    runner = CliRunner()
    path = os.path.realpath(__file__)

    result = runner.invoke(pyflyte.main,
                           ["run", path, "t_no_deck"])
    print("Local Execution: ", result.output)

    result = runner.invoke(pyflyte.main,
                           ["run", "--remote", path,"wf"])
    print("Remote Execution: ", result.output)

Setup process

single binary.

flyte: this branch
flytekit: flyteorg/flytekit#2779
flyteconsole: flyteorg/flyteconsole#890

Screenshots

flytekit branch:
flyteorg/flytekit#2779

NEW FLYTEKIT, NO DECK, RUNNING With Deck, SUCCEED, and FAILED

OSS-STREAMING-DECK-small.mov

OLD FLYTEKIT, NO DECK, RUNNING With Deck, SUCCEED, and FAILED

OSS-STREAMING-DECK-OLD-FLYTEKIT-small.mov

Check all the applicable boxes

I updated the documentation accordingly.
All new and existing tests passed.
All commits are signed-off.

Related PRs

follow up questions

should we support Abort phase for the streaming deck?

should we support EPhaseAbort in this file?

https://github.com/flyteorg/flyte/blob/b3330ba4430538f91ae9fc7d868a29a2e96db8bd/flytepropeller/pkg/controller/nodes/handler/transition_info.go

how can we support the auto-refresh UX?

Signed-off-by: Future-Outlier <[email protected]>

Signed-off-by: Future-Outlier <[email protected]> Co-authored-by: Yi Cheng <[email protected]> Co-authored-by: pingsutw <[email protected]>

codecov · 2024-11-27T15:45:04Z

Codecov Report

Attention: Patch coverage is 31.81818% with 60 lines in your changes missing coverage. Please review.

Project coverage is 36.97%. Comparing base (ab04192) to head (4068043).
Report is 30 commits behind head on master.

Files with missing lines	Patch %	Lines
...lytepropeller/pkg/controller/nodes/task/handler.go	31.03%	51 Missing and 9 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6053      +/-   ##
==========================================
- Coverage   37.08%   36.97%   -0.11%     
==========================================
  Files        1318     1318              
  Lines      132284   132511     +227     
==========================================
- Hits        49062    49001      -61     
- Misses      78950    79250     +300     
+ Partials     4272     4260      -12

Flag	Coverage Δ
unittests-datacatalog	`51.58% <ø> (ø)`
unittests-flyteadmin	`54.06% <100.00%> (-0.04%)`	⬇️
unittests-flytecopilot	`30.99% <ø> (ø)`
unittests-flytectl	`62.29% <ø> (-0.05%)`	⬇️
unittests-flyteidl	`7.23% <ø> (-0.01%)`	⬇️
unittests-flyteplugins	`53.85% <ø> (+0.11%)`	⬆️
unittests-flytepropeller	`42.55% <31.03%> (-0.09%)`	⬇️
unittests-flytestdlib	`55.18% <ø> (-2.36%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Future-Outlier <[email protected]>

hamersaw · 2024-11-27T16:52:38Z

flytepropeller/pkg/controller/nodes/task/handler.go

 	switch pluginTrns.pInfo.Phase() {
 	case pluginCore.PhaseSuccess:
+		// This is to prevent the console from potentially checking the deck URI that does not exist if in final phase(PhaseSuccess).
+		err = pluginTrns.RemoveNonexistentDeckURI(ctx, tCtx)


Does this do a head call on the deck URI for every task that succeeds? Two thoughts here:
(1) does the flyteadmin merge algorithm then remove the deckURI from the execution metadata?
(2) this is incurring a 20-30ms performance degredation to every task execution

will take a look tmr, thank you!!!

Does this do a head call on the deck URI for every task that succeeds?

yes it will do a head call by RemoteFileOutputReader

flyte/flyteplugins/go/tasks/pluginmachinery/ioutils/remote_file_output_reader.go

Lines 306 to 313 in b3330ba

func (r RemoteFileOutputReader) DeckExists(ctx context.Context) (bool, error) {

md, err := r.store.Head(ctx, r.outPath.GetDeckPath())

if err != nil {

return false, err

}

return md.Exists(), nil

}

how do you know the performance degradation?
did you use grafana or other performance tools?

does the flyteadmin merge algorithm then remove the deckURI from the execution metadata?

flyteadmin will set the deckURI in the execution metadata to nil if the propeller removes it.

cc @hamersaw

Future-Outlier · 2024-11-27T17:02:52Z

How to test it?

start a new sandbox

flytectl demo start --image futureoutlier/sandbox:deck-1205-1138 --force

checkout streaming deck flytekit branch

cd flytekit
gh pr checkout 2779

run a failure task (show deck after it failed)

from flytekit import ImageSpec, task, workflow
from flytekit.deck import Deck

flytekit_hash = "473ae1119af6f86c26c0790dee0affa3eb29be64"
flytekit = f"git+https://github.com/flyteorg/flytekit.git@{flytekit_hash}"

# Define custom image for the task
custom_image = ImageSpec(packages=[flytekit],
                            apt_packages=["git"],
                            registry="localhost:30000",
                            env={"FLYTE_SDK_LOGGING_LEVEL": 10},
                         )

@task(enable_deck=True, container_image=custom_image)
def t_deck():
    import time
    """
    1st deck only show timeline deck
    2nd will show
    """
    for i in range(5):
        Deck.publish()
        # # raise Exception("This is an exception")
        time.sleep(3)

@workflow
def wf():
    t_deck()

if __name__ == "__main__":
    from flytekit.clis.sdk_in_container import pyflyte
    from click.testing import CliRunner
    import os

    runner = CliRunner()
    path = os.path.realpath(__file__)

    # result = runner.invoke(pyflyte.main,
    #                        ["run", path, "wf"])
    # print("Local Execution: ", result.output)

    result = runner.invoke(pyflyte.main,
                           ["run", "--remote", path,"wf"])
    # "--remote"
    print("Remote Execution: ", result.output)

EngHabu · 2024-11-28T00:38:55Z

Mind adding screenshots for the rendered deck and refresh to the PR description?

Future-Outlier · 2024-11-28T01:52:29Z

Mind adding screenshots for the rendered deck and refresh to the PR description?

Yes no problem

Future-Outlier · 2024-11-28T02:10:23Z

Mind adding screenshots for the rendered deck and refresh to the PR description?

its provided!
#6053 (comment)

Signed-off-by: Future-Outlier <[email protected]>

Future-Outlier and others added 2 commits November 27, 2024 23:36

add tests from Yi-Cheng

54aa165

Signed-off-by: Future-Outlier <[email protected]>

helped by Kevin and Yi-Cheng

9ed6b6e

Signed-off-by: Future-Outlier <[email protected]> Co-authored-by: Yi Cheng <[email protected]> Co-authored-by: pingsutw <[email protected]>

lint

4b4f6bd

Signed-off-by: Future-Outlier <[email protected]>

hamersaw reviewed Nov 27, 2024

View reviewed changes

nit

dd774cb

Signed-off-by: Future-Outlier <[email protected]>

Future-Outlier requested a review from hamersaw December 12, 2024 13:37

Future-Outlier added 3 commits December 13, 2024 23:18

add comments

0bb8e91

Signed-off-by: Future-Outlier <[email protected]>

add comments and better solution for backward compativle

25fea29

Signed-off-by: Future-Outlier <[email protected]>

better comments

4e24e91

Signed-off-by: Future-Outlier <[email protected]>

Future-Outlier mentioned this pull request Dec 17, 2024

[flyte deck] Streaming Decks flyteorg/flytekit#2779

Open

3 tasks

Future-Outlier added 3 commits December 18, 2024 23:39

DeckStatus

8d1d0e4

Signed-off-by: Future-Outlier <[email protected]>

rename GetDeckStatus

31853bb

Signed-off-by: Future-Outlier <[email protected]>

comments

4068043

Signed-off-by: Future-Outlier <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[flytepropeller][flyteadmin] Streaming Decks V2 #6053

[flytepropeller][flyteadmin] Streaming Decks V2 #6053

Future-Outlier commented Nov 27, 2024 •

edited

Loading

codecov bot commented Nov 27, 2024 •

edited

Loading

hamersaw Nov 27, 2024

Future-Outlier Nov 27, 2024

Future-Outlier Nov 28, 2024 •

edited

Loading

Future-Outlier Nov 28, 2024

Future-Outlier Nov 28, 2024

Future-Outlier Nov 28, 2024

Future-Outlier commented Nov 27, 2024 •

edited

Loading

EngHabu commented Nov 28, 2024

Future-Outlier commented Nov 28, 2024

Future-Outlier commented Nov 28, 2024

	nev, err := ToNodeExecutionEvent(
	nCtx.NodeExecutionMetadata().GetNodeExecutionID(),
	p,
	nCtx.InputReader().GetInputPath().String(),
	nCtx.NodeStatus(),
	nCtx.ExecutionContext().GetEventVersion(),
	nCtx.ExecutionContext().GetParentInfo(), nCtx.Node(),
	c.clusterID,
	nCtx.NodeStateReader().GetDynamicNodeState().Phase,
	c.eventConfig,
	targetEntity)

	nativeURL = node.GetClosure().GetDeckUri()
	}
	} else {
	return nil, errors.NewFlyteAdminErrorf(codes.InvalidArgument, "unsupported source [%v]", reflect.TypeOf(req.GetSource()))
	}

	if len(nativeURL) == 0 {
	return nil, errors.NewFlyteAdminErrorf(codes.Internal, "no deckUrl found for request [%+v]", req)
	}

	ref := storage.DataReference(nativeURL)
	meta, err := s.dataStore.Head(ctx, ref)
	if err != nil {
	return nil, errors.NewFlyteAdminErrorf(codes.Internal, "failed to head object before signing url. Error: %v", err)
	}

	func (r RemoteFileOutputReader) DeckExists(ctx context.Context) (bool, error) {
	md, err := r.store.Head(ctx, r.outPath.GetDeckPath())
	if err != nil {
	return false, err
	}

	return md.Exists(), nil
	}

[flytepropeller][flyteadmin] Streaming Decks V2 #6053

Are you sure you want to change the base?

[flytepropeller][flyteadmin] Streaming Decks V2 #6053

Conversation

Future-Outlier commented Nov 27, 2024 • edited Loading

Tracking issue

Why are the changes needed?

What changes were proposed in this pull request?

Concept:

Life Cycle:

use new flytekit > 1.14.0

old flytekit <= 1.14.0

How was this patch tested?

Setup process

Screenshots

NEW FLYTEKIT, NO DECK, RUNNING With Deck, SUCCEED, and FAILED

OLD FLYTEKIT, NO DECK, RUNNING With Deck, SUCCEED, and FAILED

Check all the applicable boxes

Related PRs

follow up questions

codecov bot commented Nov 27, 2024 • edited Loading

Codecov Report

hamersaw Nov 27, 2024

Choose a reason for hiding this comment

Future-Outlier Nov 27, 2024

Choose a reason for hiding this comment

Future-Outlier Nov 28, 2024 • edited Loading

Choose a reason for hiding this comment

Future-Outlier Nov 28, 2024

Choose a reason for hiding this comment

Future-Outlier Nov 28, 2024

Choose a reason for hiding this comment

Future-Outlier Nov 28, 2024

Choose a reason for hiding this comment

Future-Outlier commented Nov 27, 2024 • edited Loading

How to test it?

EngHabu commented Nov 28, 2024

Future-Outlier commented Nov 28, 2024

Future-Outlier commented Nov 28, 2024

Future-Outlier commented Nov 27, 2024 •

edited

Loading

codecov bot commented Nov 27, 2024 •

edited

Loading

Future-Outlier Nov 28, 2024 •

edited

Loading

Future-Outlier commented Nov 27, 2024 •

edited

Loading