Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] FlyteFile from a remote path should be read correctly in a task #6090

Open
2 tasks done
Future-Outlier opened this issue Dec 6, 2024 · 1 comment · May be fixed by flyteorg/flytekit#2991
Open
2 tasks done
Assignees
Labels
bug Something isn't working flytekit FlyteKit Python related issue

Comments

@Future-Outlier
Copy link
Member

Future-Outlier commented Dec 6, 2024

Describe the bug

right now, this can't work.

from flytekit import task, workflow
from flytekit.types.file import FlyteFile

@task
def create_ff() -> FlyteFile:
    full_s3_path = "s3://my-s3-bucket/test.json"
    ff = FlyteFile(full_s3_path)
    with open(ff, "r") as f:
        content = f.read()
        print(content)
        print(type(content))
    return ff

Expected behavior

This should work.

Additional context to reproduce

Error message

/Users/future-outlier/miniconda3/envs/dev/bin/python /Users/future-outlier/code/dev/flytekit/build/dataclass_remote_api/json_example.py 
23:05:24.489138 INFO     file.py:252 - Using flytectl/YAML config               
                         /Users/future-outlier/.flyte/config-sandbox.yaml       
23:05:26.822284 INFO     utils.py:344 - AsyncTranslate literal to python value. 
                         [Time: 0.000002s]                                      
23:05:26.823714 INFO     utils.py:344 - Translate literal to python value.      
                         [Time: 0.001452s]                                      
23:05:26.824123 INFO     base_task.py:751 - Invoking json_example.create_ff with
                         inputs: {}                                             
23:05:26.824461 INFO     utils.py:344 - Execute user level code. [Time:         
                         0.000017s]                                             
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /Users/future-outlier/code/dev/flytekit/build/dataclass_remote_api/json_exam │
│ ple.py:32 in <module>                                                        │
│                                                                              │
│ ❱ 32wf()                                                                │
│                                                                              │
│ /Users/future-outlier/code/dev/flytekit/flytekit/core/workflow.py:308 in     │
│ __call__                                                                     │
│                                                                              │
│ ❱ 308 │   │   │   raise exc                                                  │
│                                                                              │
│ /Users/future-outlier/code/dev/flytekit/flytekit/core/workflow.py:301 in     │
│ __call__                                                                     │
│                                                                              │
│ ❱ 301 │   │   │   return flyte_entity_call_handler(self, *args, **input_kwar │
│                                                                              │
│ /Users/future-outlier/code/dev/flytekit/flytekit/core/promise.py:1459 in     │
│ flyte_entity_call_handler                                                    │
│                                                                              │
│ ❱ 1459 │   │   │   result = cast(LocallyExecutable, entity).local_execute(ch │
│                                                                              │
│ /Users/future-outlier/code/dev/flytekit/flytekit/core/workflow.py:327 in     │
│ local_execute                                                                │
│                                                                              │
│ ❱ 327 │   │   function_outputs = self.execute(**kwargs_literals)             │
│                                                                              │
│ /Users/future-outlier/code/dev/flytekit/flytekit/core/workflow.py:829 in     │
│ execute                                                                      │
│                                                                              │
│ ❱ 829 │   │   return self._workflow_function(**kwargs)                       │
│                                                                              │
│ /Users/future-outlier/code/dev/flytekit/build/dataclass_remote_api/json_exam │
│ ple.py:27 in wf                                                              │
│                                                                              │
│ ❱ 27ff = create_ff()                                                    │
│                                                                              │
│ /Users/future-outlier/code/dev/flytekit/flytekit/core/base_task.py:364 in    │
│ __call__                                                                     │
│                                                                              │
│ ❱ 364 │   │   return flyte_entity_call_handler(self, *args, **kwargs)  # typ │
│                                                                              │
│ /Users/future-outlier/code/dev/flytekit/flytekit/core/promise.py:1451 in     │
│ flyte_entity_call_handler                                                    │
│                                                                              │
│ ❱ 1451 │   │   │   return cast(LocallyExecutable, entity).local_execute(ctx, │
│                                                                              │
│ /Users/future-outlier/code/dev/flytekit/flytekit/core/base_task.py:341 in    │
│ local_execute                                                                │
│                                                                              │
│ ❱ 341 │   │   │   outputs_literal_map = self.sandbox_execute(ctx, input_lite │
│                                                                              │
│ /Users/future-outlier/code/dev/flytekit/flytekit/core/base_task.py:421 in    │
│ sandbox_execute                                                              │
│                                                                              │
│ ❱ 421 │   │   return self.dispatch_execute(ctx, input_literal_map)           │
│                                                                              │
│ /Users/future-outlier/code/dev/flytekit/flytekit/core/base_task.py:754 in    │
│ dispatch_execute                                                             │
│                                                                              │
│ ❱ 754 │   │   │   │   │   native_outputs = self.execute(**native_inputs)     │
│                                                                              │
│ /Users/future-outlier/code/dev/flytekit/flytekit/core/python_function_task.p │
│ y:204 in execute                                                             │
│                                                                              │
│ ❱ 204 │   │   │   return self._task_function(**kwargs)                       │
│                                                                              │
│ /Users/future-outlier/code/dev/flytekit/build/dataclass_remote_api/json_exam │
│ ple.py:10 in create_ff                                                       │
│                                                                              │
│ ❱ 10with open(ff, "r") as f:                                            │
╰──────────────────────────────────────────────────────────────────────────────╯
FileNotFoundError: [Errno 2] No such file or directory: 
's3://my-s3-bucket/test.json'

Process finished with exit code 1

Screenshots

No response

Are you sure this issue hasn't been raised already?

  • Yes

Have you read the Code of Conduct?

  • Yes
@Future-Outlier Future-Outlier added bug Something isn't working untriaged This issues has not yet been looked at by the Maintainers labels Dec 6, 2024
@JiangJiaWei1103
Copy link
Contributor

#take

@JiangJiaWei1103 JiangJiaWei1103 linked a pull request Dec 8, 2024 that will close this issue
3 tasks
@eapolinario eapolinario added flytekit FlyteKit Python related issue and removed untriaged This issues has not yet been looked at by the Maintainers labels Dec 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working flytekit FlyteKit Python related issue
Projects
Status: Backlog
Development

Successfully merging a pull request may close this issue.

3 participants