Code

Offload literals#2872

Code

Merged

eapolinario merged 28 commits intomasterflyteorg/flytekit:masterfrom offload-literalsflyteorg/flytekit:offload-literalsCopy head branch name to clipboardNov 22, 2024

+717-18

Commits

Commits on Oct 18, 2024

• wip - Implement offloading of literals

  Show description for 12f9edf
  eapolinario

  committed
  12f9edf

• Fix use of metadata bucket prefix

  Show description for 5fefa85
  eapolinario

  committed
  5fefa85

• Fix repeated use of uri

  Show description for 1668fda
  eapolinario

  committed
  1668fda

• Add temporary representation for offloaded literal

  Show description for 853df62
  eapolinario

  committed
  853df62

Commits on Oct 22, 2024

• Add one unit test

  Show description for 5e53a1b
  eapolinario

  committed
  5e53a1b

• Add another test

  Show description for 177368d
  eapolinario

  committed
  177368d

• Stylistic changes to the two tests

  Show description for 5fc2e84
  eapolinario

  committed
  5fc2e84

Commits on Oct 28, 2024

• Add test for min offloading threshold set to 1MB

  Show description for db48d18
  eapolinario

  committed
  db48d18

• Pick a unique engine-dir for tests

  Show description for 6884ee0
  eapolinario

  committed
  6884ee0

• s/new_outputs/literal_map_copy/

  Show description for 5a6423c
  eapolinario

  committed
  5a6423c

• Remove unused constant

  Show description for dbfea93
  eapolinario

  committed
  dbfea93

• Merge remote-tracking branch 'origin' into offload-literals

  Show description for adeed34
  eapolinario

  committed
  adeed34

Commits on Oct 29, 2024

• Use output_prefix in definition of offloaded literals

  Show description for 25908d9
  eapolinario

  committed
  25908d9

Commits on Nov 7, 2024

• Add initial version of pbhash.py

  Show description for e827693
  eapolinario

  committed
  e827693

• Add tests to verify that overriding the hash is carried over to offloaded literals

  Show description for e0e2016
  eapolinario

  committed
  e0e2016

• Add a few more tests

  Show description for b284492
  eapolinario

  committed
  b284492

• Always import ParamSpec from typing_extensions

  Show description for b579b83
  eapolinario

  committed
  b579b83

• Fix lint warnings

  Show description for 8c2336e
  eapolinario

  committed
  8c2336e

• Merge remote-tracking branch 'origin' into offload-literals

  Show description for c28d537
  eapolinario

  committed
  c28d537

Commits on Nov 8, 2024

• Set inferred_type using the task type interface

  Show description for 37b2bb4
  eapolinario

  committed
  37b2bb4

• Add comment about offloaded literals files and how they are uploaded to the metadata bucket

  Show description for e25496d
  eapolinario

  committed
  e25496d

Commits on Nov 18, 2024

• Add offloading_enabled

  Show description for 9276e96
  eapolinario

  committed
  9276e96

Commits on Nov 19, 2024

• Add more unit tests including a negative test

  Show description for a8bdbca
  eapolinario

  committed
  a8bdbca

• Merge remote-tracking branch 'origin' into offload-literals

  Show description for fe822b9
  eapolinario

  committed
  fe822b9

• Fix bad merge

  Show description for a4fcfab
  eapolinario

  committed
  a4fcfab

• Incorporate feedback.

  Show description for b3a1b0d
  eapolinario

  committed
  b3a1b0d

Commits on Nov 21, 2024

• Fix image name (unrelated to this PR - just a nice-to-have to decrease flakiness)

  Show description for 32c5896
  eapolinario

  committed
  32c5896

• Add is_map_task to _dispatch_execute

  Show description for 12e194a
  eapolinario

  committed
  12e194a