Flyte v1.10.0 milestone release

[flyte-bot]

Flyte v1.10.0 Release

This release marks the move to the (monorepo](#4014). From now on the development of backend components is going to be done in the Flyte main repo.

Flyte Agents are also getting a major lift. They are not only more performant, but we're also announcing support for a variety of agents, including Airflow, Memverge, Snowflake, Databricks.

Programmatically consuming inputs and outputs using flyteremote became a lot easier with the introduction of code snippets directly in the UI. For example, notice the small blurbs below inputs and outputs in the side panel:

You'll now be able to use offloaded types in eager workflows.

More ergonomic improvements to pyflyte, including the inclusion of a progress bar, the ability to activate launchplans, and the ability to interact with gate nodes in local executions.

And much more. Here's the exhaustive list of changes:

Flytekit

• Better error messaging for overrides by @kumare3 in Better error messaging for overrides flytekit#1807
• Run remote Launchplan from pyflyte run by @kumare3 in Run remote Launchplan from pyflyte run flytekit#1785
• Add is none function by @pingsutw in Add is none function flytekit#1757
• Dynamic workflow should not throw nested task warning by @oliverhu in Dynamic workflow should not throw nested task warning flytekit#1812
• Add a manual image building GH action by @wild-endeavor in Add a manual image building GH action flytekit#1816
• Enable Azure Workload Identity for fsspec in flytekit by @fiedlerNr9 in Enable Azure Workload Identity for fsspec in flytekit flytekit#1813
• Fix list of annotated structured dataset by @wild-endeavor in Fix list of annotated structured dataset flytekit#1817
• Support the flytectl config.yaml admin.clientSecretEnvVar option in flytekit by @chaohengstudent in Support the flytectl config.yaml admin.clientSecretEnvVar option in flytekit flytekit#1819
• Async agent delete function for while loop case by @Future-Outlier in Async agent delete function for while loop case flytekit#1802
• fix docs warnings by @samhita-alla in fix docs warnings flytekit#1827
• Fix extract_task_module by @pingsutw in Fix extract_task_module flytekit#1829
• Feat: Add type support for pydantic BaseModels by @ArthurBook in Feat: Add type support for pydantic BaseModels flytekit#1660
• Make FlyteRemote example slightly more copy/pastable by @katrogan in Make FlyteRemote example slightly more copy/pastable flytekit#1830
• Pyflyte meta inputs by @kumare3 in Pyflyte meta inputs flytekit#1823
• Use mashumaro to serialize/deserialize dataclass by @hhcs9527 in Use mashumaro to serialize/deserialize dataclass flytekit#1735
• Databricks Agent by @Future-Outlier in Databricks Agent flytekit#1797
• Prometheus metrics by @pingsutw in Prometheus metrics flytekit#1815
• Pyflyte register optionally activates schedule by @kumare3 in Pyflyte register optionally activates schedule flytekit#1832
• Remove versions 3.9 and 3.10 by @wild-endeavor in Remove versions 3.9 and 3.10 flytekit#1831
• Snowflake agent by @hhcs9527 in Snowflake agent flytekit#1799
• Update agent metric name by @pingsutw in Update agent metric name flytekit#1835
• MemVerge MMCloud Agent by @edwinyyyu in MemVerge MMCloud Agent flytekit#1821
• Add download badges to the readme by @pingsutw in Add download badges to the readme flytekit#1836
• Eager local entrypoint and support for offloaded types by @cosmicBboy in Eager local entrypoint and support for offloaded types flytekit#1833
• update requirements and add snowflake agent to api reference by @samhita-alla in update requirements and add snowflake agent to api reference flytekit#1838
• Fix: Make sure decks created in elastic task workers are transferred to parent process by @fg91 in Fix: Make sure decks created in elastic task workers are transferred to parent process flytekit#1837
• add accept grpc by @wild-endeavor in add accept grpc flytekit#1841
• Feat: Enable flytekit to authenticate with proxy in front of FlyteAdmin by @fg91 in Feat: Enable flytekit to authenticate with proxy in front of FlyteAdmin flytekit#1787
• Backfill command now supports failure-policy by @kumare3 in Backfill command now supports failure-policy flytekit#1840
• Pass cluster pool when creating executions by @iaroslav-ciupin in Pass cluster pool when creating executions flytekit#1208
• Add more clear error message when fetching secrets by @ysysys3074 in Add more clear error message when fetching secrets flytekit#1847
• Pyflyte run workflows correctly handles Optional[TYPE] = None by @cosmicBboy in Pyflyte run workflows correctly handles Optional[TYPE] = None flytekit#1849
• Bump gitpython from 3.1.32 to 3.1.35 in /tests/flytekit/integration/remote/mock_flyte_repo/workflows by @dependabot in Bump gitpython from 3.1.32 to 3.1.35 in /tests/flytekit/integration/remote/mock_flyte_repo/workflows flytekit#1828
• Bump cryptography from 41.0.3 to 41.0.4 in /tests/flytekit/integration/remote/mock_flyte_repo/workflows by @dependabot in Bump cryptography from 41.0.3 to 41.0.4 in /tests/flytekit/integration/remote/mock_flyte_repo/workflows flytekit#1844
• update codecov config in pythonbuild ci by @cosmicBboy in update codecov config in pythonbuild ci flytekit#1852
• Improved gate node handling by @kumare3 in Improved gate node handling flytekit#1850
• update codecov yaml, make eager wf test more stable by @cosmicBboy in update codecov yaml, make eager wf test more stable flytekit#1854
• more codecov updates by @cosmicBboy in more codecov updates flytekit#1856
• make mlflow plugin work with python 3.11 by @bcvanmeurs in make mlflow plugin work with python 3.11 flytekit#1855
• Agent get secret function by @Future-Outlier in Agent get secret function flytekit#1851
• Better Azure blob storage support by @Tom-Newton in Better Azure blob storage support  flytekit#1842
• Remove Envd from dockerfile by @pingsutw in Remove Envd from dockerfile flytekit#1814
• Update agent uri format by @hhcs9527 in Update agent uri format flytekit#1861
• modified setup.py in flytekit-mmcloud by @helenzhangyc in modified setup.py in flytekit-mmcloud flytekit#1864
• Label request_failure metric with error_code by @pingsutw in Label request_failure metric with error_code flytekit#1862
• add back Edwin's name by @helenzhangyc in add back Edwin's name flytekit#1868
• Fix py38 aync agent service and add async agent test by @Future-Outlier in Fix py38 aync agent service and add async agent test flytekit#1866
• Pin because not installing by @wild-endeavor in Pin because not installing flytekit#1872
• Bump pillow from 10.0.0 to 10.0.1 in /tests/flytekit/integration/remote/mock_flyte_repo/workflows by @dependabot in Bump pillow from 10.0.0 to 10.0.1 in /tests/flytekit/integration/remote/mock_flyte_repo/workflows flytekit#1867
• Prune hosted github runner to free up space for sandbox by @jeevb in Prune hosted github runner to free up space for sandbox flytekit#1875
• Fixes Auth issue, where refresh token is expired by @kumare3 in Fixes Auth issue, where refresh token is expired flytekit#1873
• Name field is incorrect assigned by @kumare3 in Name field is incorrect assigned flytekit#1874
• [Docs]Updated examples for customizing resources by @LunarMarathon in [Docs]Updated examples for customizing resources flytekit#1871
• change key path to SNOWFLAKE_PRIVATE_KEY by @hhcs9527 in change key path to SNOWFLAKE_PRIVATE_KEY flytekit#1879
• Run notebook task on aws batch by @pingsutw in Run notebook task on aws batch flytekit#1859
• Automatically Convert name Argument and registry Argument to Lowercase in ImageSpec by @Future-Outlier in Automatically Convert name Argument and registry Argument to Lowercase in ImageSpec flytekit#1881
• Dep: Pin min required flytekit version in IAP plugin by @fg91 in Dep: Pin min required flytekit version in IAP plugin flytekit#1878
• Remove duplicate click option by @wild-endeavor in Remove duplicate click option flytekit#1883
• Bump urllib3 from 1.26.16 to 1.26.17 in /tests/flytekit/integration/remote/mock_flyte_repo/workflows by @dependabot in Bump urllib3 from 1.26.16 to 1.26.17 in /tests/flytekit/integration/remote/mock_flyte_repo/workflows flytekit#1865
• Improve error message in ListTransformer when not passing a list by @fg91 in Improve error message in ListTransformer when not passing a list flytekit#1882
• Update typing_extension Any import to typing due to typing_extension version compatibility by @hfurkanvural in Update typing_extension Any import to typing due to typing_extension version compatibility flytekit#1877
• Bump gitpython from 3.1.35 to 3.1.37 in /tests/flytekit/integration/remote/mock_flyte_repo/workflows by @dependabot in Bump gitpython from 3.1.35 to 3.1.37 in /tests/flytekit/integration/remote/mock_flyte_repo/workflows flytekit#1885
• Airflow agent by @pingsutw in Airflow agent flytekit#1725
• Add mmcloud plugin into the agent image by @pingsutw in Add mmcloud plugin into the agent image flytekit#1891
• Fix map_task sensitive to argument order by @chaohengstudent in Fix map_task sensitive to argument order flytekit#1888
• Add enable_deck to task decorator by @eapolinario in Add enable_deck to task decorator flytekit#1898
• Add image transformer by @pingsutw in Add image transformer flytekit#1901
• add default project domain in execute launch plan by @troychiu in add default project domain in execute launch plan flytekit#1894
• Add Databricks and Snowflake agent to Dockerfile by @Future-Outlier in Add Databricks and Snowflake agent to Dockerfile flytekit#1905
• Fix module loading bug in instance tracker class by @cosmicBboy in Fix module loading bug in instance tracker class flytekit#1902

Flyteadmin

• Fixed bug for incorrect name (Fixed bug for incorrect name #4175)
• Add version control for ArrayNode eventing (Add version control for ArrayNode eventing #4165)
• Update golang experimental dep (Update golang experimental dep #4177)
• Add Additional gRPC service handler hook (Add Additional gRPC service handler hook #4171)
• Handle batched TaskExecutionEvent reasons ([BUG] Increase admin max header data response size #615)
• Lint flyteadmin/auth/handlers_test.go
• #patch Update FromWorkflowModel API (Release 0.9.0 #617)
• add codespell: config, workflow and have typos fixed (add codespell: config, workflow and have typos fixed #4054)
• Handle batched TaskExecutionEvent reasons ([BUG] Increase admin max header data response size #615)
• Update FromWorkflowModel API (Release 0.9.0 #617)
• Update boilerplate version (Fast Register #605)
• Reduce a warning to info (Update docs #612)
• Logout hook plugin (SdkTask/SdkLaunchPlan.Fetch go/nogo #611)
• Expire flyte_idt cookie at logout (Making old tasks work in new workflows #610)
• Use configured domain on expired cookies (Revisit Typed Schema #608)
• Fix list named entities (Map tasks #609)
• Merge pull request from GHSA-r847-6w6h-r8g4
• Update boilerplate version (Update design doc #604)
• Handle grpc error explicitly (Branches #602)
• Introduce pre redirect hook plugin during auth callback (Arbitrary container images #601)
• Add default cluster pool assignments to config (Hive tasks #600)
• Implement Sandbox notifications processor and publisher (Typed Schema #595)

Flytepropeller

• [Extended Resources] GPU Accelerators ([Extended Resources] GPU Accelerators #4172)
• [Monorepo] Enable proxy-authorization in admin client ([Monorepo] Enable proxy-authorization in admin client #4189)
• (monorepo--importing-flyteidl-452) Add version control for ArrayNode eventing (Add version control for ArrayNode eventing #4165)
• Update golang experimental dep (Update golang experimental dep #4177)
• Moving from flytepropeller - Adding flags for ignore-retry-cause and default-max-attempts (Moving from flytepropeller - Adding flags for ignore-retry-cause and default-max-attempts #4153)
• [Monorepo] Rename flyteidl ([Monorepo] Rename flyteidl #4166)
• Revert add supported task types for agent service by default for task types (Revert add supported task types for agent service by default for task types #4162)
• reverting to parallelism increases only on non-terminal phase updates ([BUG] FlyteAdmin should validate launch plan labels are k8s compliant #624)
• Add k8s events to task phase updates (Hive tasks #600)
• Ray log links (Attempt RTFD combined repo #620)
• Add supportTaskTypes for agentservice without write it in config twice. (Update docs #612)
• Bump flytestdlib version (Milestone v0.9.0 docs #619)
• added check for invalid argument code on node event recording failure (#613)
• start informerFactory by default. (Add docs for fast register #614)
• bumped flyteplugins to v1.1.23 ([BUG][UI] Schedules using day-of-week are offset forward by one day in the UI #616)
• fixing boilerplate (Release 0.9.0 #617)
• Supporting using arraynode as a subnode (Shift operator #606)
• fixing max parallelism (SdkTask/SdkLaunchPlan.Fetch go/nogo #611)
• adding controller-runtime manager metrics handler to http endpoint (All existing task types available in new style #607)
• pulled latest boilerplate (Map tasks #609)
• Move caching to node executor for fast cache hits (Updates to Getting Started docs #485)

Flyteplugins

• Update the item only if it exists in the cache (Update the item only if it exists in the cache #4117)
• [Extended Resources] GPU Accelerators ([Extended Resources] GPU Accelerators #4172)
• Implement Spark pod template overrides (Implement Spark pod template overrides #4183)
• Support Evaluator in Kubeflow TensorFlow Training Operator (Support Evaluator in Kubeflow TensorFlow Training Operator #4168)
• Agent Get Pending Message (Agent Get Pending Message #4106)
• Moving from flytepropeller - Adding flags for ignore-retry-cause and default-max-attempts (Moving from flytepropeller - Adding flags for ignore-retry-cause and default-max-attempts #4153)
• change to tf job worker spec ([flyteplugins] Make the TF Job Worker Spec Correct #4169)
• Return ResourceWrapper without pointer (Return ResourceWrapper without pointer #4115)
• Revert add supported task types for agent service by default for task types (Revert add supported task types for agent service by default for task types #4162)
• Plugin changes for plumbing k8s events into TaskExecutionEvent (Updates Readme: features, talks and coming soon #406)
• Fix flyteplugins lint errors
• add codespell: config, workflow and have typos fixed (add codespell: config, workflow and have typos fixed #4054)
• Add supportTaskTypes for agentservice without write it in config twice. ([FlyteCtl] Delete Execution #398)
• Create a FileOutput reader if the agent produce file output ([FlyteCtl] Create workflow #391)
• Bump flytestdlib version ([FlyteCtl] Update Outputs #404)
• feat: Dask add pod template support (Flyte v0.5.0 release #374)
• Mark webapi task failure as retry limit exceeded ([FlyteCtl] Create LaunchPlan #392)
• Make rendering template in agent client temporary ([FlyteCtl] Create Execution #395)
• enable secrets in non-python k8s tasks ([FlyteCtl] Update MatchableEntity #401)
• Rename agent chart (Rename agent chart #4019)
• Fix resources requests for AWS Batch plugin ([FlyteCtl] Get MatchableEntity #400)
• Transition to Queue if the JobCondition is empty ([FlyteCtl] Create Tasks #387)
• Fix databricks plugin test Naming Error ([FlyteCtl] Get LaunchPlan #393)
• Updating boilerplate to lock golangci-lint version ([FlyteCtl] Update LaunchPlan #394)
• Fix double http in the Spark Driver UI Link ([FlyteCtl] Get Workflow #389)
• Render task template in the agent client ([FlyteCtl] Create Project #384)

Flyteconsole

• show correct details panel io ([Docs]Documentation for various plugins and how to use/configure them #820)
• resume form should support all known types ([Docs]Documentation of various configuration knobs in Flyte #819)
• optional list arg is causing ui launch form to white screen ([Plugin] BigQuery plugin #817)
• breadcrumb dropdown render ([BUG] Latest flytestdlib doesn't create container when it doesn't exist #815)
• switching projects doesn't respect domain value (Test resolver #814)
• invalid item in dropdown ([Plugin][Flytekit] flytekit storage layer as plugins #809)
• flyte decks height (Add an enter #813)

datacatalog

• b5a958f Refactoring profile handler into entrypoints (Updated the slack link to point to Google form for now #113)

---

This discussion was created from the release Flyte v1.10.0 milestone release.