Merge branch 'master' into update_end2end_execute

docs/deployment/deployment/cloud_simple.rst

@@ -29,9 +29,11 @@ these prerequisites.
 
 .. note::
 
-   `Union.AI <https://www.union.ai/>`__ plans to open-source a reference
-   implementation of these requirements for the major cloud providers in early
-   2023.
+
+   `Union.ai <https://www.union.ai/>`__ maintains a `set of Terraform scripts <https://github.com/unionai-oss/deploy-flyte>`__ that automate the configuration
+   of prerequisites and Flyte installation on AWS, GCP, or Azure.
+
+   A community-maintained guide to manually prepare an EKS environment and deploy Flyte is available `here <https://github.com/davidmirror-ops/flyte-the-hard-way/tree/main>`__
 
 ***************
 Installation

docs/user_guide/data_types_and_io/structureddataset.md

@@ -68,7 +68,7 @@ First, initialize column types you want to extract from the `StructuredDataset`.
 
 ```{literalinclude} /examples/data_types_and_io/data_types_and_io/structured_dataset.py
 :caption: data_types_and_io/structured_dataset.py
-:lines: 31-32
+:lines: 36-37
 ```
 
 Define a task that opens a structured dataset by calling `all()`.
@@ -78,7 +78,7 @@ For instance, you can use ``pa.Table`` to convert the Pandas DataFrame to a PyAr
 
 ```{literalinclude} /examples/data_types_and_io/data_types_and_io/structured_dataset.py
 :caption: data_types_and_io/structured_dataset.py
-:lines: 42-52
+:lines: 47-57
 ```
 
 The code may result in runtime failures if the columns do not match.
@@ -91,7 +91,7 @@ and enable the CSV serialization by annotating the structured dataset with the C
 
 ```{literalinclude} /examples/data_types_and_io/data_types_and_io/structured_dataset.py
 :caption: data_types_and_io/structured_dataset.py
-:lines: 58-72
+:lines: 63-77
 ```
 
 ## Storage driver and location
@@ -230,14 +230,14 @@ and the byte format, which in this case is `PARQUET`.
 
 ```{literalinclude} /examples/data_types_and_io/data_types_and_io/structured_dataset.py
 :caption: data_types_and_io/structured_dataset.py
-:lines: 128-130
+:lines: 133-135
 ```
 
 You can now use `numpy.ndarray` to deserialize the parquet file to NumPy and serialize a task's output (NumPy array) to a parquet file.
 
 ```{literalinclude} /examples/data_types_and_io/data_types_and_io/structured_dataset.py
 :caption: data_types_and_io/structured_dataset.py
-:lines: 135-148
+:lines: 140-153
 ```
 
 :::{note}
@@ -248,7 +248,7 @@ You can run the code locally as follows:
 
 ```{literalinclude} /examples/data_types_and_io/data_types_and_io/structured_dataset.py
 :caption: data_types_and_io/structured_dataset.py
-:lines: 152-156
+:lines: 157-161
 ```
 
 ### The nested typed columns
@@ -261,7 +261,7 @@ Nested field StructuredDataset should be run when flytekit version > 1.11.0.
 
 ```{literalinclude} /examples/data_types_and_io/data_types_and_io/structured_dataset.py
 :caption: data_types_and_io/structured_dataset.py
-:lines: 158-285
+:lines: 163-290
 ```
 
 [flytesnacks]: https://github.com/flyteorg/flytesnacks/tree/master/examples/data_types_and_io/

docs/user_guide/flyte_agents/developing_agents.md

@@ -7,6 +7,7 @@ jupytext:
 ---
 
 (developing_agents)=
+
 # Developing agents
 
 The Flyte agent framework enables rapid agent development, since agents are decoupled from the core FlytePropeller engine. Rather than building a complete gRPC service from scratch, you can implement an agent as a Python class, easing development. Agents can be tested independently and deployed privately, making maintenance easier and giving you more flexibility and control over development.
@@ -20,8 +21,9 @@ We strongly encourage you to contribute your agent to the Flyte community. To do
 ```
 
 There are two types of agents: **async** and **sync**.
-* **Async agents** enable long-running jobs that execute on an external platform over time. They communicate with external services that have asynchronous APIs that support `create`, `get`, and `delete` operations. The vast majority of agents are async agents.
-* **Sync agents** enable request/response services that return immediate outputs (e.g. calling an internal API to fetch data or communicating with the OpenAI API).
+
+- **Async agents** enable long-running jobs that execute on an external platform over time. They communicate with external services that have asynchronous APIs that support `create`, `get`, and `delete` operations. The vast majority of agents are async agents.
+- **Sync agents** enable request/response services that return immediate outputs (e.g. calling an internal API to fetch data or communicating with the OpenAI API).
 
 ```{note}
 
@@ -41,6 +43,17 @@ To create a new async agent, extend the [`AsyncAgentBase`](https://github.com/fl
 - `get`: This method retrieves the job resource (jobID or output literal) associated with the task, such as a BigQuery job ID or Databricks task ID.
 - `delete`: Invoking this method will send a request to delete the corresponding job.
 
+```{note}
+
+When users use the `create` method to create a new job, with its job ID, they can use the `get` method with job ID to 
+check the execution state is succeeded or not.
+
+Exceptional `delete` case:
+If users interrupt a task while it is running, FlytePropeller will invoke the `delete` method to the corresponding 
+job.
+
+```
+
 ```python
 from typing import Optional
 from dataclasses import dataclass
@@ -113,6 +126,7 @@ AgentRegistry.register(OpenAIAgent())
 ```
 
 #### Sensor interface specification
+
 With the agent framework, you can easily build a custom sensor in Flyte to watch certain events or monitor the bucket in your workflow.
 
 To create a new sensor, extend the `[BaseSensor](https://github.com/flyteorg/flytekit/blob/master/flytekit/sensor/base_sensor.py#L43)` class and implement the `poke` method, which checks whether a specific condition is met.
@@ -130,7 +144,6 @@ class FileSensor(BaseSensor):
         return fs.exists(path)
 ```
 
-
 ### 2. Test the agent
 
 You can test your agent in a {ref}`local Python environment <testing_agents_locally>` or in a {ref}`local development cluster <testing_agents_in_a_local_development_cluster>`.
@@ -181,29 +194,29 @@ By default, all agent requests will be sent to the default agent service. Howeve
 you can route particular task requests to designated agent services by adjusting the FlytePropeller configuration.
 
 ```yaml
- plugins:
-   agent-service:
-     # By default, all requests will be sent to the default agent.
-     defaultAgent:
-       endpoint: "dns:///flyteagent.flyte.svc.cluster.local:8000"
-       insecure: true
-       timeouts:
-         # CreateTask, GetTask and DeleteTask are for async agents.
-         # ExecuteTaskSync is for sync agents.
-         CreateTask: 5s
-         GetTask: 5s
-         DeleteTask: 5s
-         ExecuteTaskSync: 10s
-       defaultTimeout: 10s
-     agents:
-       custom_agent:
-         endpoint: "dns:///custom-flyteagent.flyte.svc.cluster.local:8000"
-         insecure: false
-         defaultServiceConfig: '{"loadBalancingConfig": [{"round_robin":{}}]}'
-         timeouts:
-           GetTask: 5s
-         defaultTimeout: 10s
-     agentForTaskTypes:
-       # It will override the default agent for custom_task, which means propeller will send the request to this agent.
-       - custom_task: custom_agent
+plugins:
+  agent-service:
+    # By default, all requests will be sent to the default agent.
+    defaultAgent:
+      endpoint: "dns:///flyteagent.flyte.svc.cluster.local:8000"
+      insecure: true
+      timeouts:
+        # CreateTask, GetTask and DeleteTask are for async agents.
+        # ExecuteTaskSync is for sync agents.
+        CreateTask: 5s
+        GetTask: 5s
+        DeleteTask: 5s
+        ExecuteTaskSync: 10s
+      defaultTimeout: 10s
+    agents:
+      custom_agent:
+        endpoint: "dns:///custom-flyteagent.flyte.svc.cluster.local:8000"
+        insecure: false
+        defaultServiceConfig: '{"loadBalancingConfig": [{"round_robin":{}}]}'
+        timeouts:
+          GetTask: 5s
+        defaultTimeout: 10s
+    agentForTaskTypes:
+      # It will override the default agent for custom_task, which means propeller will send the request to this agent.
+      - custom_task: custom_agent
 ```