add flyte test setup

flyte.qmd

@@ -1,3 +1,4 @@
+# ![](images/favicon-flyte-docs.webp){.logo_h1_inline} Flyte
 ## ![](images/favicon-flyte-docs.webp){.logo_h2_inline} Flyte
 
 - [Flyte](https://flyte.org/) is a highly scalable cloud-native workflow orchestration platform on top of containers and kubernetes
@@ -14,7 +15,12 @@
 - **Launch Plan**: A scheduled execution of a workflow
 
 
-## Flyte Architecture
+## Flyte Test Setup
+
+![](images/flyte-test-setup.svg){fig-align="center"}
+
+Thanks to Martin Wendt (ITS) for providing a comprehensive testing setup including Kubernetes (k3s), Flyte, and MinIO.
+
 
 ## Flyte Command-line Interface
 
@@ -28,7 +34,7 @@ flytectl create project --name seminar --id seminar --description "seminar showc
 
 ## Flyte Tasks
 
-- Tasks are strongly typed (See [Flyte Type Mapping](https://docs.flyte.org/en/latest/user_guide/data_types_and_io/index.html#data-types-and-io))
+- Tasks are strongly typed (see [Flyte Type Mapping](https://docs.flyte.org/en/latest/user_guide/data_types_and_io/index.html#data-types-and-io))
 - Tasks are regular Python function decorated with `@task` 
 - Tasks are designed to be idempotent (multiple executions don't change the result).
 
@@ -50,7 +56,7 @@ todo ...
 
 A Flyte task operates within its own container and runs on a Kubernetes pod
 
-![](images/union_tasks.png)
+![](images/union_tasks.png){fig-align="center"}
 
 
 ## ImageSpec
@@ -77,6 +83,7 @@ custom_image = ImageSpec(
 - Clone and recover executions
 
 
-## Flyte Demo: PyTorch MNIST Classifier 
+## Flyte Demo
 
-[Source](https://docs.flyte.org/en/latest/flytesnacks/examples/mnist_classifier/pytorch_single_node_and_gpu.html)
+- [GitHub repo](https://github.com/BerndDoser/flyte-mnist)
+- [PyTorch MNIST Classifier](https://docs.flyte.org/en/latest/flytesnacks/examples/mnist_classifier/pytorch_single_node_and_gpu.html)

hpc-ai.qmd

@@ -1,3 +1,11 @@
+# HPC vs AI/ML Infrastructure
+## {background-image="images/ai_vs_hpc.png"}
+
+::: aside
+Generated by FLUX.1 on fal.ai
+:::
+
+
 ## HPC vs AI/ML Infrastructure {background-image="images/ai_vs_hpc.png" background-opacity=0.2}
 
 ::: {layout='[1,1]' layout-valign=center}
@@ -16,6 +24,3 @@
 
 :::
 :::
-
-<!-- Show the background image generated with FLUX -->
-## {background-image="images/ai_vs_hpc.png"}

index.qmd

@@ -4,8 +4,9 @@
 - Solutions for Workflow Orchestration
   - StreamFlow
   - Flyte
-- HPC vs AI/ML Infrastructure
-- File vs Object Storage
+- Infrastructure for Machine Learning Workflows
+  - HPC vs AI/ML
+  - File vs Object Storage
 
 
 {{< include workflows.qmd >}}

streamflow.qmd

@@ -1,9 +1,22 @@
+# StreamFlow
 ## StreamFlow
 
-- Relies on the [Common Workflow Language](https://www.commonwl.org/) (CWL)
-- [BioExcel CoE Building Blocks](https://bioexcel.eu/biobb-new/) is using CWL for interoperable and reproducible biomolecular simulation workflows.
-- StreamFlow connects CWL with HPC
-- Supports Containers (Singularity), Kubernetes and SLURM
+- [StreamFlow](https://streamflow.di.unito.it/documentation/0.2/) is a general framework for workflow orchestration
+- Relies on the [Common Workflow Language](https://www.commonwl.org/) (CWL)\
+  and connects the CWL with a deployment model
+- Workflows can be deployed to different environments:
+  - Containers/Singularity
+  - Slurm
+  - Kubernetes
+
+::: {.callout-note}
+[BioExcel Building Blocks](https://bioexcel.eu/biobb-new/) is a **showpiece** using CWL for biomolecular simulation.
+:::
+
+
+## StreamFlow Architecture
+
+![](images/streamflow-model.png){fig-align="center"}
 
 
 ## Common Workflow Language (CWL)
@@ -25,16 +38,6 @@
 :::
 
 
-## Visualizing CWL
-
-- [Web service for visualization](https://view.commonwl.org/)
-
-
-## StreamFlow Architecture
-
-![](images/streamflow-model.png){fig-align="center"}
-
-
 ## StreamFlow Deployment
 
 ::: {layout='[1,1]'}
@@ -52,14 +55,11 @@
 :::
 
 
-## Supported Features
+## StreamFlow Demo
 
-  - [Parallelization](https://www.commonwl.org/features/#parallelization-and-scale-with-cwl)
-  - [Caching](https://www.commonwl.org/user_guide/topics/troubleshooting.html#run-cwltool-with-cachedir)
-    (vs [Flyte caching](https://docs.flyte.org/en/latest/user_guide/development_lifecycle/caching.html#caching))
-  - Nested workflows
-  - Looping / scattering tasks
-  - Conditional workflows
+- [GitHub repo](https://github.com/BerndDoser/streamflow-example)
 
 
+<!-- ## Visualizing CWL
 
+- [Web service for visualization](https://view.commonwl.org/) -->

styles.css

@@ -1,3 +1,8 @@
+.logo_h1_inline{
+    margin: 0!important;
+    height: 70px;
+}
+
 .logo_h2_inline{
     margin: 0!important;
     height: 45px;

workflows.qmd

@@ -53,12 +53,20 @@ K. L. Polsterer, B. Doser, A. Fehlner and S. Trujillo-Gomez [ADASS (2024)]().
 :::
 
 
-## Example Workflow
+## General Requirements for Workflows
+
+  - Parallelization: Run independent tasks automatically in parallel 
+  - Caching: Avoid recomputing successful tasks
+  - Nesting: Reuse workflows as tasks
+  - Looping: Repeat tasks based on conditions
+  - Scattering: Distribute data to multiple tasks
+  - Conditionals: Branching based on conditions
+
 
 ![](images/flyte-ui_mnist-workflow.png)
 
 
-## Workflow Orchestration Management
+## Options to Generate a Workflow
 
 - General purpose programming languages (e.g Python)
 - Domain specific language (e.g. [Common Workflow Language](https://www.commonwl.org/))