Skip to content

Commit

Permalink
Update previous versions to docusaurus 3.6.
Browse files Browse the repository at this point in the history
Remove 1.24.0 and 1.24.1 versions.

Signed-off-by: Jakub Dardzinski <[email protected]>
  • Loading branch information
JDarDagran committed Nov 7, 2024
1 parent f76e90b commit 20aa696
Show file tree
Hide file tree
Showing 517 changed files with 186 additions and 27,102 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,8 @@ spark.openlineage.transport.headers.X-Some-Extra-Header=abc
spark.openlineage.transport.compression=gzip
```

<details><summary>URL parsing within Spark integration</summary>
<details>
<summary>URL parsing within Spark integration</summary>
<p>

You can supply http parameters using values in url, the parsed `spark.openlineage.*` properties are located in url as follows:
Expand Down
3 changes: 3 additions & 0 deletions versioned_docs/version-1.21.1/client/python.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,9 @@
sidebar_position: 5
---

import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

# Python

## Overview
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ title: Apache Airflow
---

:::caution
This page is about Airflow's external integration that works mainly for Airflow versions <2.7.
This page is about Airflow's external integration that works mainly for Airflow versions \<2.7.
[If you're using Airflow 2.7+, look at native Airflow OpenLineage provider documentation.](https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/index.html) <br /><br />

The ongoing development and enhancements will be focused on the `apache-airflow-providers-openlineage` package,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ title: Exposing Lineage in Airflow Operators
---

:::caution
This page is about Airflow's external integration that works mainly for Airflow versions <2.7.
This page is about Airflow's external integration that works mainly for Airflow versions \<2.7.
[If you're using Airflow 2.7+, look at native Airflow OpenLineage provider documentation.](https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/index.html) <br /><br />

The ongoing development and enhancements will be focused on the `apache-airflow-providers-openlineage` package,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ title: Custom Extractors
---

:::caution
This page is about Airflow's external integration that works mainly for Airflow versions <2.7.
This page is about Airflow's external integration that works mainly for Airflow versions \<2.7.
[If you're using Airflow 2.7+, look at native Airflow OpenLineage provider documentation.](https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/index.html) <br /><br />

The ongoing development and enhancements will be focused on the `apache-airflow-providers-openlineage` package,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ title: Testing Custom Extractors
---

:::caution
This page is about Airflow's external integration that works mainly for Airflow versions <2.7.
This page is about Airflow's external integration that works mainly for Airflow versions \<2.7.
[If you're using Airflow 2.7+, look at native Airflow OpenLineage provider documentation.](https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/index.html) <br /><br />

The ongoing development and enhancements will be focused on the `apache-airflow-providers-openlineage` package,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ title: Job Hierarchy
---

:::caution
This page is about Airflow's external integration that works mainly for Airflow versions <2.7.
This page is about Airflow's external integration that works mainly for Airflow versions \<2.7.
[If you're using Airflow 2.7+, look at native Airflow OpenLineage provider documentation.](https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/index.html) <br /><br />

The ongoing development and enhancements will be focused on the `apache-airflow-providers-openlineage` package,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ title: Manually Annotated Lineage
---

:::caution
This page is about Airflow's external integration that works mainly for Airflow versions <2.7.
This page is about Airflow's external integration that works mainly for Airflow versions \<2.7.
[If you're using Airflow 2.7+, look at native Airflow OpenLineage provider documentation.](https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/index.html) <br /><br />

The ongoing development and enhancements will be focused on the `apache-airflow-providers-openlineage` package,
Expand Down
6 changes: 3 additions & 3 deletions versioned_docs/version-1.21.1/integrations/airflow/older.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ title: Supported Airflow versions
---

:::caution
This page is about Airflow's external integration that works mainly for Airflow versions <2.7.
This page is about Airflow's external integration that works mainly for Airflow versions \<2.7.
[If you're using Airflow 2.7+, look at native Airflow OpenLineage provider documentation.](https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/index.html) <br /><br />

The ongoing development and enhancements will be focused on the `apache-airflow-providers-openlineage` package,
Expand All @@ -16,7 +16,7 @@ while the `openlineage-airflow` will primarily be updated for bug fixes.
##### Airflow 2.7+

This package **should not** be used starting with Airflow 2.7.0 and **can not** be used with Airflow 2.8+.
It was designed as Airflow's external integration that works mainly for Airflow versions <2.7.
It was designed as Airflow's external integration that works mainly for Airflow versions \<2.7.
For Airflow 2.7+ use the native Airflow OpenLineage provider
[package](https://airflow.apache.org/docs/apache-airflow-providers-openlineage) `apache-airflow-providers-openlineage`.

Expand Down Expand Up @@ -44,6 +44,6 @@ openlineage.lineage_backend.OpenLineageBackend

The OpenLineageBackend does not take into account manually configured inlets and outlets.

##### Airflow <2.1
##### Airflow \<2.1

OpenLineage does not work with versions older than Airflow 2.1.
4 changes: 2 additions & 2 deletions versioned_docs/version-1.21.1/integrations/airflow/usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ title: Using the Airflow Integration
---

:::caution
This page is about Airflow's external integration that works mainly for Airflow versions <2.7.
This page is about Airflow's external integration that works mainly for Airflow versions \<2.7.
[If you're using Airflow 2.7+, look at native Airflow OpenLineage provider documentation.](https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/index.html) <br /><br />

The ongoing development and enhancements will be focused on the `apache-airflow-providers-openlineage` package,
Expand All @@ -14,7 +14,7 @@ while the `openlineage-airflow` will primarily be updated for bug fixes. See [al
#### PREREQUISITES

- [Python 3.8](https://www.python.org/downloads)
- [Airflow >= 2.1,<2.8](https://pypi.org/project/apache-airflow)
- [Airflow >= 2.1,\<2.8](https://pypi.org/project/apache-airflow)

To use the OpenLineage Airflow integration, you'll need a running [Airflow instance](https://airflow.apache.org/docs/apache-airflow/stable/start.html). You'll also need an OpenLineage-compatible [backend](https://github.com/OpenLineage/OpenLineage#scope).

Expand Down
10 changes: 5 additions & 5 deletions versioned_docs/version-1.21.1/integrations/flink.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,11 +118,11 @@ and allows all the configuration features present there to be used. The configur

The following parameters can be specified:

| Parameter | Definition | Example |
|------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------|
| openlineage.transport.type | The transport type used for event emit, default type is `console` | http |
| openlineage.facets.disabled | List of facets to disable, enclosed in `[]` (required from 0.21.x) and separated by `;`, default is `[spark_unknown;spark.logicalPlan;]` (currently must contain `;`) | \[some_facet1;some_facet1\] |
| openlineage.job.owners.<ownership-type\> | Specifies ownership of the job. Multiple entries with different types are allowed. Config key name and value are used to create job ownership type and name (available since 1.13). | openlineage.job.owners.team="Some Team" |
| Parameter | Definition | Example |
|-------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------|
| openlineage.transport.type | The transport type used for event emit, default type is `console` | http |
| openlineage.facets.disabled | List of facets to disable, enclosed in `[]` (required from 0.21.x) and separated by `;`, default is `[spark_unknown;spark.logicalPlan;]` (currently must contain `;`) | \[some_facet1;some_facet1\] |
| openlineage.job.owners.\<ownership-type\> | Specifies ownership of the job. Multiple entries with different types are allowed. Config key name and value are used to create job ownership type and name (available since 1.13). | openlineage.job.owners.team="Some Team" |

## Transports

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,4 +22,4 @@ The following parameters can be specified:
| spark.openlineage.jobName.appendDatasetName | Decides whether output dataset name should be appended to job name. By default `true`. | false |
| spark.openlineage.jobName.replaceDotWithUnderscore | Replaces dots in job name with underscore. Can be used to mimic legacy behaviour on Databricks platform. By default `false`. | false |
| spark.openlineage.debugFacet | Determines whether debug facet shall be generated and included within the event. Set `enabled` to turn it on. By default, facet is disabled. | enabled |
| spark.openlineage.job.owners.<ownership-type\> | Specifies ownership of the job. Multiple entries with different types are allowed. Config key name and value are used to create job ownership type and name (available since 1.13). | spark.openlineage.job.owners.team="Some Team" |
| spark.openlineage.job.owners.\<ownership-type\> | Specifies ownership of the job. Multiple entries with different types are allowed. Config key name and value are used to create job ownership type and name (available since 1.13). | spark.openlineage.job.owners.team="Some Team" |
Loading

0 comments on commit 20aa696

Please sign in to comment.