From 8a20d7d8fb7213d3172c23f51b6813313bcba984 Mon Sep 17 00:00:00 2001 From: Meira Josephy <144697924+mjosephym@users.noreply.github.com> Date: Sun, 1 Dec 2024 17:06:30 +0200 Subject: [PATCH 1/7] config, includes --- data-explorer/fluent-bit.md | 39 ++---- .../includes/cross-repo/fluent-bit-2.md | 130 +++++++++++++++--- .../includes/cross-repo/fluent-bit-3.md | 106 ++++++++++++++ .../includes/cross-repo/fluent-bit.md | 64 +-------- 4 files changed, 224 insertions(+), 115 deletions(-) create mode 100644 data-explorer/includes/cross-repo/fluent-bit-3.md diff --git a/data-explorer/fluent-bit.md b/data-explorer/fluent-bit.md index 130c649ad9..c61b7c0a8c 100644 --- a/data-explorer/fluent-bit.md +++ b/data-explorer/fluent-bit.md @@ -3,21 +3,12 @@ title: Ingest data with Fluent Bit into Azure Data Explorer description: Learn how to ingest (load) data into Azure Data Explorer from Fluent Bit. ms.reviewer: ramacg ms.topic: how-to -ms.date: 06/27/2024 +ms.date: 12/01/2024 --- # Ingest data with Fluent Bit into Azure Data Explorer -[Fluent Bit](https://github.com/fluent/fluent-bit/tree/master) is an open-source agent that collects logs, metrics, and traces from various sources. It allows you to filter, modify, and aggregate event data before sending it to storage. Azure Data Explorer is a fast and highly scalable data exploration service for log and telemetry data. This article guides you through the process of using Fluent Bit to send data to Azure Data Explorer. - -In this article, you'll learn how to: - -> [!div class="checklist"] -> -> * [Create a table to store your logs](#create-a-table-to-store-your-logs) -> * [Register a Microsoft Entra app with permissions to ingest data](#register-a-microsoft-entra-app-with-permissions-to-ingest-data) -> * [Configure Fluent Bit to send logs to your table](#configure-fluent-bit-to-send-logs-to-your-table) -> * [Verify that data has landed in your table](#verify-that-data-has-landed-in-your-table) +[!INCLUDE [fluent-bit](includes/cross-repo/fluent-bit.md)] For a complete list of data connectors, see [Data connectors overview](integrate-overview.md). @@ -25,31 +16,17 @@ For a complete list of data connectors, see [Data connectors overview](integrate * [Fluent Bit](https://docs.fluentbit.io/manual/installation/getting-started-with-fluent-bit). * An Azure Data Explorer cluster and database. [Create a cluster and database](create-cluster-and-database.md). +* A query environment. For more information, see [Query integrations overview](integrate-query-overview.md). +* Your Kusto cluster URI for the *Ingestion_endpoint* value in the format *https://ingest-\.\.kusto.windows.net*. For more information, see [Add a cluster connection](add-cluster-connection.md#add-a-cluster-connection). You can use any of the available [Query tools](integrate-query-overview.md) for your query environment. -[!INCLUDE [fluent-bit](includes/cross-repo/fluent-bit.md)] - -## Configure Fluent Bit to send logs to your table - -To configure Fluent Bit to send logs to your Azure Data Explorer table, create a [classic mode](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/classic-mode/configuration-file) or [YAML mode](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/yaml/configuration-file) configuration file with the following output properties: - -| Field | Description | -| --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Name | `azure_kusto` | -| Match | A pattern to match against the tags of incoming records. It's case-sensitive and supports the star (`*`) character as a wildcard. | -| Tenant_Id | **Directory (tenant) ID** from [Register a Microsoft Entra app with permissions to ingest data](#register-a-microsoft-entra-app-with-permissions-to-ingest-data). | -| Client_Id | **Application (client) ID** from [Register a Microsoft Entra app with permissions to ingest data](#register-a-microsoft-entra-app-with-permissions-to-ingest-data). | -| Client_Secret | The client secret key value [Register a Microsoft Entra app with permissions to ingest data](#register-a-microsoft-entra-app-with-permissions-to-ingest-data). | -| Ingestion_Endpoint | Use the **Data Ingestion URI** found in the [Azure portal](https://ms.portal.azure.com/) under your cluster overview. | -| Database_Name | The name of the database that contains your logs table. | -| Table_Name | The name of the table from [Create a table to store your logs](#create-a-table-to-store-your-logs). | -| Ingestion_Mapping_Reference | The name of the ingestion mapping from [Create a table](#create-a-table-to-store-your-logs). If you didn't create an ingestion mapping, remove the property from the configuration file. | - -To see an example configuration file, select the relevant tab: - [!INCLUDE [fluent-bit-2](includes/cross-repo/fluent-bit-2.md)] + + ## Related content +* [Data integrations overview](integrate-data-overview.md) +* [Kusto Query Language (KQL) overview](/kusto/query/) * [Write queries](/kusto/query/tutorials/learn-common-operators?view=azure-data-explorer&preserve-view=true) diff --git a/data-explorer/includes/cross-repo/fluent-bit-2.md b/data-explorer/includes/cross-repo/fluent-bit-2.md index 18cc1fbc98..06a430c0b3 100644 --- a/data-explorer/includes/cross-repo/fluent-bit-2.md +++ b/data-explorer/includes/cross-repo/fluent-bit-2.md @@ -1,7 +1,87 @@ --- ms.topic: include -ms.date: 06/27/2024 +ms.date: 12/01/2024 --- +## Create a Microsoft Entra service principal + +The Microsoft Entra service principal can be created through the [Azure portal](/azure/active-directory/develop/howto-create-service-principal-portal) or programmatically, as in the following example. + +This service principal is the identity used by the connector to write data to your table in Kusto. You grant permissions for this service principal to access Kusto resources. + +[!INCLUDE [entra-service-principal](../entra-service-principal.md)] + +## Create a target table + +Fluent Bit forwards logs in JSON format with three properties: `log` ([dynamic](/azure/data-explorer/kusto/query/scalar-data-types/dynamic)), `tag` ([string](/azure/data-explorer/kusto/query/scalar-data-types/string)), and `timestamp` ([datetime](/azure/data-explorer/kusto/query/scalar-data-types/datetime)). + +You can create a table with columns for each of these properties. Alternatively, if you have structured logs, you can create a table with log properties mapped to custom columns. To learn more, select the relevant tab. + +### [Default schema](#tab/default) + +To create a table for incoming logs from Fluent Bit: + +1. Browse to your query environment. +1. Select the database where you'd like to create the table. +1. Run the following [`.create table` command](/azure/data-explorer/kusto/management/create-table-command): + + ```kusto + .create table FluentBitLogs (log:dynamic, tag:string, timestamp:datetime) + ``` + + The incoming JSON properties are automatically mapped into the correct column. + +### [Custom schema](#tab/custom) + +To create a table for incoming structured logs from Fluent Bit: + +1. Browse to your query environment. +1. Select the database where you'd like to create the table. +1. Run the [`.create table` command](/azure/data-explorer/kusto/management/create-table-command). For example, if your logs contain three fields named `myString`, `myInteger`, and `myDynamic`, you can create a table with the following schema: + + ```kusto + .create table FluentBitLogs (myString:string, myInteger:int, myDynamic: dynamic, timestamp:datetime) + ``` + +1. Create a [JSON mapping](/azure/data-explorer/kusto/management/mappings) to map log properties to the appropriate columns. The following command creates a mapping based on the example in the previous step: + + ```kusto + .create-or-alter table FluentBitLogs ingestion json mapping "LogMapping" + ```[ + {"column" : "myString", "datatype" : "string", "Properties":{"Path":"$.log.myString"}}, + {"column" : "myInteger", "datatype" : "int", "Properties":{"Path":"$.log.myInteger"}}, + {"column" : "myDynamic", "datatype" : "dynamic", "Properties":{"Path":"$.log.myInteger"}}, + {"column" : "timestamp", "datatype" : "datetime", "Properties":{"Path":"$.timestamp"}} + ]``` + ``` + +--- + +## Grant permissions to the service principal + +1. Grant the service principal from [Create a Microsoft Entra service principal](#create-a-microsoft-entra-service-principal) [database ingestor](/azure/data-explorer/kusto/access-control/role-based-access-control) role permissions to work with the database. For more information, see [Examples](/azure/data-explorer/kusto/management/manage-database-security-roles). Replace the placeholder *DatabaseName* with the name of the target database and *ApplicationID* with the `AppId` value you saved when creating a Microsoft Entra service principal. + + ```kusto + .add database ingestors ('aadapp=;') + ``` + +## Configure Fluent Bit to send logs to your table + +To configure Fluent Bit to send logs to your Azure Data Explorer table, create a [classic mode](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/classic-mode/configuration-file) or [YAML mode](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/yaml/configuration-file) configuration file with the following output properties: + +| Field | Description | +|--|--| +| Name | `azure_kusto` | +| Match | A pattern to match against the tags of incoming records. It's case-sensitive and supports the star (`*`) character as a wildcard. | +| tenant_id | **Tenant ID** from [Create a Microsoft Entra service principal](#create-a-microsoft-entra-service-principal). | +| client_id | **Application ID** from [Create a Microsoft Entra service principal](#create-a-microsoft-entra-service-principal). | +| client_secret | The client secret key value (password) from [Create a Microsoft Entra service principal](#create-a-microsoft-entra-service-principal). | +| Ingestion_Endpoint | Enter the value as described for [Ingestion_Endpoint](#ingestion-uri). | +| Database_Name | The name of the database that contains your logs table. | +| Table_Name | The name of the table from [Create a target table](#create-a-target-table). | +| Ingestion_Mapping_Reference | The name of the ingestion mapping from [Create a target table](#create-a-target-table). If you didn't create an ingestion mapping, remove the property from the configuration file. | + +To see an example configuration file, select the relevant tab: + ### [Classic mode](#tab/classic) ```txt @@ -23,14 +103,18 @@ ms.date: 06/27/2024 Refresh_Interval 10 [OUTPUT] - Name azure_kusto - Match * - Tenant_Id azure-tenant-id - Client_Id azure-client-id - Client_Secret azure-client-secret - Ingestion_Endpoint azure-data-explorer-ingestion-endpoint - Database_Name azure-data-explorer-database-name - Table_Name azure-data-explorer-table-name + match * + name azure_kusto + tenant_id + client_id + client_secret + ingestion_endpoint + database_name + table_name + ingestion_mapping_reference + ingestion_endpoint_connect_timeout + compression_enabled + ingestion_resources_refresh_interval ``` ### [YAML mode](#tab/yaml) @@ -69,23 +153,25 @@ config: outputs: | [OUTPUT] - Name azure_kusto - Match * - Tenant_Id azure-tenant-id - Client_Id azure-client-id - Client_Secret azure-client-secret - Ingestion_Endpoint azure-data-explorer-ingestion-endpoint - Database_Name azure-data-explorer-database-name - Table_Name azure-data-explorer-table-name + match * + name azure_kusto + tenant_id + client_id + client_secret + ingestion_endpoint + database_name + table_name + ingestion_mapping_reference + ingestion_endpoint_connect_timeout + compression_enabled + ingestion_resources_refresh_interval ``` --- -## Verify that data has landed in your table +## Confirm data ingestion -Once the configuration is complete, logs should arrive in your table. - -1. To verify that logs are ingested, run the following query: +1. Once data arrives in the table, confirm the transfer of data, by checking the row count: ```Kusto FluentBitLogs @@ -97,4 +183,4 @@ Once the configuration is complete, logs should arrive in your table. ```Kusto FluentBitLogs | take 100 - ``` \ No newline at end of file + ``` diff --git a/data-explorer/includes/cross-repo/fluent-bit-3.md b/data-explorer/includes/cross-repo/fluent-bit-3.md new file mode 100644 index 0000000000..a1aab81d8b --- /dev/null +++ b/data-explorer/includes/cross-repo/fluent-bit-3.md @@ -0,0 +1,106 @@ +--- +ms.topic: include +ms.date: 12/01/2024 +--- +### [Classic mode](#tab/classic) + +```txt +[SERVICE] + Daemon Off + Flush 1 + Log_Level trace + HTTP_Server On + HTTP_Listen 0.0.0.0 + HTTP_Port 2020 + Health_Check On + +[INPUT] + Name tail + Path /var/log/containers/*.log + Tag kube.* + Mem_Buf_Limit 1MB + Skip_Long_Lines On + Refresh_Interval 10 + +[OUTPUT] + match * + name azure_kusto + tenant_id + client_id + client_secret + ingestion_endpoint + database_name + table_name + ingestion_mapping_reference + ingestion_endpoint_connect_timeout + compression_enabled + ingestion_resources_refresh_interval +``` + +### [YAML mode](#tab/yaml) + +```yaml +config: + service: | + [SERVICE] + Daemon Off + Flush 1 + Log_Level trace + HTTP_Server On + HTTP_Listen 0.0.0.0 + HTTP_Port 2020 + Health_Check On + + inputs: | + [INPUT] + Name tail + Path /var/log/containers/*.log + multiline.parser docker, cri + Tag kube.* + Mem_Buf_Limit 1MB + Skip_Long_Lines On + Refresh_Interval 10 + + filters: | + [FILTER] + Name kubernetes + Match kube.* + Merge_Log On + Merge_Log_key log_processed + K8S-Logging.Parser On + K8S-Logging.Exclude Off + + + outputs: | + [OUTPUT] + match * + name azure_kusto + tenant_id + client_id + client_secret + ingestion_endpoint + database_name + table_name + ingestion_mapping_reference + ingestion_endpoint_connect_timeout + compression_enabled + ingestion_resources_refresh_interval +``` + +--- + +## Confirm data ingestion + +1. Once data arrives in the table, confirm the transfer of data, by checking the row count: + + ```Kusto + FluentBitLogs + | count + ``` + +1. To view a sample of log data, run the following query: + + ```Kusto + FluentBitLogs + | take 100 + ``` \ No newline at end of file diff --git a/data-explorer/includes/cross-repo/fluent-bit.md b/data-explorer/includes/cross-repo/fluent-bit.md index 62117a3265..bf9e23d353 100644 --- a/data-explorer/includes/cross-repo/fluent-bit.md +++ b/data-explorer/includes/cross-repo/fluent-bit.md @@ -4,66 +4,6 @@ ms.date: 06/27/2024 --- ## Create a table to store your logs -Fluent Bit forwards logs in JSON format with three properties: `log` ([dynamic](/azure/data-explorer/kusto/query/scalar-data-types/dynamic)), `tag` ([string](/azure/data-explorer/kusto/query/scalar-data-types/string)), and `timestamp` ([datetime](/azure/data-explorer/kusto/query/scalar-data-types/datetime)). - -You can create a table with columns for each of these properties. Alternatively, if you have structured logs, you can create a table with log properties mapped to custom columns. To learn more, select the relevant tab. - -### [Default schema](#tab/default) - -To create a table for incoming logs from Fluent Bit: - -1. Browse to your query environment. -1. Select the database where you'd like to create the table. -1. Run the following [`.create table` command](/azure/data-explorer/kusto/management/create-table-command): - - ```kusto - .create table FluentBitLogs (log:dynamic, tag:string, timestamp:datetime) - ``` - - The incoming JSON properties are automatically mapped into the correct column. - -### [Custom schema](#tab/custom) - -To create a table for incoming structured logs from Fluent Bit: - -1. Browse to your query environment. -1. Select the database where you'd like to create the table. -1. Run the [`.create table` command](/azure/data-explorer/kusto/management/create-table-command). For example, if your logs contain three fields named `myString`, `myInteger`, and `myDynamic`, you can create a table with the following schema: - - ```kusto - .create table FluentBitLogs (myString:string, myInteger:int, myDynamic: dynamic, timestamp:datetime) - ``` - -1. Create a [JSON mapping](/azure/data-explorer/kusto/management/mappings) to map log properties to the appropriate columns. The following command creates a mapping based on the example in the previous step: - - ```kusto - .create-or-alter table FluentBitLogs ingestion json mapping "LogMapping" - ```[ - {"column" : "myString", "datatype" : "string", "Properties":{"Path":"$.log.myString"}}, - {"column" : "myInteger", "datatype" : "int", "Properties":{"Path":"$.log.myInteger"}}, - {"column" : "myDynamic", "datatype" : "dynamic", "Properties":{"Path":"$.log.myInteger"}}, - {"column" : "timestamp", "datatype" : "datetime", "Properties":{"Path":"$.timestamp"}} - ]``` - ``` - ---- - -## Register a Microsoft Entra app with permissions to ingest data - -The Microsoft Entra service principal can be created through the [Azure portal](/azure/active-directory/develop/howto-create-service-principal-portal) or programatically, as in the following example. - -This service principal is the identity used by the connector to write data your table in Kusto. You'll later grant permissions for this service principal to access Kusto resources. - -[!INCLUDE [entra-service-principal](../entra-service-principal.md)] - - -## Grant permissions to the service principal - -Run the following command, replacing `` with the name of the database: - -```kusto -.add database MyDatabase ingestors ('aadapp=;') -``` - -This command grants the application permissions to ingest data into your table. For more information, see [role-based access control](/azure/data-explorer/kusto/access-control/role-based-access-control). +[Fluent Bit](https://github.com/fluent/fluent-bit/tree/master) is an open-source agent that collects logs, metrics, and traces from various sources. It allows you to filter, modify, and aggregate event data before sending it to storage. Azure Data Explorer is a fast and highly scalable data exploration service for log and telemetry data. This article guides you through the process of using Fluent Bit to send data to your KQL database. +This article shows how to ingest data with Fluent Bit. From 3a9ac7bef14129fc814c83420ef42cd4f241ee7d Mon Sep 17 00:00:00 2001 From: Meira Josephy <144697924+mjosephym@users.noreply.github.com> Date: Mon, 2 Dec 2024 14:16:08 +0200 Subject: [PATCH 2/7] edits --- data-explorer/fluent-bit.md | 4 +- .../includes/cross-repo/fluent-bit-2.md | 66 +++++++++++-------- .../includes/cross-repo/fluent-bit.md | 3 +- 3 files changed, 39 insertions(+), 34 deletions(-) diff --git a/data-explorer/fluent-bit.md b/data-explorer/fluent-bit.md index c61b7c0a8c..16b58e606d 100644 --- a/data-explorer/fluent-bit.md +++ b/data-explorer/fluent-bit.md @@ -3,7 +3,7 @@ title: Ingest data with Fluent Bit into Azure Data Explorer description: Learn how to ingest (load) data into Azure Data Explorer from Fluent Bit. ms.reviewer: ramacg ms.topic: how-to -ms.date: 12/01/2024 +ms.date: 12/02/2024 --- # Ingest data with Fluent Bit into Azure Data Explorer @@ -19,8 +19,6 @@ For a complete list of data connectors, see [Data connectors overview](integrate * A query environment. For more information, see [Query integrations overview](integrate-query-overview.md). * Your Kusto cluster URI for the *Ingestion_endpoint* value in the format *https://ingest-\.\.kusto.windows.net*. For more information, see [Add a cluster connection](add-cluster-connection.md#add-a-cluster-connection). -You can use any of the available [Query tools](integrate-query-overview.md) for your query environment. - [!INCLUDE [fluent-bit-2](includes/cross-repo/fluent-bit-2.md)] diff --git a/data-explorer/includes/cross-repo/fluent-bit-2.md b/data-explorer/includes/cross-repo/fluent-bit-2.md index 06a430c0b3..7575170651 100644 --- a/data-explorer/includes/cross-repo/fluent-bit-2.md +++ b/data-explorer/includes/cross-repo/fluent-bit-2.md @@ -1,6 +1,6 @@ --- ms.topic: include -ms.date: 12/01/2024 +ms.date: 12/02/2024 --- ## Create a Microsoft Entra service principal @@ -58,7 +58,7 @@ To create a table for incoming structured logs from Fluent Bit: ## Grant permissions to the service principal -1. Grant the service principal from [Create a Microsoft Entra service principal](#create-a-microsoft-entra-service-principal) [database ingestor](/azure/data-explorer/kusto/access-control/role-based-access-control) role permissions to work with the database. For more information, see [Examples](/azure/data-explorer/kusto/management/manage-database-security-roles). Replace the placeholder *DatabaseName* with the name of the target database and *ApplicationID* with the `AppId` value you saved when creating a Microsoft Entra service principal. +Grant the service principal from [Create a Microsoft Entra service principal](#create-a-microsoft-entra-service-principal) [database ingestor](/azure/data-explorer/kusto/access-control/role-based-access-control) role permissions to work with the database. For more information, see [Examples](/azure/data-explorer/kusto/management/manage-database-security-roles). Replace the placeholder *DatabaseName* with the name of the target database and *ApplicationID* with the `AppId` value you saved when creating a Microsoft Entra service principal. ```kusto .add database ingestors ('aadapp=;') @@ -66,19 +66,27 @@ To create a table for incoming structured logs from Fluent Bit: ## Configure Fluent Bit to send logs to your table -To configure Fluent Bit to send logs to your Azure Data Explorer table, create a [classic mode](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/classic-mode/configuration-file) or [YAML mode](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/yaml/configuration-file) configuration file with the following output properties: +To configure Fluent Bit to send logs to your table in Kusto, create a [classic mode](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/classic-mode/configuration-file) or [YAML mode](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/yaml/configuration-file) configuration file with the following output properties: | Field | Description | |--|--| | Name | `azure_kusto` | | Match | A pattern to match against the tags of incoming records. It's case-sensitive and supports the star (`*`) character as a wildcard. | -| tenant_id | **Tenant ID** from [Create a Microsoft Entra service principal](#create-a-microsoft-entra-service-principal). | -| client_id | **Application ID** from [Create a Microsoft Entra service principal](#create-a-microsoft-entra-service-principal). | +| tenant_id | The tenant ID from [Create a Microsoft Entra service principal](#create-a-microsoft-entra-service-principal). | +| client_id | The application ID from [Create a Microsoft Entra service principal](#create-a-microsoft-entra-service-principal). | | client_secret | The client secret key value (password) from [Create a Microsoft Entra service principal](#create-a-microsoft-entra-service-principal). | -| Ingestion_Endpoint | Enter the value as described for [Ingestion_Endpoint](#ingestion-uri). | -| Database_Name | The name of the database that contains your logs table. | -| Table_Name | The name of the table from [Create a target table](#create-a-target-table). | -| Ingestion_Mapping_Reference | The name of the ingestion mapping from [Create a target table](#create-a-target-table). If you didn't create an ingestion mapping, remove the property from the configuration file. | +| ingestion_endpoint | Enter the value as described for [Ingestion_Endpoint](#ingestion-uri). | +| database_name | The name of the database that contains your logs table. | +| table_name | The name of the table from [Create a target table](#create-a-target-table). | +| ingestion_mapping_reference | The name of the ingestion mapping from [Create a target table](#create-a-target-table). If you didn't create an ingestion mapping, remove the property from the configuration file. | +| log_key | Key name of the log content. For instance, `log`. | +| tag_key | The key name of tag. Ignored if `include_tag_key` is false. | +| include_time_key | A timestamp is appended to output, if enabled. Uses the `time_key` property. | +| time_key | The key name for the timestamp in the log records. Ignored if `include_time_key` false. | +| ingestion_endpoint_connect_timeout| The connection timeout of various Kusto endpoints in seconds. | +|compression_enabled | Sends compressed HTTP payload (gzip) to Kusto, if enabled. | +|ingestion_resources_refresh_interval | The ingestion resources refresh interval of Kusto endpoint in seconds. | +| workers | The number of [workers](https://docs.fluentbit.io/manual/administration/multithreading#outputs) to perform flush operations for this output. | To see an example configuration file, select the relevant tab: @@ -105,16 +113,16 @@ To see an example configuration file, select the relevant tab: [OUTPUT] match * name azure_kusto - tenant_id - client_id - client_secret - ingestion_endpoint - database_name - table_name - ingestion_mapping_reference - ingestion_endpoint_connect_timeout - compression_enabled - ingestion_resources_refresh_interval + tenant_id + client_id + client_secret + ingestion_endpoint + database_name + table_name + ingestion_mapping_reference + ingestion_endpoint_connect_timeout + compression_enabled + ingestion_resources_refresh_interval ``` ### [YAML mode](#tab/yaml) @@ -155,16 +163,16 @@ config: [OUTPUT] match * name azure_kusto - tenant_id - client_id - client_secret - ingestion_endpoint - database_name - table_name - ingestion_mapping_reference - ingestion_endpoint_connect_timeout - compression_enabled - ingestion_resources_refresh_interval + tenant_id + client_id + client_secret + ingestion_endpoint + database_name + table_name + ingestion_mapping_reference + ingestion_endpoint_connect_timeout + compression_enabled + ingestion_resources_refresh_interval ``` --- diff --git a/data-explorer/includes/cross-repo/fluent-bit.md b/data-explorer/includes/cross-repo/fluent-bit.md index bf9e23d353..1cc02ed6df 100644 --- a/data-explorer/includes/cross-repo/fluent-bit.md +++ b/data-explorer/includes/cross-repo/fluent-bit.md @@ -2,8 +2,7 @@ ms.topic: include ms.date: 06/27/2024 --- -## Create a table to store your logs -[Fluent Bit](https://github.com/fluent/fluent-bit/tree/master) is an open-source agent that collects logs, metrics, and traces from various sources. It allows you to filter, modify, and aggregate event data before sending it to storage. Azure Data Explorer is a fast and highly scalable data exploration service for log and telemetry data. This article guides you through the process of using Fluent Bit to send data to your KQL database. +[Fluent Bit](https://github.com/fluent/fluent-bit/tree/master) is an open-source agent that collects logs, metrics, and traces from various sources. It allows you to filter, modify, and aggregate event data before sending it to storage. This article guides you through the process of using Fluent Bit to send data to your KQL database. This article shows how to ingest data with Fluent Bit. From 5c5d9bb7b9d3cdd8d142574b3a604616ae602b8e Mon Sep 17 00:00:00 2001 From: Meira Josephy <144697924+mjosephym@users.noreply.github.com> Date: Tue, 3 Dec 2024 14:36:59 +0200 Subject: [PATCH 3/7] changes --- .../includes/cross-repo/fluent-bit-2.md | 26 ++++++++++++++++--- 1 file changed, 23 insertions(+), 3 deletions(-) diff --git a/data-explorer/includes/cross-repo/fluent-bit-2.md b/data-explorer/includes/cross-repo/fluent-bit-2.md index 7575170651..6d907e7fb3 100644 --- a/data-explorer/includes/cross-repo/fluent-bit-2.md +++ b/data-explorer/includes/cross-repo/fluent-bit-2.md @@ -1,6 +1,6 @@ --- ms.topic: include -ms.date: 12/02/2024 +ms.date: 12/03/2024 --- ## Create a Microsoft Entra service principal @@ -68,8 +68,8 @@ Grant the service principal from [Create a Microsoft Entra service principal](#c To configure Fluent Bit to send logs to your table in Kusto, create a [classic mode](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/classic-mode/configuration-file) or [YAML mode](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/yaml/configuration-file) configuration file with the following output properties: -| Field | Description | -|--|--| +| Field | Description | Required | Default | +|--|--|--|--| | Name | `azure_kusto` | | Match | A pattern to match against the tags of incoming records. It's case-sensitive and supports the star (`*`) character as a wildcard. | | tenant_id | The tenant ID from [Create a Microsoft Entra service principal](#create-a-microsoft-entra-service-principal). | @@ -88,6 +88,26 @@ To configure Fluent Bit to send logs to your table in Kusto, create a [classic m |ingestion_resources_refresh_interval | The ingestion resources refresh interval of Kusto endpoint in seconds. | | workers | The number of [workers](https://docs.fluentbit.io/manual/administration/multithreading#outputs) to perform flush operations for this output. | +| Field | Description | Required | Default | +|--|--|--|--| +| Name | The pipeline name. | | `azure_kusto`| +|--|--|--|--| +| tenant_id | The tenant ID from [Create a Microsoft Entra service principal](#create-a-microsoft-entra-service-principal). | :heavy_check_mark: | | +| client_id | The application ID from [Create a Microsoft Entra service principal](#create-a-microsoft-entra-service-principal). | :heavy_check_mark: | | +| client_secret | The client secret key value (password) from [Create a Microsoft Entra service principal](#create-a-microsoft-entra-service-principal). | :heavy_check_mark: | | +| ingestion_endpoint | Enter the value as described for [Ingestion_Endpoint](#ingestion-uri). | :heavy_check_mark: | | +| database_name | The name of the database that contains your logs table. | :heavy_check_mark: | | +| table_name | The name of the table from [Create a target table](#create-a-target-table). | :heavy_check_mark: | | +| ingestion_mapping_reference | The name of the ingestion mapping from [Create a target table](#create-a-target-table). If you didn't create an ingestion mapping, remove the property from the configuration file. | | | +| log_key | Key name of the log content. For instance, `log`. | | `log` | +| tag_key | The key name of tag. Ignored if `include_tag_key` is false. | | `tag` | +| include_time_key | A timestamp is appended to output, if enabled. Uses the `time_key` property. | | `true` | +| time_key | The key name for the timestamp in the log records. Ignored if `include_time_key` false. | | `timestamp` | +| ingestion_endpoint_connect_timeout The connection timeout of various Kusto endpoints in seconds. | | | `60s` | +| compression_enabled | Sends compressed HTTP payload (gzip) to Kusto, if enabled. | | `true` | +| ingestion_resources_refresh_interval | The ingestion resources refresh interval of Kusto endpoint in seconds. | | `3600` | +| workers | The number of [workers](https://docs.fluentbit.io/manual/administration/multithreading#outputs) to perform flush operations for this output. | | `0` | + To see an example configuration file, select the relevant tab: ### [Classic mode](#tab/classic) From 17d0746e9bc8dab0ab0cb49b147b462a919814a5 Mon Sep 17 00:00:00 2001 From: Meira Josephy <144697924+mjosephym@users.noreply.github.com> Date: Tue, 3 Dec 2024 14:57:17 +0200 Subject: [PATCH 4/7] fix --- data-explorer/includes/cross-repo/fluent-bit-2.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/data-explorer/includes/cross-repo/fluent-bit-2.md b/data-explorer/includes/cross-repo/fluent-bit-2.md index 6d907e7fb3..d589f715e9 100644 --- a/data-explorer/includes/cross-repo/fluent-bit-2.md +++ b/data-explorer/includes/cross-repo/fluent-bit-2.md @@ -60,9 +60,9 @@ To create a table for incoming structured logs from Fluent Bit: Grant the service principal from [Create a Microsoft Entra service principal](#create-a-microsoft-entra-service-principal) [database ingestor](/azure/data-explorer/kusto/access-control/role-based-access-control) role permissions to work with the database. For more information, see [Examples](/azure/data-explorer/kusto/management/manage-database-security-roles). Replace the placeholder *DatabaseName* with the name of the target database and *ApplicationID* with the `AppId` value you saved when creating a Microsoft Entra service principal. - ```kusto - .add database ingestors ('aadapp=;') - ``` +```kusto +.add database ingestors ('aadapp=;') +``` ## Configure Fluent Bit to send logs to your table From 190351ae7208d99c652ea77929ec9719560235d0 Mon Sep 17 00:00:00 2001 From: Meira Josephy <144697924+mjosephym@users.noreply.github.com> Date: Tue, 3 Dec 2024 15:31:33 +0200 Subject: [PATCH 5/7] edit --- .../includes/cross-repo/fluent-bit-2.md | 20 ------------------- 1 file changed, 20 deletions(-) diff --git a/data-explorer/includes/cross-repo/fluent-bit-2.md b/data-explorer/includes/cross-repo/fluent-bit-2.md index d589f715e9..1b216d278d 100644 --- a/data-explorer/includes/cross-repo/fluent-bit-2.md +++ b/data-explorer/includes/cross-repo/fluent-bit-2.md @@ -68,26 +68,6 @@ Grant the service principal from [Create a Microsoft Entra service principal](#c To configure Fluent Bit to send logs to your table in Kusto, create a [classic mode](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/classic-mode/configuration-file) or [YAML mode](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/yaml/configuration-file) configuration file with the following output properties: -| Field | Description | Required | Default | -|--|--|--|--| -| Name | `azure_kusto` | -| Match | A pattern to match against the tags of incoming records. It's case-sensitive and supports the star (`*`) character as a wildcard. | -| tenant_id | The tenant ID from [Create a Microsoft Entra service principal](#create-a-microsoft-entra-service-principal). | -| client_id | The application ID from [Create a Microsoft Entra service principal](#create-a-microsoft-entra-service-principal). | -| client_secret | The client secret key value (password) from [Create a Microsoft Entra service principal](#create-a-microsoft-entra-service-principal). | -| ingestion_endpoint | Enter the value as described for [Ingestion_Endpoint](#ingestion-uri). | -| database_name | The name of the database that contains your logs table. | -| table_name | The name of the table from [Create a target table](#create-a-target-table). | -| ingestion_mapping_reference | The name of the ingestion mapping from [Create a target table](#create-a-target-table). If you didn't create an ingestion mapping, remove the property from the configuration file. | -| log_key | Key name of the log content. For instance, `log`. | -| tag_key | The key name of tag. Ignored if `include_tag_key` is false. | -| include_time_key | A timestamp is appended to output, if enabled. Uses the `time_key` property. | -| time_key | The key name for the timestamp in the log records. Ignored if `include_time_key` false. | -| ingestion_endpoint_connect_timeout| The connection timeout of various Kusto endpoints in seconds. | -|compression_enabled | Sends compressed HTTP payload (gzip) to Kusto, if enabled. | -|ingestion_resources_refresh_interval | The ingestion resources refresh interval of Kusto endpoint in seconds. | -| workers | The number of [workers](https://docs.fluentbit.io/manual/administration/multithreading#outputs) to perform flush operations for this output. | - | Field | Description | Required | Default | |--|--|--|--| | Name | The pipeline name. | | `azure_kusto`| From 9e1f2a4b8941206321b64fc918e35722c19ba234 Mon Sep 17 00:00:00 2001 From: Meira Josephy <144697924+mjosephym@users.noreply.github.com> Date: Tue, 3 Dec 2024 15:50:52 +0200 Subject: [PATCH 6/7] edit --- data-explorer/includes/cross-repo/fluent-bit-2.md | 1 - 1 file changed, 1 deletion(-) diff --git a/data-explorer/includes/cross-repo/fluent-bit-2.md b/data-explorer/includes/cross-repo/fluent-bit-2.md index 1b216d278d..dd3e0fdd7a 100644 --- a/data-explorer/includes/cross-repo/fluent-bit-2.md +++ b/data-explorer/includes/cross-repo/fluent-bit-2.md @@ -71,7 +71,6 @@ To configure Fluent Bit to send logs to your table in Kusto, create a [classic m | Field | Description | Required | Default | |--|--|--|--| | Name | The pipeline name. | | `azure_kusto`| -|--|--|--|--| | tenant_id | The tenant ID from [Create a Microsoft Entra service principal](#create-a-microsoft-entra-service-principal). | :heavy_check_mark: | | | client_id | The application ID from [Create a Microsoft Entra service principal](#create-a-microsoft-entra-service-principal). | :heavy_check_mark: | | | client_secret | The client secret key value (password) from [Create a Microsoft Entra service principal](#create-a-microsoft-entra-service-principal). | :heavy_check_mark: | | From c6da38011ebaaf2f2df7fd5b3d7a97e22220cada Mon Sep 17 00:00:00 2001 From: Meira Josephy <144697924+mjosephym@users.noreply.github.com> Date: Tue, 3 Dec 2024 17:10:08 +0200 Subject: [PATCH 7/7] edit --- data-explorer/includes/cross-repo/fluent-bit-2.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data-explorer/includes/cross-repo/fluent-bit-2.md b/data-explorer/includes/cross-repo/fluent-bit-2.md index dd3e0fdd7a..99db186c95 100644 --- a/data-explorer/includes/cross-repo/fluent-bit-2.md +++ b/data-explorer/includes/cross-repo/fluent-bit-2.md @@ -82,7 +82,7 @@ To configure Fluent Bit to send logs to your table in Kusto, create a [classic m | tag_key | The key name of tag. Ignored if `include_tag_key` is false. | | `tag` | | include_time_key | A timestamp is appended to output, if enabled. Uses the `time_key` property. | | `true` | | time_key | The key name for the timestamp in the log records. Ignored if `include_time_key` false. | | `timestamp` | -| ingestion_endpoint_connect_timeout The connection timeout of various Kusto endpoints in seconds. | | | `60s` | +| ingestion_endpoint_connect_timeout | The connection timeout of various Kusto endpoints in seconds. | | `60s` | | compression_enabled | Sends compressed HTTP payload (gzip) to Kusto, if enabled. | | `true` | | ingestion_resources_refresh_interval | The ingestion resources refresh interval of Kusto endpoint in seconds. | | `3600` | | workers | The number of [workers](https://docs.fluentbit.io/manual/administration/multithreading#outputs) to perform flush operations for this output. | | `0` |