Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Repo sync for protected branch #2377

Merged
merged 6 commits into from
Sep 3, 2024
Merged

Repo sync for protected branch #2377

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
2877cb6
changed output, description
mjosephym Aug 29, 2024
f42f166
Merge pull request #6203 from mjosephym/kusto-uf-295662-series-asin
prmerger-automator[bot] Sep 3, 2024
0f1d529
Update mappings.md
YaelSchuster Sep 3, 2024
9c764bc
Merge pull request #6209 from MicrosoftDocs/YaelSchuster-patch-3
prmerger-automator[bot] Sep 3, 2024
276e096
Merge pull request #6212 from MicrosoftDocs/main
ttorble Sep 3, 2024
a58e617
Merging changes synced from https://github.com/MicrosoftDocs/dataexpl…
Sep 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 15 additions & 15 deletions data-explorer/kusto/management/mappings.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ Ingestion mappings are used during ingestion to map incoming data to columns ins

Data Explorer supports different types of mappings, both row-oriented (CSV, JSON, AVRO and W3CLOGFILE), and column-oriented (Parquet and ORC).

Ingestion mappings can be [pre-created](create-ingestion-mapping-command.md) and can be referenced from the ingest command using `ingestionMappingReference` parameters. Although, ingestion is possible without specifying a mapping. For more information, see [identity mapping](#identity-mapping).
Ingestion mappings can be [precreated](create-ingestion-mapping-command.md) and can be referenced from the ingest command using `ingestionMappingReference` parameters. Ingestion is possible without specifying a mapping. For more information, see [identity mapping](#identity-mapping).

Each element in the mapping list is constructed from three fields:

Expand Down Expand Up @@ -49,7 +49,7 @@ The following table defines mapping types to be used when ingesting or querying

## Identity mapping

Ingestion is possible without specifying `ingestionMapping` or `ingestionMappingReference` properties. The data will be mapped using an identity data mapping derived from the table's schema. The table schema will remain the same. `format` property should be specified. See [ingestion formats](../ingestion-supported-formats.md).
Ingestion is possible without specifying `ingestionMapping` or `ingestionMappingReference` properties. The data is mapped using an identity data mapping derived from the table's schema. The table schema remains the same. `format` property should be specified. See [ingestion formats](../ingestion-supported-formats.md).

| Format type | Format | Mapping logic |
|---------------------------------------------------------------------------------------------------------|--------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------|
Expand All @@ -61,19 +61,19 @@ Ingestion is possible without specifying `ingestionMapping` or `ingestionMapping

## Mapping transformations

Some of the data format mappings (Parquet, JSON and AVRO) support simple and useful ingest-time transformations. Where the scenario requires more complex processing at ingest time, use [Update policy](show-table-update-policy-command.md), which allows defining lightweight processing using KQL expression.
Some of the data format mappings (Parquet, JSON, and AVRO) support simple and useful ingest-time transformations. Where the scenario requires more complex processing at ingest time, use [Update policy](show-table-update-policy-command.md), which allows defining lightweight processing using KQL expression.

| Path-dependant transformation | Description | Conditions |
|-------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------|
| PropertyBagArrayToDictionary | Transforms JSON array of properties, such as `{events:[{"n1":"v1"},{"n2":"v2"}]}`, to dictionary and serializes it to valid JSON document, such as `{"n1":"v1","n2":"v2"}`. | Available for `JSON`, `Parquet`, `AVRO` and `ORC` mapping types. |
| SourceLocation | Name of the storage artifact that provided the data, type string (for example, the blob's "BaseUri" field).|Available for `CSV`, `JSON`, `Parquet`, `AVRO`, `ORC` and `W3CLOGFILE` mapping types. |
| SourceLineNumber | Offset relative to that storage artifact, type long (starting with '1' and incrementing per new record). | Available for `CSV`, `JSON`, `Parquet`, `AVRO`, `ORC` and `W3CLOGFILE` mapping types. |
| DateTimeFromUnixSeconds | Converts number representing unix-time (seconds since 1970-01-01) to UTC datetime string.| Available for `JSON`, `Parquet`, `AVRO` and `ORC` mapping types. |
| DateTimeFromUnixMilliseconds | Converts number representing unix-time (milliseconds since 1970-01-01) to UTC datetime string. | Available for `JSON`, `Parquet`, `AVRO` and `ORC` mapping types. |
| DateTimeFromUnixMicroseconds | Converts number representing unix-time (microseconds since 1970-01-01) to UTC datetime string. | Available for `JSON`, `Parquet`, `AVRO` and `ORC` mapping types. |
| DateTimeFromUnixNanoseconds | Converts number representing unix-time (nanoseconds since 1970-01-01) to UTC datetime string. | Available for `JSON`, `Parquet`, `AVRO` and `ORC` mapping types. |
| DropMappedFields | Maps an object in the JSON document to a column and removes any nested fields already referenced by other column mappings. | Available for `JSON`, `Parquet`, `AVRO` and `ORC` mapping types. |
| BytesAsBase64 | Treats the data as byte array and converts it to a base64-encoded string. | Available for `AVRO` mapping type. For `ApacheAvro` format, the schema type of the mapped data field should be `bytes` or `fixed` Avro type. For `Avro` format, the field should be an array containing byte values from [0-255] range. `null` is ingested if the data does not represent a valid byte array. |
| PropertyBagArrayToDictionary | Transforms JSON array of properties, such as `{events:[{"n1":"v1"},{"n2":"v2"}]}`, to dictionary and serializes it to valid JSON document, such as `{"n1":"v1","n2":"v2"}`. | Available for `JSON`, `Parquet`, `AVRO`, and `ORC` mapping types. |
| SourceLocation | Name of the storage artifact that provided the data, type string (for example, the blob's "BaseUri" field).|Available for `CSV`, `JSON`, `Parquet`, `AVRO`, `ORC`, and `W3CLOGFILE` mapping types. |
| SourceLineNumber | Offset relative to that storage artifact, type long (starting with '1' and incrementing per new record). | Available for `CSV`, `JSON`, `Parquet`, `AVRO`, `ORC`, and `W3CLOGFILE` mapping types. |
| DateTimeFromUnixSeconds | Converts number representing unix-time (seconds since 1970-01-01) to UTC datetime string.| Available for `CSV`, `JSON`, `Parquet`, `AVRO`, and `ORC` mapping types. |
| DateTimeFromUnixMilliseconds | Converts number representing unix-time (milliseconds since 1970-01-01) to UTC datetime string. | Available for `CSV`, `JSON`, `Parquet`, `AVRO`, and `ORC` mapping types. |
| DateTimeFromUnixMicroseconds | Converts number representing unix-time (microseconds since 1970-01-01) to UTC datetime string. | Available for `CSV`, `JSON`, `Parquet`, `AVRO`, and `ORC` mapping types. |
| DateTimeFromUnixNanoseconds | Converts number representing unix-time (nanoseconds since 1970-01-01) to UTC datetime string. | Available for `CSV`, `JSON`, `Parquet`, `AVRO`, and `ORC` mapping types. |
| DropMappedFields | Maps an object in the JSON document to a column and removes any nested fields already referenced by other column mappings. | Available for `JSON`, `Parquet`, `AVRO`, and `ORC` mapping types. |
| BytesAsBase64 | Treats the data as byte array and converts it to a base64-encoded string. | Available for `AVRO` mapping type. For `ApacheAvro` format, the schema type of the mapped data field should be `bytes` or `fixed` Avro type. For `Avro` format, the field should be an array containing byte values from [0-255] range. `null` is ingested if the data doesn't represent a valid byte array. |

### Mapping transformation examples

Expand Down Expand Up @@ -122,17 +122,17 @@ Given the following AVRO file contents:
}
```

The following data mapping maps the id column twice, with and without the transformation.
The following data mapping maps the ID column twice, with and without the transformation.

```json
[
{ "Column": "Id", "Properties": { "Path": "$.props.id" } },
{ "Column": "ID", "Properties": { "Path": "$.props.id" } },
{ "Column": "Base64EncodedId", "Properties": { "Path": "$.props.id", "Transform":"BytesAsBase64" } },
]
```

The ingested data looks as follows:

| Id | Base64EncodedId |
| ID | Base64EncodedId |
|-------|-----------|
| `[227,131,34,92,28,91,65,72,134,138,9,133,51,45,104,52]` | `44MiXBxbQUiGigmFMy1oNA==` |
6 changes: 4 additions & 2 deletions data-explorer/kusto/query/series-asin-function.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ title: series_asin()
description: Learn how to use the series_asin() function to calculate the element-wise arcsine function of the numeric series input.
ms.reviewer: alexans
ms.topic: reference
ms.date: 08/11/2024
ms.date: 08/29/2024
---
# series_asin()

Expand All @@ -29,6 +29,8 @@ Dynamic array of calculated arcsine function values. Any non-numeric element yie

## Example

The following example creates a dynamic array, `arr`, with the value `[-1,0,1]`. It then extends the results with column `arr_asin`, containing the results of the `series_asin()` function applied to the `arr` array.

:::moniker range="azure-data-explorer"
> [!div class="nextstepaction"]
> <a href="https://dataexplorer.azure.com/clusters/help/databases/Samples?query=H4sIAAAAAAAAAysoyswrUUgsKlKwVUipzEvMzUzWiNY11DHQMYzV5KpRSK0oSc1LASmITyzOzAOqKk4tykwtBvM0gMKaAOO+/MVCAAAA" target="_blank">Run the query</a>
Expand All @@ -43,4 +45,4 @@ print arr = dynamic([-1,0,1])

|arr|arr_asin|
|---|---|
|[-6.5,0,8.2]|[1.5707963267948966,0.0,1.5707963267948966]|
|[-1,0,1]|["-1.5707963267948966",0,"1.5707963267948966"]|