[FLINK-31989][docs] Update english docs for KinesisStreamsSource and DynamoDbStreamsSource #179
Conversation
hlteoh37
changed the title
| <td>yes</td> | ||
| <td style="word-wrap: break-word;">16</td> | ||
| <td>Integer</td> | ||
| <td>Request threshold for uncompleted requests by <code>KinesisAsyncClient</code>before blocking new write requests and applying backpressure.</td> |
There was a problem hiding this comment.
This is actually something that should be explained in DataStream too.
Also, explaining that a "request" is a batch
There was a problem hiding this comment.
In general, the batching mechanism would deserve a dedicated chapter, possibly in DataStream docs, and linked from here
There was a problem hiding this comment.
Let's focus the changes for this PR to the source docs.
| Kinesis data streams consist of one or more shards, and the `sink.partitioner` option allows you to control how records written into a multi-shard Kinesis-backed table will be partitioned between its shards. | ||
| Valid values are: | ||
|
|
||
| * `fixed`: Kinesis `PartitionKey` values derived from the Flink subtask index, so each Flink partition ends up in at most one Kinesis partition (assuming that no re-sharding takes place at runtime). |
There was a problem hiding this comment.
It would be clearer putting this are mutually exclusive options.
Partitioning is defined by either using
PARTITION BYin the table definition or by specifying specifysink.partitioner. Using both will result in a configuration error.
Valid values for
sink.partitioner:
fixed...random...- Custom
FixedKinesisPartitionersubclass...
There was a problem hiding this comment.
Let's focus the changes for this PR to the source docs.
|
|
||
| {{< hint info >}} | ||
| Records written into tables defining a `PARTITION BY` clause will always be partitioned based on a concatenated projection of the `PARTITION BY` fields. | ||
| In this case, the `sink.partitioner` field cannot be used to modify this behavior (attempting to do this results in a configuration error). |
There was a problem hiding this comment.
If alternative options are explained above, this line becomes redundant
There was a problem hiding this comment.
Let's focus the changes for this PR to the source docs.
|
|
||
| # Data Type Mapping | ||
|
|
||
| Kinesis stores records as Base64-encoded binary data objects, so it doesn't have a notion of internal record structure. |
There was a problem hiding this comment.
Text formats, such as
jsonorcsvare written to Kinesis without modifications. Binary formats such asavroare Base64-encoded and then written to Kinesis as text.
(is this right?)
There was a problem hiding this comment.
I'm not sure actually!
Let's focus the changes for this PR to the source docs.
| <td>no</td> | ||
| <td style="word-wrap: break-word;">JOB_MANAGED</td> | ||
| <td>String</td> | ||
| <td>Only applicable to EFO <code>ReaderType</code>. Determine if the EFO consumer is managed by the Flink job <code>JOB_MANAGED|SELF_MANAGED</code>.</td> |
There was a problem hiding this comment.
Sorry, I copied&pasted the deleted title from the other docs.
It was "EFO Stream Consumer Lifecycle Management".
But we can skip linking across docs for now
Purpose of the change
Updates English docs for
KinesisStreamsSourceandDynamoDbStreamsSourceVerifying this change
This change is a docs change without any test coverage.
Significant changes
(Please check any boxes [x] if the answer is "yes". You can first publish the PR and check them afterwards, for convenience.)
@Public(Evolving))