Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using Python KCL library for mutli-stream processing #172

Open
DewanChaulagain opened this issue May 5, 2022 · 1 comment
Open

Using Python KCL library for mutli-stream processing #172

DewanChaulagain opened this issue May 5, 2022 · 1 comment

Comments

@DewanChaulagain
Copy link

I am using amazon-kclpy==2.0.6 to process kinesis streams. This works well for single stream processing. From the Java library, I see that multi-stream support was introduced in the java library version 2.3.0. Also worth noting that amazon-kclpy v2.0.6 uses Amazon Kinesis Client v2.3.9 as dependency.

The official docs released with the Java library suggests

With this new capability, you can update the list of streams at runtime for multi-stream processing in a scalable KCL application without redeploying the application

My use case requires one application to listen/process multiple Kinesis streams. As the stream names are static, could even provide the stream names as part of the configuration itself.

The issue:

As the python client interacts with the Java Multilangdaemon, we have the ability to provide the configuration file to the MultilangDaemon. Yet :

  1. We cannot provide multiple stream names ( a list of names) as the streamName in properties file. This field is expected to be a string.
  2. We cannot provide two properties file ( with each file set up with different streamName to the Multilangdaemon.

How can we configure the KCL Python to use multiple stream? Thank you!

@harrydaniels-IW
Copy link

I also need this, are there any plans to update KCL Library to support multiple streams?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants