[QUESTION] Does intermittent_producer.md apply to kvssink?

[pwolfe1]

I am writing an application that uses GStreamer to intermittently stream video from an RTSP camera to KVS. Something I have not been able to figure out is the "correct" way of doing this. I have tried terminating and re-creating the GStreamer pipeline that ends in kvssink each time the stream is meant to start, but I have seen issues with the KVS SDK not being fully cleaned up with this approach (often logging a 0x5200052 error after the GStreamer pipeline has been destroyed). The other approach as I see it is to set the GStreamer pipeline to PAUSED rather than destroying it, and to resume it when desired.

This has seemed to work well more or less, although I am seeing the following log which has me concerned:

2023-05-24T18:58:53.042Z [INFO] (Copier) LivestreamManager: stdout. 2023-05-24 18:56:00 [548380061840] WARN - curlCompleteSync(): curl perform failed for url https://s-1ca2069c.kinesisvideo.us-east-1.amazonaws.com/putMedia with result Timeout was reached: Operation too slow. Less than 30 bytes/sec transferred the last 30 seconds. {scriptName=LivestreamManager.lifecycle.Run, serviceName=LivestreamManager, currentState=RUNNING}
2023-05-24T19:04:09.163Z [INFO] (Copier) LivestreamManager: stdout. 2023-05-24 19:03:57 [547541217424] WARN - notifyDataAvailable(): Failed to un-pause curl with error: 43. Curl object 0x7f74016800. {scriptName=LivestreamManager.lifecycle.Run, serviceName=LivestreamManager, currentState=RUNNING}

I am using meta-aws to add kvssink to my Yocto distro. Based on the recipe I am fairly confident I am on a version >= 3.1 (The line established in intermittent_producer.md). But these warnings lead me to think that the < 30 bytes in 30 seconds is causing an issue.

Something else I noticed is that the Upload stream handle increments each time the stream starts. Does this mean resources are not being freed properly or is it okay for this handle # to increment indefinitely?

I've also found that when I try to dynamically change the streamName for my kvssink each time the stream starts, the
pipeline seems to hang and no data is delivered to KVS. Is this unsupported behavior? We are more than able to work around it if so but just wanted to ask for clarification.

Apologies if this is answered elsewhere and thank you for taking the time to read and respond!

[pwolfe1]

I have now seen some additional logging when waiting longer than 10 minutes between streams.

2023-05-25T16:26:05.492Z [INFO] (Copier) LivestreamManager: stdout. 2023-05-25 16:10:02 [547247599760] DEBUG - putStreamCurlHandler(): Network thread for Kinesis Video stream: <stream_name> with upload handle: 2 exited. http status: 0. {scriptName=services.LivestreamManager.lifecycle.Run, serviceName=LivestreamManager, currentState=RUNNING}
2023-05-25T16:26:05.492Z [INFO] (Copier) LivestreamManager: stdout. 2023-05-25 16:10:02 [547247599760] WARN - putStreamCurlHandler(): Stream with streamHandle 547543368112 uploadHandle 2 has exited without triggering end-of-stream. Service call result: 599. {scriptName=services.LivestreamManager.lifecycle.Run, serviceName=LivestreamManager, currentState=RUNNING}
2023-05-25T16:26:05.492Z [INFO] (Copier) LivestreamManager: stdout. 2023-05-25 16:10:02 [547247599760] INFO - kinesisVideoStreamTerminated(): Stream 0x7f7c20c1b0 terminated upload handle 2 with service call result 599.. {scriptName=servicesLivestreamManager.lifecycle.Run, serviceName=LivestreamManager, currentState=RUNNING}
2023-05-25T16:26:05.492Z [INFO] (Copier) LivestreamManager: stdout. 2023-05-25 16:10:02 [547247599760] DEBUG - defaultStreamStateTransitionHook(): Stream state machine retry count: 2. {scriptName=services.LivestreamManager.lifecycle.Run, serviceName=LivestreamManager, currentState=RUNNING}

I can see that status code 599 means SERVICE_CALL_NETWORK_CONNECTION_TIMEOUT. Eventually (after multiple retries) it seems like a new uploadHandle is created and the stream starts properly.

Will changing the connection-staleness parameter for kvssink affect this?

[disa6302]

@pwolfe1 ,

Intermittent producer does apply to kvssink. As far as 0x5200052 error goes, we fixed it in our recent commits in the develop branch and will be included in the next release.

The sink does not specifically listen on PAUSED and react right now, if say the data stops coming in or an EOS is detected, we end up stopping the pipeline and that is why you see this issue. So tearing down and starting is the way to go.

About upload stream handle, it is expected to increase when the credentials are refreshed which happens or certain errors occur. So it is expected.

[pwolfe1]

Intermittent producer does apply to kvssink. As far as 0x5200052 error goes, we fixed it in our recent commits in the develop branch and will be included in the next release.

That is awesome, I am looking forward to trying it out.

Thanks @disa6302!