Transcript and S3 audio file clipping at start of ingestion when cold-starting

[OhansonB]

We are using this solution to provide a voicemail functionality within Amazon Connect, by starting the capture of audio within the Amazon Connect contact flow and essentially prompting the caller to speak to themselves (i.e., "please leave a message after the tone" and then playing a long period of silence via an SSML 'Get Customer Input' block).

We have found that under light and intermittent loads most transcripts and audio files are missing the first six or seven seconds, meaning that vital information such as callers name or reason for calling is missing from the transcript and audio file.

Investigation has suggested that this is related to the Lambda function cold-starting (1), and the Java runtime associated with the kvsTranscriber Lambda function being slower to start-up than some other runtimes. We have resolved this through the implementation of three changes to the kvsTranscriber function:

1. Implementation of Provisioned Concurrency (2) (where the provisioned concurrency is equal to the number of expected concurrent transcription sessions to ensure an equal number of permanently provisioned Lambda execution environments). This prevents the cold-start of the Lambda function itself
2. Increase the amount of memory on the Lambda function from 512MB to 1024MB, which also increases the CPU of the function to simply try and make the function run faster
3. Introduce an environment variable to tweak the profiling behaviour of the Java runtime (3) ("JAVA_TOOL_OPTIONS": "-XX:+TieredCompilation -XX:TieredStopAtLevel=1") which appears to have made the Lambda function run faster too

There are a number of ways to solve the issue around the Lambda cold start (including the standard EventBridge based warmer where the kvsConsumer Lambda function already appears to have handling code to invoke the kvsTranscriber function as well), and I am not wholly sure whether increasing the amount of memory on the function has had an appreciable benefit compared to the other changes. However, I have mentioned these as the basis for points of investigation by individuals more familiar with the solution as these have been the changes that we have made to resolve this issue; particularly also given that I and my team am very unfamiliar with Java.

It is worth stating that with the increased amount of memory that each provisioned concurrency incurs a cost of approximately $13.00 per month, and therefore this is a potentially expensive solution compared to EventBridge-based warming, but for the relatively low concurrency required for our own use-case we have found this to be acceptable; hence the reason for an issue rather than a pull request.