Google cloud speech to text : Dealing with high amount of input data in realtime

[sychopath]

Scenario: We are receiving huge amount of voice data streams which could be upto couple of minutes long. There will huge set of data streams as an input every second. This a real time application where I get live data from multiple users and return response from api.

I am using google' AsyncStreamingRecognize to create stream. This stream open, read and write functions are blocking as I cannot proceed with other operation until success is returned by get() functions. Due to this my threads responsible for dealing with stream data are getting blocked.

Is there a better way to where I can write high amount of input data without my thread being blocked for previous write function ? The code/setup works fine for a single call in real time. But I doubt how to design it well that could handle huge realtime incoming traffic

[coryan]

This stream open, read and write functions are blocking as I cannot proceed with other operation until success is returned by get() functions.

You can set up a callback to be triggered when the google::cloud::future<> is satisfied. That makes your application non-blocking and can process multiple streams with fewer threads. For an example see:

https://github.com/GoogleCloudPlatform/cpp-samples/blob/775a2ac29173cdae7dc984c68cd2a35f4e629dc3/speech/api/streaming_transcribe_singlethread.cc#L35

You would need to create one of these Handler objects for each stream.

With that said, if you can use C++20 coroutines, then consider doing so. The code is far more readable than using callbacks, and it basically does the same thing:

https://github.com/GoogleCloudPlatform/cpp-samples/blob/775a2ac29173cdae7dc984c68cd2a35f4e629dc3/speech/api/streaming_transcribe_coroutines.cc#L53

In either case, you may want to increase the number of background threads if a single CPU cannot keep up. There is an option to set the size of the background thread pool. Eventually you may need to create your own thread pool.

[sychopath]

Thanks for sharing @coryan !!

In either case, you may want to increase the number of background threads if a single CPU cannot keep up. There is an option to set the size of the background thread pool. Eventually you may need to create your own thread pool.

I am exploring an option of creating my own thread pool.

https://github.com/GoogleCloudPlatform/cpp-samples/blob/775a2ac29173cdae7dc984c68cd2a35f4e629dc3/speech/api/streaming_transcribe_coroutines.cc#L53

I was going through your code and trying develop an understanding. I found something which I do not fully understand :

g::CompletionQueue cq;

Can you help me understand what exactly is significance of completion queue in your code ?

[coryan]

Can you help me understand what exactly is significance of completion queue in your code ?

A google::cloud::CompletionQueue is a wrapper around grpc::CompletionQueue, but with a (I hope) nicer interface. It may be easier to explain the problem these things solve so you can get some context.

Assume you have an application that is using asynchronous operations (RPCs and timers in our case). You may have many asynchronous operations pending at a time. You want to get notified when an operation completes. Those notifications cannot run in the application threads. The whole reason you wanted to use asynchronous operations was to free your threads to do other work. Therefore, you need a pool of background threads to wait for these asynchronous operations and then to run the notifications.

It would be convenient to have a meeting point where the things that start asynchronous operations can say "please use this pool of background threads to run my notifications".

It would also be convenient if you could say "these are the background threads that will run notifications".

A google::cloud::CompletionQueue is that meeting point. It performs the I/O in the background and notifies your application when the operation (RPC or timer) completes.

Because of my background and biases, I think of a CompletionQueue as the "Asynchronous Operation Processor" actor in the Proactor Pattern. The wikipedia article is too brief, but there are some good links there, such as:

https://www.dre.vanderbilt.edu/~schmidt/PDF/Proactor.pdf
https://www.boost.org/doc/libs/1_80_0/doc/html/boost_asio/overview/core/async.html

Typically the client library creates a google::cloud::CompletionQueue for you, but sometimes you need to create your own. Maybe because you want access to it in order to schedule timers (the motivation in that example you linked), or maybe because you want to tweak the properties of the background threads.

[lucas-0liveira]

It seems like you're using Google's AsyncStreamingRecognize for processing streaming data, but you're encountering issues with blocking functions, particularly with regards to writing large amounts of input data. To handle high traffic in real-time scenarios, you might want to consider implementing a more asynchronous approach.

One suggestion is to utilize asynchronous I/O libraries or frameworks, like Python's asyncio module. This would allow you to perform non-blocking I/O operations, ensuring that your threads are not unnecessarily held up.

[coryan]

This looks like a LLM-generated answer. It does not add any new information, and it is (I think) misleading as it mentions Python in the wrong context.

[lucas-0liveira]

sorry :/