Improve rpc_soak and channel_soak test to cover concurrency in Go

[zbilun]

RELEASE NOTES:

• Improved the rpc_soak and channel_soak tests to support concurrency in Go. This change increases the test coverage of the C2P E2E load test and follows a similar approach used in the Java implementation.

This PR aims to enhance the test coverage of the C2P E2E load test by improving the rpc_soak and channel_soak tests to support concurrency in Go. The updated logic closely follows the approach used in the Java implementation, which has already been merged.

rpc_soak:
The client performs many large_unary RPCs in sequence over the same channel. The test can run in either a concurrent or non-concurrent mode, depending on the number of threads specified (soak_num_threads).

channel_soak:
Similar to rpc_soak, but this time each RPC is performed on a new channel. The channel is created just before each RPC and is destroyed just after.

Note on Concurrent Execution and Channel Creation:
In a concurrent execution setting (i.e., when soak_num_threads > 1), each thread performs a portion of the total soak_iterations and creates and destroys its own channel for each RPC iteration.

Thread-specific logs will include the thread_id, helping to track performance across threads, especially when each thread is managing its own channel lifecycle.

@townba
@dfawley

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 82.19%. Comparing base (66ba4b2) to head (0395c79).
Report is 22 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7926      +/-   ##
==========================================
+ Coverage   82.04%   82.19%   +0.14%     
==========================================
  Files         377      381       +4     
  Lines       38180    38539     +359     
==========================================
+ Hits        31326    31676     +350     
- Misses       5551     5560       +9     
  Partials     1303     1303              

see 48 files with indirect coverage changes

[dfawley]

Most of my comments are actually pretty superficial. Thanks for all the cleanups since the first round!

[dfawley]

This feels awkward. How about just a ChannelForTest function that is a closure returning conn in the shared case and creates a new connection in the non-shared case. And the closure can return a cleanup function that is a nop in the shared case, or closes the connection in the non-shared case.

[zbilun]

If I understand correctly, you’re suggesting introducing a boolean flag to determine whether we should create a new channel or continue using the existing one. Please feel free to correct me if I’ve misunderstood your comments.

In the previous version of the interop soak test (in Java), this approach was used. However, Alex worked to reduce the number of flags in order to keep the code cleaner. As a result, I refactored it into the current implementation while trying to maintain alignment with the Java version.

As I mentioned in my earlier response, we can certainly note these potential improvements and refactor ideas once the Go and C++ versions are released. This will allow us to ensure we adopt the best approach across languages.
What do you think? Thank you for your feedback!