Calls issued from clients to StatelessWorker grains should be spread across silos

[ReubenBond]

Fixes #9149

Microsoft Reviewers: Open in CodeFlow

[JorgeCandeias]

Will this apply to calls from clients to silos or also from within a silo?
Meaning, will a call originating from within silo to a stateless worker pool continue to hit the local pool, or can it get sent to a pool in some other silo?

[ReubenBond]

This PR only affects external clients. Within a silo the calls will still be sent to the local silo if compatible. Otherwise, they will be sent to a random compatible silo. In the latter case, the random compatible silo is cached as the location for the worker so it will not spread load since it treats it as a normal grain.

We could additionally avoid caching placement for stateless workers but I'm not sure we should. The distinction between external & internal does raise doubt over whether it's good to have this functionality at all. I welcome your thoughts. Also, @benjaminpetit, wdyt?

[JorgeCandeias]

Changing that behaviour for internal silo calls would be a severe breaking change for us. We rely on the current routing behaviour to avoid serialization/hops of too-expensive-to-serialize models that still require stateful fan-out processing, even if locally.

I'm not against changing this, as the alternative can have other benefits wrt scale-out scenarios. However if you decide to, pretty please do mention that in the release notes in big red caps.