ORSet with large data faces maximum-frame-size limitation during sync

[qwas368]

Version Information
Version of Akka.NET?
1.5.13
Which Akka.NET Modules?
Akka.DistributedData

Describe the bug
I am learning to use Distributed Data and know that ORSet can be used to store data sets. So, I tested its capacity by writing the following code to generate a large ORSet:

for(int i = 0; i < 20000; i++ )
{
    _ = await _replicator.Ask<IUpdateResponse>(Update(
        _key,
        ORSet<String>.Empty,
        Akka.DistributedData.WriteLocal.Instance,
        old => old.Add(Cluster, Guid.NewGuid().ToString())));
}

As you can see, this is a dataset consisting of 20,000 strings. There were no issues during this process. However, when I started another Node and attempted to sync the data, the node containing the data threw the following error:

[ERROR][06/12/2024 07:09:55.377Z][Thread 0031][akka.tcp://[email protected]:7001/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FopcSystem%4010.6.26.49%3A7002-2/endpointWriter] Transient association error (association remains live)
Cause: Akka.Remote.OversizedPayloadException: Discarding oversized payload sent to [akka.tcp://[email protected]:7002/user/ddataReplicator#1203595200]: max allowed size 128000 bytes, actual size of encoded Akka.DistributedData.Internal.Gossip was 512225 bytes.

I started trying to adjust parameters, mainly distributed-data.max-delta-elements, but it had no effect. I suspect that max-delta-elements is used to split multiple IReplicatedData, while the entire ORSet is considered as one. Is that correct? This confuses me as ORSet's IDeltaReplicatedData and max-delta-elements don't seem to be actually related.

Expected behavior
The expected behavior is that ORSet syncs data incrementally in the background, unless the code explicitly reads it. The maximum-frame-size limitation should apply to individual elements within the collection, not the entire ORSet.

Environment
Are you running on Linux? Windows? Docker? Which version of .NET?
.net framework 4.7.2

[Aaronontheweb]

I started trying to adjust parameters, mainly distributed-data.max-delta-elements, but it had no effect. I suspect that max-delta-elements is used to split multiple IReplicatedData, while the entire ORSet is considered as one. Is that correct? This confuses me as ORSet's IDeltaReplicatedData and max-delta-elements don't seem to be actually related.

I guess we'll need to sanity check to see whether or not this setting is actually being honored by DData, so we'll follow up on that.

As a work-around you can increase the Akka.Remote max-fame-size to some larger value for the time being.

akka {
    remote {
        dot-netty.tcp {
            maximum-frame-size = 1 MiB
            send-buffer-size = 2 MiB
            receive-buffer-size = 2 MiB
        }
    }
}

[Aaronontheweb]

Looks like we did address a bug related to this in #5157 - is it possible that the elements in your ORSet<T> are too large to fit into the frame size?

[qwas368]

Looks like we did address a bug related to this in #5157 - is it possible that the elements in your ORSet<T> are too large to fit into the frame size?

This seems to be two separate issues. The element in ORSet<string> that I am adding is a GUID string, which will not exceed the maximum-frame-size.

I tried to debug it, but I'm not familiar with the Akka source code. I only know that it retrieves the entire ORSet data and sends it out from here.I hope this helps with the issue.

akka.net/src/contrib/cluster/Akka.DistributedData/Replicator.cs

Line 1144 in f2b99b4

var g = new Gossip(keys.ToImmutableDictionary(x => x, _ => GetData(_)), !otherDifferentKeys.IsEmpty);

[Arkatufus]

Currently we do not have any settings that chunks the ORSet entries itself inside a gossip.
The "akka.cluster.distributed-data.max-delta-elements" controls the maximum number of data entries that is being sent per Gossip message, it will not split the data entry itself.
In your case, there is only 1 ORSet data entry that is being sent inside the gossip message, the problem is that 1 data contains 20000 elements, which, even after being compressed, exceeds the network TCP frame size.