Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add compression option #159

Open
Max-Meldrum opened this issue Sep 20, 2021 · 3 comments
Open

Add compression option #159

Max-Meldrum opened this issue Sep 20, 2021 · 3 comments
Assignees
Labels
Milestone

Comments

@Max-Meldrum
Copy link
Contributor

Add compression feature to the networking layer using zstd.

We can also add a compression level variable to the NetworkConfig to allow users to configure compression ratio / speed tradeoffs.

@Max-Meldrum Max-Meldrum self-assigned this Sep 20, 2021
@Bathtor Bathtor added this to the 1.0.0 milestone Sep 20, 2021
@Bathtor
Copy link
Contributor

Bathtor commented Sep 20, 2021

I do like having it in config instead of feature flag tbh. We already have so many feature flags. If it's not a huge performance drain when deactivated via config I think that'll be the nicer way for a feature like this.

@Max-Meldrum Max-Meldrum changed the title Compression feature flag Add compression option Sep 20, 2021
@Max-Meldrum
Copy link
Contributor Author

Update: it was trickier than expected.

Ideal scenario? Compression and decompression happen on the component thread in order to (1) reduce the load on the network buffers; (2) avoid having the network thread doing the compression/decompression.

This is basically what happens if one does the compress/decompress by hand in Serializer/Deserializer. While the compression could be done in the serialise_msg function, the blocker is more on how to do the decompression at the component thread. That is, making sure the data is decompressed before receive_network is called.

// ActorRaw
self.receive_network(nm) 

May have another go at it down the line..

@Bathtor
Copy link
Contributor

Bathtor commented Nov 2, 2021

Just an idea: In stream-oriented protocols we may benefit from a quick compression at the network level if we put it before the buffer flushing. Because we may actually have collected multiple small messages worth of data there that can be compressed while individual small messages cannot be compressed well. I don't know exactly how that logic there works right now (@adamhass ?) but this is basically what's happening in Kompics iirc.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants