Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stream Arrow RecordBatches using mpsc channel and tokio_streams #481

Merged

Conversation

jonmmease
Copy link

Fixes compilation of #462 by using an mpsc channel and tokio_streams.

The reason this is more complicated that what you were trying @domoritz , is that the Arrow type provided by the duckdb crate is not Send. This means that it (and objects that depend on it) cannot be transferred across threads. This is why it doesn't work to build a stream using the stream! macro and return it from inside the conn.interact closure.

Using mpsc channels like I am here is a common workaround for dealing with non-send objects. Here the Arrow object stays in the thread that constructed it, and pushes the record batches through the channel back to the main thread. This channel is wrapped into a stream using the ReceiverStream from the tokio_stream crate.

I didn't actually test this, just got it compiling, so give it a try and see if it works the way you want.

@domoritz domoritz merged commit ffc034a into uwdata:dom/rust-stream Aug 14, 2024
@domoritz
Copy link
Member

Thank you

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants