Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support VE-VE data transfer #276

Open
wgip opened this issue Oct 1, 2021 · 1 comment
Open

Support VE-VE data transfer #276

wgip opened this issue Oct 1, 2021 · 1 comment
Labels
optimization Optimization tasks

Comments

@wgip
Copy link
Contributor

wgip commented Oct 1, 2021 •
edited
Loading

It comes in 2 flavours:

  • Across 2 plans that support VE - but within the same VE node
  • In a shuffle, where data needs to be moved to another VE node

We would need to implement a ColumnarVector that refers to a memory location x VE node, as well as plans to convert between in-Spark ColumnarVector and VEColumnarVector
@wgip wgip added the optimization Optimization tasks label Oct 1, 2021
@saudet
Copy link
Collaborator

saudet commented Oct 1, 2021

I'm not too sure about the specifics, yet, but it also happens within the same plan on the same node, for example:
https://github.com/NVIDIA/spark-rapids/tree/branch-21.10/docs/dev#how-the-rapids-plugin-works

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
optimization Optimization tasks
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@saudet @wgip