Adds distributed row gatherer #1589

MarcelKoch · 2024-04-04T10:49:19Z

This PR adds a distributed row gatherer. This operator essentially provides the communication required in our matrix apply.

Besides the normal apply (which is blocking), it also provides two asynchronous calls. One version has an additional workspace parameter which is used as send buffer. This version can be called multiple times without restrictions, if different workspaces are used for each call. The other version doesn't have a workspace parameter, and instead uses an internal buffer. As a consequence, this function can only be called a second time, if the request of the previous call has been waited on. Otherwise, this function will throw.

This is the second part of splitting up #1546.

It also introduces some intermediate changes, which could be extracted out beforehand:

~~a type-erased DenseCache~~
~~makingdetail::run easier to use~~ now part of Use index_map in distributed::matrix #1544

PR Stack:

MarcelKoch · 2024-04-22T08:29:49Z

One issue that I have is the constructor. It takes a collective_communicator and an index_map. The index_map already defines the communication pattern, so the collective_communicator has to match that.
One option might be to have a virtual function like

std::unique_ptr<collective_communicator> create_with_same_type(communicator, index_map);

If I can't come up with anything better, I guess I will use that.

pratikvn · 2024-04-26T09:50:57Z

Do we need to have the std::future setup for the release ? Can we remove that for now and just use a normal synchronous approach ? I think that is a significant change that maybe needs more thought and probably a separate PR.

core/distributed/matrix.cpp

yhmtsai · 2024-10-07T13:26:19Z

core/multigrid/pgm.cpp

-                      send_sizes.data(), send_offsets.data(), type, recv_ptr,
-                      recv_sizes.data(), recv_offsets.data(), type);
+    coll_comm
+        ->i_all_to_all_v(use_host_buffer ? exec->get_master() : exec, send_ptr,


any difference between using all_to_all_v vs i_all_to_all_v? I assume all_to_all_v also update the interface

all_to_all_v is a blocking call, while i_all_to_all_v is non-blocking. Right now the collective_communicator only provides the non-blocking interface, since it is more general.

include/ginkgo/core/distributed/row_gatherer.hpp

yhmtsai · 2024-10-07T13:28:56Z

include/ginkgo/core/distributed/row_gatherer.hpp

+ * auto x = matrix::Dense<double>::create(...);
+ *
+ * auto future = rg->apply_async(b, x);
+ * // do some computation that doesn't modify b, or access x


I think it access x but it is unclear when it will be accessed before the wait

I guess this just meant to say that you can't expect any meaningful data when accessing x before the wait has completed.

I think I get it wrong.
Is the comment here to describe that user can do something safely after the call or the apply_async behavior?
My comment was based on that it is the behavior of the apply_async because apply_async definitely accesses x.
If it is for user action during async and wait, then it is correct.

core/distributed/row_gatherer.cpp

yhmtsai · 2024-10-23T08:37:11Z

core/distributed/row_gatherer.cpp

+                    workspace.set_executor(mpi_exec);
+                    if (send_size_in_bytes > workspace.get_size()) {
+                        workspace.resize_and_reset(sizeof(ValueType) *
+                                                   send_size[0] * send_size[1]);
+                    }


combining them to assign the workspace directly?

Combine how? Do you mean like

workspace = array<char>(mpi_exec, sizeof(ValueType) * send_size[0] * send_size[1]);

yhmtsai · 2024-10-23T08:41:27Z

core/distributed/row_gatherer.cpp

+                    req = coll_comm_->i_all_to_all_v(
+                        mpi_exec, send_ptr, type.get(), recv_ptr, type.get());


send_buffer might be on the host but the recv_ptr(x_local) might be on the device

I have a check above to ensure that the memory space of the recv buffer is accessible from the mpi executor. So if GPU aware MPI is used, it should work (even if send buffer is on the host and recv buffer in the device or vice versa). Otherwise an exception will be thrown.

core/test/mpi/distributed/row_gatherer.cpp

- only allocate if necessary - synchronize correct executor Co-authored-by: Pratik Nayak <[email protected]>

- split tests into core and backend part - fix formatting - fix openmpi pre 4.1.x macro Co-authored-by: Pratik Nayak <[email protected]> Co-authored-by: Yu-Hsiang M. Tsai <[email protected]> Signed-off-by: Marcel Koch <[email protected]>

MarcelKoch self-assigned this Apr 4, 2024

ginkgo-bot added reg:build This is related to the build system. reg:testing This is related to testing. mod:core This is related to the core module. type:matrix-format This is related to the Matrix formats labels Apr 4, 2024

MarcelKoch requested a review from pratikvn April 4, 2024 10:49

MarcelKoch force-pushed the distributed-row-gatherer branch from 6b4521b to ae60198 Compare April 4, 2024 11:00

MarcelKoch force-pushed the neighborhood-communicator branch from 6acf7c4 to 8aa6ab9 Compare April 4, 2024 11:00

MarcelKoch force-pushed the distributed-row-gatherer branch 2 times, most recently from 49557f1 to 4a79442 Compare April 5, 2024 08:18

MarcelKoch modified the milestone: Ginkgo 1.8.0 Apr 5, 2024

MarcelKoch force-pushed the neighborhood-communicator branch from 8aa6ab9 to 77398bd Compare April 17, 2024 16:28

MarcelKoch force-pushed the distributed-row-gatherer branch from 4a79442 to 172eb7d Compare April 17, 2024 16:28

MarcelKoch requested a review from upsj April 19, 2024 09:20

MarcelKoch mentioned this pull request Apr 19, 2024

Adds sparse communicator class #1546

Closed

7 tasks

MarcelKoch force-pushed the neighborhood-communicator branch from 77398bd to d278cad Compare April 19, 2024 14:39

MarcelKoch force-pushed the distributed-row-gatherer branch 2 times, most recently from 98fa10a to 79de4c3 Compare April 19, 2024 16:19

This was referenced Apr 22, 2024

Add segmented array type #1545

Merged

Distributed Index Map #1543

Merged

Adds Index map device kernels #1579

Merged

Use index_map in distributed::matrix #1544

Merged

MarcelKoch force-pushed the distributed-row-gatherer branch from 79de4c3 to b0e5c92 Compare April 22, 2024 11:11

MarcelKoch force-pushed the neighborhood-communicator branch from d278cad to d6112ef Compare April 22, 2024 11:11

MarcelKoch force-pushed the distributed-row-gatherer branch from b0e5c92 to 775854a Compare April 25, 2024 07:16

MarcelKoch force-pushed the neighborhood-communicator branch from d6112ef to 1582673 Compare April 25, 2024 07:16

MarcelKoch force-pushed the neighborhood-communicator branch from db7f6ed to 0ad4ee8 Compare August 9, 2024 13:40

MarcelKoch force-pushed the distributed-row-gatherer branch from bfc5233 to 8697971 Compare August 16, 2024 15:21

MarcelKoch force-pushed the neighborhood-communicator branch from 0ad4ee8 to 1f49b91 Compare August 16, 2024 15:21

MarcelKoch requested review from upsj and removed request for upsj August 27, 2024 12:05

MarcelKoch force-pushed the distributed-row-gatherer branch from 8697971 to 341e781 Compare October 7, 2024 13:06

MarcelKoch force-pushed the neighborhood-communicator branch from 1f49b91 to 4db050c Compare October 7, 2024 13:06

pratikvn reviewed Oct 7, 2024

View reviewed changes

core/distributed/matrix.cpp Show resolved Hide resolved

yhmtsai requested changes Oct 23, 2024

View reviewed changes

MarcelKoch force-pushed the neighborhood-communicator branch from 4db050c to 1ebe59f Compare October 23, 2024 13:32

MarcelKoch force-pushed the distributed-row-gatherer branch from b2025a8 to f77cb6c Compare October 23, 2024 14:17

MarcelKoch requested a review from yhmtsai October 24, 2024 10:47

MarcelKoch force-pushed the distributed-row-gatherer branch from f77cb6c to c827b23 Compare October 30, 2024 15:10

MarcelKoch force-pushed the neighborhood-communicator branch from 1ebe59f to e7d32a1 Compare October 30, 2024 15:10

MarcelKoch force-pushed the distributed-row-gatherer branch from c827b23 to 2a54c3e Compare October 30, 2024 15:30

MarcelKoch modified the milestones: Ginkgo 1.9.0, Ginkgo 1.10.0 Dec 9, 2024

MarcelKoch force-pushed the neighborhood-communicator branch from 1216932 to ceb6f2e Compare December 17, 2024 13:59

MarcelKoch force-pushed the distributed-row-gatherer branch from 2a54c3e to bd358fc Compare December 18, 2024 09:19

MarcelKoch force-pushed the neighborhood-communicator branch from ceb6f2e to 807118c Compare December 18, 2024 09:27

MarcelKoch force-pushed the distributed-row-gatherer branch from bd358fc to a52ba0d Compare December 18, 2024 09:27

MarcelKoch and others added 8 commits December 19, 2024 15:14

[dist-rg] adds distributed row-gatherer

32a35a4

[dist-rg] handle copy to host buffer

88629fc

[dist-rg] use mpi request instead of future

84dd51b

[dist-rg] adds distributed row-gatherer tests

16830d8

[dist-mat] use row-gatherer

39539a2

[pgm] use row-gatherer from matrix

d87725f

[dist-rg] review update:

3048f5e

- only allocate if necessary - synchronize correct executor Co-authored-by: Pratik Nayak <[email protected]>

[dist-rg] review updates:

08c1f4e

- split tests into core and backend part - fix formatting - fix openmpi pre 4.1.x macro Co-authored-by: Pratik Nayak <[email protected]> Co-authored-by: Yu-Hsiang M. Tsai <[email protected]> Signed-off-by: Marcel Koch <[email protected]>

MarcelKoch force-pushed the distributed-row-gatherer branch from a52ba0d to 08c1f4e Compare December 19, 2024 14:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds distributed row gatherer #1589

Adds distributed row gatherer #1589

MarcelKoch commented Apr 4, 2024 •

edited

Loading

MarcelKoch commented Apr 22, 2024

pratikvn commented Apr 26, 2024

yhmtsai Oct 7, 2024

MarcelKoch Oct 23, 2024

yhmtsai Oct 7, 2024

MarcelKoch Oct 23, 2024

yhmtsai Oct 23, 2024

yhmtsai Oct 23, 2024

MarcelKoch Oct 23, 2024

yhmtsai Oct 23, 2024

MarcelKoch Oct 23, 2024

		req = coll_comm_->i_all_to_all_v(
		mpi_exec, send_ptr, type.get(), recv_ptr, type.get());

Adds distributed row gatherer #1589

Are you sure you want to change the base?

Adds distributed row gatherer #1589

Conversation

MarcelKoch commented Apr 4, 2024 • edited Loading

MarcelKoch commented Apr 22, 2024

pratikvn commented Apr 26, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MarcelKoch commented Apr 4, 2024 •

edited

Loading