Add local to global index mapping #1707

MarcelKoch · 2024-10-23T15:12:31Z

This PR adds a local-to-global mapping to the index_map class. It is necessary for the follow-up PR #1639. This also introduces a device-side view of a partition, and adds another member to the partition which stores the range-ids segmented by their part-id.

(I know that the branch name is wrong...)

yhmtsai

not finish review yet.
why is it required for PGM?
after merging row and column, do they still need to map back to the global index?
you need to reindex all of them because the size will be the smaller than the original one after coarsening.

yhmtsai · 2024-12-18T08:09:16Z

core/distributed/device_partition.hpp

+//
+// SPDX-License-Identifier: BSD-3-Clause
+
+#ifndef GINKGO_PARTITION_HPP


nit:

Suggested change

#ifndef GINKGO_PARTITION_HPP

#ifndef GKO_CORE_DISTRIBUTED_DEVICE_PARTITION_HPP_

yhmtsai · 2024-12-18T08:10:27Z

core/distributed/device_partition.hpp

+/**
+ * Create device_segmented_array from a segmented_array.
+ */


forgot to update documentation?

yhmtsai · 2024-12-18T08:10:41Z

core/distributed/device_partition.hpp

+ * Explicitly create a const version of device_segmented_array.
+ *
+ * This is mostly relevant for tests.


yhmtsai · 2024-12-18T08:12:42Z

core/distributed/device_partition.hpp

+ */
+template <typename LocalIndexType, typename GlobalIndexType>
+constexpr device_partition<const LocalIndexType, const GlobalIndexType>
+to_device_const(


is there any difference between to_device_const and to_device?

yhmtsai · 2024-12-18T08:14:09Z

core/distributed/index_map_kernels.hpp

+ *
+ * The relevant input parameter from the index map are:
+ *
+ * - partition:  the global partition


Suggested change

* - partition: the global partition

* - partition: the global partition

yhmtsai · 2024-12-18T08:55:52Z

reference/distributed/partition_kernels.cpp

+              });
+
+    sizes.resize_and_reset(num_parts);
+    std::fill_n(sizes.get_data(), num_parts, int64(0));


Suggested change

std::fill_n(sizes.get_data(), num_parts, int64(0));

std::fill_n(sizes.get_data(), num_parts, zero<int64>());

yhmtsai · 2024-12-18T08:56:59Z

test/distributed/index_map_kernels.cpp

+        auto seed = engine_dist(rd);
+        std::cout << "seed = " << seed << std::endl;
+        engine.seed(490729788);


some leftover?

yhmtsai · 2024-12-18T12:14:55Z

core/distributed/partition.cpp

+    exec->run(partition::make_build_ranges_by_part(
+        part_ids_.get_const_data(), get_num_ranges(), get_num_parts(),
+        range_ids, num_ranges_per_part));
+    ranges_by_part_ = segmented_array<size_type>::create_from_sizes(
+        std::move(range_ids), num_ranges_per_part);


currently, this rely on the atomic add to count the size.
then segmented_array will convert size to offset.
Maybe you can implement a similar kernel with convert_idxs_to_ptrs + additional index map from i to range_id[i] because range_part[range_id[i]] is sorted. then segmented_array can create_from_offsets directly

yhmtsai · 2024-12-18T12:29:33Z

reference/distributed/partition_helpers.hpp

+template <typename LocalIndexType, typename GlobalIndexType>
+GlobalIndexType map_to_global(
+    LocalIndexType idx,
+    device_partition<const LocalIndexType, const GlobalIndexType> partition,


Suggested change

device_partition<const LocalIndexType, const GlobalIndexType> partition,

const device_partition<const LocalIndexType, const GlobalIndexType>& partition,

yhmtsai · 2024-12-18T12:30:31Z

reference/distributed/partition_helpers.hpp

+    device_partition<const LocalIndexType, const GlobalIndexType> partition,
+    size_type range_id)
+{
+    auto range_bounds = partition.offsets_begin;


assert range_id is lower than num_range?

MarcelKoch · 2024-12-18T12:52:03Z

after merging row and column, do they still need to map back to the global index?
you need to reindex all of them because the size will be the smaller than the original one after coarsening.

@yhmtsai I don't understand your point, could you rephrase this?

yhmtsai · 2024-12-18T13:00:20Z

rank 0: holds row 0, 1, 4
rank 1: holds row 2, 3
if we mege 0, 1 and 2, 3
rank 0: 0, 4
rank 1: 2
we need to compute the size of each gap from different node and then reindex the global index
rank 0: 0, 2
rank 1: 1

MarcelKoch · 2024-12-18T13:02:37Z

@yhmtsai I think this discussion makes more sense in #1639. I don't see much relevance to this PR.

yhmtsai · 2024-12-18T13:04:11Z

yes, but if PGM does not need it, do you still need this function for something else?

MarcelKoch · 2024-12-18T13:08:25Z

I think I need at least parts of it to keep the old distributed matrix constructor around. @fritzgoebel also seems to need it for his domain decomposition matrix.

Anyway, this still sounds like a discussion for #1639.

MarcelKoch added the 1:ST:ready-for-review This PR is ready for review label Oct 23, 2024

MarcelKoch added this to the Ginkgo 1.9.0 milestone Oct 23, 2024

MarcelKoch self-assigned this Oct 23, 2024

ginkgo-bot added reg:testing This is related to testing. mod:all This touches all Ginkgo modules. labels Oct 23, 2024

MarcelKoch force-pushed the global-to-local branch from 1070d82 to 8392d07 Compare October 23, 2024 15:13

MarcelKoch mentioned this pull request Oct 23, 2024

Use Index map in PGM #1639

Open

2 tasks

fritzgoebel self-requested a review October 30, 2024 10:36

fritzgoebel mentioned this pull request Oct 31, 2024

Add a Domain decomposition Matrix format #1719

Draft

2 tasks

MarcelKoch modified the milestones: Ginkgo 1.9.0, Ginkgo 1.10.0 Dec 9, 2024

MarcelKoch added 6 commits December 16, 2024 12:15

[core] fix get_segment for const context

467e555

[dist] provide range-ids segmented by part-id

ec6cac6

[dist] add device partition

c94fc50

[imap] add local to global mapping

f188c76

[dist] fix to global mapping

ed1b4da

[dist] add round trip index mapping test

a6e1366

MarcelKoch force-pushed the global-to-local branch from 8392d07 to a6e1366 Compare December 16, 2024 14:10

MarcelKoch added 2 commits December 16, 2024 16:09

fixup! [dist] add round trip index mapping test

b9a3f24

fixup! [dist] add round trip index mapping test

c3b27c2

MarcelKoch force-pushed the global-to-local branch from c43c48a to c3b27c2 Compare December 17, 2024 11:26

MarcelKoch mentioned this pull request Dec 17, 2024

Add specific communicator for neighborhood communication #1588

Open

9 tasks

yhmtsai reviewed Dec 18, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add local to global index mapping #1707

Add local to global index mapping #1707

MarcelKoch commented Oct 23, 2024

yhmtsai left a comment

yhmtsai Dec 18, 2024

yhmtsai Dec 18, 2024

yhmtsai Dec 18, 2024

yhmtsai Dec 18, 2024

yhmtsai Dec 18, 2024

yhmtsai Dec 18, 2024

yhmtsai Dec 18, 2024

yhmtsai Dec 18, 2024

yhmtsai Dec 18, 2024

yhmtsai Dec 18, 2024

MarcelKoch commented Dec 18, 2024

yhmtsai commented Dec 18, 2024

MarcelKoch commented Dec 18, 2024

yhmtsai commented Dec 18, 2024

MarcelKoch commented Dec 18, 2024

	#ifndef GINKGO_PARTITION_HPP
	#ifndef GKO_CORE_DISTRIBUTED_DEVICE_PARTITION_HPP_

	* - partition: the global partition
	* - partition: the global partition

	std::fill_n(sizes.get_data(), num_parts, int64(0));
	std::fill_n(sizes.get_data(), num_parts, zero<int64>());

	device_partition<const LocalIndexType, const GlobalIndexType> partition,
	const device_partition<const LocalIndexType, const GlobalIndexType>& partition,

Add local to global index mapping #1707

Are you sure you want to change the base?

Add local to global index mapping #1707

Conversation

MarcelKoch commented Oct 23, 2024

yhmtsai left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MarcelKoch commented Dec 18, 2024

yhmtsai commented Dec 18, 2024

MarcelKoch commented Dec 18, 2024

yhmtsai commented Dec 18, 2024

MarcelKoch commented Dec 18, 2024