Call BranchMorpher after dw deduplication #331

kaushikcfd · 2022-05-26T02:40:20Z

Some timings on the graphs (~36k nodes) implemented in https://github.com/illinois-ceesd/drivers_y2-isolator:

With this patch:

$ time python -O -m mpi4py isolator_injection_run.py -i run_params_combustion.yaml -r restart_data/isolator_init-000000 --log --lazy
real    2m48.997s
user    2m51.500s
sys     0m10.912s

Without this patch:

$ time python -O -m mpi4py isolator_injection_run.py -i run_params_combustion.yaml -r restart_data/isolator_init-000000 --log --lazy                                                                                                                                                                                                                                             
real    3m20.631s                                                                                                                                                                                                                              
user    3m22.686s                                                                                                      
sys     0m11.08

pytato/transform/__init__.py

inducer · 2022-05-27T02:36:44Z

pytato/transform/__init__.py

+
+class EqualBranchesReuser(CopyMapper):
+    """
+    A mapper that replaces equal segments of graphs with identical objects.


Please add more context to the docstring, contrastng with CachedMapper and specifically mentioning that this is about equality after the mapper applies, and describe the resulting complexity reduction.

inducer

Thanks! A few more notes/thoughts below.

inducer · 2022-06-05T23:44:14Z

pytato/transform/__init__.py

+    A mapper that reuses the same object instances for equal segments of
+    graphs.


Suggested change

A mapper that reuses the same object instances for equal segments of

graphs.

A mapper that reuses the same object instances for segments of

graphs that become equal after processing by subclasses of this mapper,

where the equality test happens immediately after a new node is created.

inducer · 2022-06-05T23:44:51Z

pytato/transform/__init__.py

+        :class:`pytato.Array` nodes. However, they differ at the point where
+        two array expressions are compared. :class:`CopyMapper` compares array
+        expressions before the expressions are mapped i.e. repeatedly comparing
+        equal array expressions but unequal instances, and because of this it


Suggested change

equal array expressions but unequal instances, and because of this it

different instances representing equal array expressions, and, because of this, it

inducer · 2022-06-05T23:48:42Z

pytato/transform/__init__.py

+        hand, :class:`PostMapEqualNodeReuser` has linear complexity in the
+        number of nodes in the number of array expressions as the larger mapped
+        expressions already contain same instances for the predecessors,
+        resulting in a cheaper equality comparison overall.


I think what you want to say instead here is that this avoids building potentially large repeated subexpexpressions in the first place, rather than having to (expensively) merge them after the fact.

inducer · 2022-06-05T23:53:46Z

pytato/transform/__init__.py

-    return array_or_names
+    # many paths in the DAG might be semantically equivalent after DWs are
+    # deduplicated => morph them
+    return PostMapEqualNodeReuser()(array_or_names)


Hmm, wait. If it's used like this, I would argue that that's equivalent to just using a CopyMapper. I figured the data-wrapper-deduplicator should inherit from this to avoid building large identical subtrees in the first place.

Added test_post_map_equal_node_reuser_intestine to investigate whether this is equivalent to a CopyMapper, and at least in that example, it appears to be. It remains to figure out where the difference in execution time comes from.

inducer · 2022-06-05T23:53:56Z

pytato/transform/__init__.py

@@ -1782,8 +1829,11 @@ def cached_data_wrapper_if_present(ary: ArrayOrNames) -> ArrayOrNames:
                len(data_wrapper_cache),
                data_wrappers_encountered - len(data_wrapper_cache))

-    return array_or_names
+    # many paths in the DAG might be semantically equivalent after DWs are
+    # deduplicated => morph them


Still: "morph" isn't good terminology.

kaushikcfd force-pushed the branchmorpher branch from 3021d89 to 9a72b32 Compare May 26, 2022 02:46

inducer reviewed May 26, 2022

View reviewed changes

pytato/transform/__init__.py Outdated Show resolved Hide resolved

pytato/transform/__init__.py Outdated Show resolved Hide resolved

pytato/transform/__init__.py Outdated Show resolved Hide resolved

kaushikcfd force-pushed the branchmorpher branch from 9a72b32 to 3305071 Compare May 26, 2022 23:57

inducer reviewed May 27, 2022

View reviewed changes

kaushikcfd force-pushed the branchmorpher branch from 3305071 to 6fd5162 Compare May 27, 2022 17:54

inducer reviewed Jun 5, 2022

View reviewed changes

Call PostMapEqualNodesReuser after dw deduplication

3a01309

kaushikcfd force-pushed the branchmorpher branch from 6fd5162 to 3a01309 Compare May 22, 2023 13:22

test_post_map_equal_node_reuser_intestine

a3c9d06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Call BranchMorpher after dw deduplication #331

Call BranchMorpher after dw deduplication #331

kaushikcfd commented May 26, 2022

inducer May 27, 2022

kaushikcfd May 27, 2022

inducer left a comment

inducer Jun 5, 2022

inducer Jun 5, 2022

inducer Jun 5, 2022

inducer Jun 5, 2022

inducer May 23, 2023

inducer Jun 5, 2022

		A mapper that reuses the same object instances for equal segments of
		graphs.

-    A mapper that reuses the same object instances for equal segments of
-    graphs.
+    A mapper that reuses the same object instances for segments of
+    graphs that become equal after processing by subclasses of this mapper,
+    where the equality test happens immediately after a new node is created.

	equal array expressions but unequal instances, and because of this it
	different instances representing equal array expressions, and, because of this, it

Call BranchMorpher after dw deduplication #331

Are you sure you want to change the base?

Call BranchMorpher after dw deduplication #331

Conversation

kaushikcfd commented May 26, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

inducer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment