[Core] Remove push_back from ModelPart #12903

sunethwarna · 2024-12-05T06:10:14Z

📝 Description
This PR removes uses of PointerVectorSet::push_back method from the ModelPart. Followings are the modifications.

When removing entities, erase_count was not calculated correctly, hence the reserved size is wrong. This was fixed
Introduced Container class which can retireve the correct PointerVectorSet container from a given Mesh, and its std::string for nice error message output.
Refactored the methods as much as possible using templates. [Unfortunatly, Geometries container cannot be templated since it is in the ModelPart, and all the other PointerVectorSet containers are in the Mesh].

🆕 Changelog

Removes PointerVectorSet::push_back uses from the ModelPart
Fixes bugs in erase count calculation and size reservation when removing Nodes/Conditions/Elements/MasterSlaveConstraints.
Added Container class with templates to retrieve given PointerVectorSet container from a mesh.

loumalouomega · 2024-12-05T07:43:03Z

Can you use the benchmark to see if there is an improvement?

loumalouomega · 2024-12-05T07:44:12Z

I will further review once the CI passes

RiccardoRossi · 2024-12-05T09:27:47Z

kratos/includes/model_part.h

+    template<class TReturnValueType>
+    struct ReferenceGetter
+    {
+        template<class TInputValueType>


wow, this needs some explanation to me.

and the GetPointer even more...

comments are added for the use case.

RiccardoRossi · 2024-12-09T10:45:02Z

kratos/includes/model_part.h

+            });
+
+            //now add to the root model part
+            Container<TContainerType>::GetContainer(p_root_model_part->GetMesh()).insert(begin, end);


by adding always to the root all of them, a reserve is done in the pointer vector set on the root which increases its size

On the other hand, it is not always good to call shrink_to_fit as well. In the new design, if a user tries to add a range to a submodel part which is a sub-set of the one of the parent model parts, then it will not reserve the parent model parts which a super set of the range being added. I think this solves this problem partially.

In any case, I can expose the method shrink_to_fit, but I think this should be done if required only.

sorry, but then with the current implementation the size will always grow?

…k/core/model_part

sunethwarna · 2024-12-16T14:59:07Z

Hi all (@loumalouomega @roigcarlo @rubenzorrilla @RiccardoRossi @pooyan-dadvand @matekelemen @philbucher )

This is ready to be reviewed. I did the performance benchmark as well. The results are attached herewith. I did not put all the benchmarks in the benchmarks for model part because some of them are not supported by the existing implementation. Long story short -> Benchmarks shows that the new implementation is faster.

Explanation of the benchmarks:

All the benchmarks were run with 1e+2, 1e+3, 1e+4, 1e+5, 1e+6 entities being added to the model part. In the x-legend, if there is a trailing /0, that means it tries to add the entities to an empty model part / sub model part. if there is a trailin /1, that means it tries to add the entities to non-empty model part. I am here considering only 3 methods for benchmarking

CreateNewNode [BM_ModelPartCreateNewNode]
AddNodes -> Ranged insertion with begin and end [BM_ModelPartAddNodesToRootFromRange2]
AddNodes -> Adding nodes by giving a vector of ints [BM_ModelPartAddNodesToSubSubFromId1]

Serial run:

4 threads:

16 threads:

matekelemen · 2024-12-16T15:25:50Z

I have trouble understanding these graphs. Can you help me a bit plz?

sunethwarna · 2024-12-16T15:31:22Z

@matekelemen I updated the original comment with the explanations.

RiccardoRossi

blocking as there are some inefficiencies

kratos/benchmarks/model_part_benchmark.cpp

kratos/sources/model_part.cpp

RiccardoRossi · 2024-12-17T08:00:06Z

applications/HDF5Application/custom_io/hdf5_connectivities_data.cpp

+        // here we can safely use the insert with the rElements.end() as the hint
+        // because, when reading, we can assume that it was written in the
+        // sorted order within HDF5.
+        rElements.insert(rElements.end(), p_elem);


this is not efficient as you may be creating it out of order

It cannot be out of order, because we assume HDF5 files are written by the HDF5Application, hence when we write, we always write in the sorted order.

then i do not understand the role of mIds ...

wait, mIds is ordered?

i'll accept the comment, but ... what if somone writes the hdf5 "by hand"? in that case it will be enormously expensive, correct?

applications/HDF5Application/custom_io/hdf5_connectivities_data.cpp

RiccardoRossi · 2024-12-17T08:00:32Z

applications/HDF5Application/custom_io/hdf5_points_data.cpp

+        // here we can safely use the insert with the rNodes.end() as the hint
+        // because, when reading, we can assume that it was written in the
+        // sorted order within HDF5.
+        rNodes.insert(rNodes.end(), p_node);


Same comment as before

RiccardoRossi · 2024-12-17T08:02:51Z

kratos/containers/pointer_vector_set.h

@@ -1199,7 +1227,10 @@ class PointerVectorSet final
        // which is harder to guess, and cryptic. Hence, using the decltype.
        using iterator_value_type = std::decay_t<decltype(*Iterator)>;

-        if constexpr(std::is_same_v<iterator_value_type, std::remove_cv_t<TPointerType>>) {
+        if constexpr(std::is_same_v<TIteratorType, iterator> || std::is_same_v<TIteratorType, reverse_iterator>) {


@roigcarlo can you take a look into GetPointer and GetReference? these are difficult functions and i am quite afraid of their implications

kratos/includes/model_part.h

kratos/sources/model_part.cpp

RiccardoRossi · 2024-12-17T08:43:32Z

applications/HDF5Application/custom_io/hdf5_connectivities_data.cpp

+        // here we can safely use the insert with the rElements.end() as the hint
+        // because, when reading, we can assume that it was written in the
+        // sorted order within HDF5.
+        rElements.insert(rElements.end(), p_elem);


then i do not understand the role of mIds ...

RiccardoRossi · 2024-12-17T08:45:31Z

kratos/containers/pointer_vector_set.h

+            std::sort(first, last, CompareKey());
+            auto new_last = std::unique(first, last, EqualKeyTo());
+            SortedInsert(first, new_last);
+        } else {


sorry i cannot understand this ...

@Myself, this is for the case in which what i comment above is not possible ... still the comment above still holds

then i do not understand the role of mIds ...

mIds are storing the list of ids from a PVS when writing. So each rank will write its own local PVS ids which are sorted. The final list of mIds in the .h5 file won't be sorted, but it will be sorted for each partition in each rank. So when you read it also, each partition will read its portion of the mIds from the .h5 file whill will be an already sorted ids list. hence the rElements.insert(rElements.end(), p_elem) will have the hint which is valid, then PVS will do a push_back at the backend.

sorry i cannot understand this ...

This is to support iterators coming from const std::vector<NodeType::Pointer> for the insertion. There we need to sort and unique of the incoming iterators which is not allowed. So, you have to make a copy and do the sort and unique. This has additional overhead, but it cannot be avoided if we allow iterators coming from immutable containers.

yes i understand, however think at the following:

std::vector my_pointer_list .... pvs.insert(my_pointer_list.begin(), my_pointer_list.end(); //it works //but here my_pointer_list is ordered... i would not expect this. Is that something we accept? (open question...i am not sure of what would be a standardized behaviour)

indeed

const std::vector my_pointer_list .... //CONST! //pvs.insert(my_pointer_list.begin(), my_pointer_list.end(); //will int principle not compile pvs.insert(my_pointer_list.cbegin(), my_pointer_list.cend(); //i understand you will eventually remove the constness inside. Did i get it wrong? //but here my_pointer_list is NOT ordered (ok, since it was const)

but now in the PVS will i have const pointers? if i do not, aren't we removing the constness wrongly?

to me this is a difficuly programming question, but i am really not clear about what will happen and OF WHAT SHOULD HAPPEN

kratos/sources/model_part.cpp

RiccardoRossi · 2024-12-17T09:15:26Z

kratos/sources/model_part.cpp

+                << ModelPart::Container<container_type>::GetEntityName()
+                << " with Id " << *it_id << " does not exist in the root model part";
+
+            // here the hint is valid, hence this will be a simple push_back within the PVS.


oh, you mean that under the hood it will eb a push_back. Ok

RiccardoRossi · 2024-12-17T09:22:09Z

applications/HDF5Application/custom_io/hdf5_connectivities_data.cpp

+        // here we can safely use the insert with the rElements.end() as the hint
+        // because, when reading, we can assume that it was written in the
+        // sorted order within HDF5.
+        rElements.insert(rElements.end(), p_elem);


wait, mIds is ordered?

applications/HDF5Application/custom_io/hdf5_connectivities_data.cpp

RiccardoRossi · 2024-12-17T09:24:42Z

kratos/containers/pointer_vector_set.h

-        SortedInsert(first, new_last);
+        if constexpr(std::is_assignable_v<decltype(*std::declval<InputIterator>()), decltype(*std::declval<InputIterator>())>) {
+            // first sorts the input iterators and make the input unique if the iterators are assignable
+            std::sort(first, last, CompareKey());


this means that you are changing the order of the input "in place", correct?

I am not sure if this is what you would expect. I am not sure of what is the praxis in this context. Do you have it clear?

RiccardoRossi · 2024-12-17T09:30:49Z

kratos/containers/pointer_vector_set.h

+            std::sort(first, last, CompareKey());
+            auto new_last = std::unique(first, last, EqualKeyTo());
+            SortedInsert(first, new_last);
+        } else {


@Myself, this is for the case in which what i comment above is not possible ... still the comment above still holds

RiccardoRossi · 2024-12-17T09:32:54Z

kratos/utilities/embedded_skin_utility.cpp

@@ -39,8 +39,8 @@ namespace Kratos

        // Auxiliar vectors to store pointers to the new entities
        // These vectors will be use when renumbering the entities ids.
-        ModelPart::NodesContainerType new_nodes_vect;
-        ModelPart::ConditionsContainerType new_conds_vect;
+        std::vector<Node::Pointer> new_nodes_vect;


this looks ok to me but @rubenzorrilla i think this is your code, can you take a look

removing my block because of the explanation. However i stil lthink some of the comments are relvant

sunethwarna added 2 commits December 5, 2024 07:04

add additional insert and typecheck to PVS

dd02fc8

remove push_backs from ModelPart

2bf1e96

sunethwarna requested a review from a team as a code owner December 5, 2024 06:10

sunethwarna requested review from RiccardoRossi, pooyan-dadvand, roigcarlo, rubenzorrilla, philbucher, loumalouomega and matekelemen December 5, 2024 06:13

sunethwarna self-assigned this Dec 5, 2024

sunethwarna added Kratos Core C++ labels Dec 5, 2024

sunethwarna added 3 commits December 5, 2024 07:16

include reduction_utils

2c5d23f

minor

d5521ba

func name change

2f72e0c

RiccardoRossi reviewed Dec 9, 2024

View reviewed changes

sunethwarna added 9 commits December 10, 2024 12:49

fix embedded_skin_utility for Id Change

1e98ea3

allow insertion from const vectors

9c05b31

allow insertion from parent entity ranges

abbce15

fix for iterator invalidation

71ae35f

add comments

6d34602

minor signature modification

f1c4a6f

fix assign_ms_const_to_neigh_utils

35c1874

fix error msg tests

13e87e7

fix hdf5

a50fd2b

sunethwarna requested a review from a team as a code owner December 11, 2024 04:19

generalize addition from ids

8a42759

sunethwarna added 5 commits December 11, 2024 11:49

simplify

93293e0

fix ranged id addition

6adabf4

Merge remote-tracking branch 'origin/master' into pvs/remove_push_bac…

9082f96

…k/core/model_part

fix parmmg

d7d4734

add benchmarks for modelpart

d718a9a

sunethwarna requested a review from RiccardoRossi December 16, 2024 15:23

minor for comparison

4430f5e

RiccardoRossi previously requested changes Dec 17, 2024

View reviewed changes

RiccardoRossi reviewed Dec 17, 2024

View reviewed changes

sunethwarna added 4 commits December 18, 2024 10:24

add type_traits

d7e05de

allow rvalue containers in pvs insert

fedc6b3

allow rvalue containers in model part

d642295

tests for rvalue containers in PVS

ee29e90

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Core] Remove push_back from ModelPart #12903

[Core] Remove push_back from ModelPart #12903

sunethwarna commented Dec 5, 2024 •

edited

Loading

loumalouomega commented Dec 5, 2024

loumalouomega commented Dec 5, 2024

RiccardoRossi Dec 5, 2024

sunethwarna Dec 10, 2024

RiccardoRossi Dec 9, 2024

sunethwarna Dec 10, 2024 •

edited

Loading

RiccardoRossi Dec 17, 2024

sunethwarna commented Dec 16, 2024 •

edited

Loading

matekelemen commented Dec 16, 2024

sunethwarna commented Dec 16, 2024

RiccardoRossi left a comment

RiccardoRossi Dec 17, 2024

sunethwarna Dec 17, 2024

RiccardoRossi Dec 17, 2024

RiccardoRossi Dec 17, 2024

RiccardoRossi Dec 17, 2024

RiccardoRossi Dec 17, 2024

sunethwarna Dec 17, 2024

RiccardoRossi Dec 17, 2024

RiccardoRossi Dec 17, 2024

RiccardoRossi Dec 17, 2024

RiccardoRossi Dec 17, 2024

sunethwarna Dec 17, 2024

sunethwarna Dec 17, 2024

RiccardoRossi Dec 17, 2024

RiccardoRossi Dec 17, 2024

sunethwarna Dec 17, 2024

RiccardoRossi Dec 17, 2024

RiccardoRossi Dec 17, 2024

RiccardoRossi Dec 17, 2024

RiccardoRossi Dec 17, 2024

[Core] Remove push_back from ModelPart #12903

Are you sure you want to change the base?

[Core] Remove push_back from ModelPart #12903

Conversation

sunethwarna commented Dec 5, 2024 • edited Loading

loumalouomega commented Dec 5, 2024

loumalouomega commented Dec 5, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sunethwarna Dec 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sunethwarna commented Dec 16, 2024 • edited Loading

matekelemen commented Dec 16, 2024

sunethwarna commented Dec 16, 2024

RiccardoRossi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sunethwarna commented Dec 5, 2024 •

edited

Loading

sunethwarna Dec 10, 2024 •

edited

Loading

sunethwarna commented Dec 16, 2024 •

edited

Loading