Enable PIO with mpiuni #205
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| This is a dummy file needed to satisfy PIO's FindMPISERIAL.cmake. | ||
|
|
||
| This is needed because, when building with mpiuni, we tell PIO that this | ||
| mpiuni directory is the location of mpi-serial (since we use mpiuni for | ||
| the same purpose as mpi-serial). But for FindMPISERIAL.cmake to succeed, | ||
| it needs to find a libmpi-serial.a in the mpi-serial directory; hence, | ||
| we put this file here to trick it into thinking that this is truly an | ||
| mpi-serial installation. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -55,8 +55,28 @@ static int num_attr = 1,mpi_tag_ub = 100000000; | |
|
|
||
| /* | ||
| To avoid problems with prototypes to the system memcpy() it is duplicated here | ||
|
|
||
| This version also supports checking for MPI_IN_PLACE | ||
| */ | ||
| int MPIUNI_Memcpy(void *a,const void* b,int n,enum CheckForMPIInPlace_Flag check_flag) { | ||
| switch(check_flag) { | ||
| case CHECK_FOR_MPI_IN_PLACE_NONE: | ||
| // No pre-check in this case; proceed to the actual memcpy | ||
| break; | ||
| case CHECK_FOR_MPI_IN_PLACE_SOURCE: | ||
| if (b == MPI_IN_PLACE) { | ||
| // If the source is MPI_IN_PLACE, do nothing | ||
| return 0; | ||
| } | ||
| break; | ||
| case CHECK_FOR_MPI_IN_PLACE_DEST: | ||
| if (a == MPI_IN_PLACE) { | ||
| // If the dest is MPI_IN_PLACE, do nothing | ||
| return 0; | ||
| } | ||
| break; | ||
| } | ||
|
|
||
|
Comment on lines
+61
to
+79
There was a problem hiding this comment. Most of the changes I made to mpiuni are additions, but this one is a behavior change. I have done a fair amount of testing of this change and it feels pretty safe to me, but I'd be happy to have a second set of eyes on it. |
||
| int i; | ||
| char *aa= (char*)a; | ||
| char *bb= (char*)b; | ||
|
|
@@ -179,6 +199,84 @@ int Petsc_MPI_Finalize(void) | |
| return 0; | ||
| } | ||
|
|
||
| int ESMC_MPI_Alltoallw(void *sendbuf, int *sendcounts, int *sdispls, | ||
| MPI_Datatype *sendtypes, void *recvbuf, int *recvcounts, | ||
| int *rdispls, MPI_Datatype *recvtypes, MPI_Comm comm) | ||
| { | ||
| // Since we are only implementing this for the single-processor case, the counts, displs | ||
| // and types arguments should all have length 1. We assume that's the case in this | ||
| // implementation. | ||
|
|
||
| // Displacements are not implemented so return an error code if they are non-zero | ||
| if (sdispls[0] != 0 || rdispls[0] != 0) { | ||
| return MPI_ERR_INTERN; | ||
| } | ||
|
|
||
| MPIUNI_Memcpy(recvbuf, sendbuf, sendcounts[0]*sendtypes[0], CHECK_FOR_MPI_IN_PLACE_SOURCE); | ||
| return MPI_SUCCESS; | ||
| } | ||
|
|
||
| int ESMC_MPI_Scatterv(void *sendbuf, int *sendcounts, int *displs, | ||
| MPI_Datatype sendtype, void *recvbuf, int recvcount, | ||
| MPI_Datatype recvtype, int root, MPI_Comm comm) | ||
| { | ||
| // Since we are only implementing this for the single-processor case, the sendcounts and | ||
| // displs arguments should have length 1. We assume that's the case in this | ||
| // implementation. | ||
|
|
||
| // Displacements are not implemented so return an error code if they are non-zero | ||
| if (displs[0] != 0) { | ||
| return MPI_ERR_INTERN; | ||
| } | ||
|
|
||
| MPIUNI_Memcpy(recvbuf, sendbuf, sendcounts[0]*sendtype, CHECK_FOR_MPI_IN_PLACE_DEST); | ||
| return MPI_SUCCESS; | ||
| } | ||
|
|
||
| int ESMC_MPI_Type_create_hvector(int count, int blocklength, MPI_Aint stride, | ||
|
There was a problem hiding this comment. Note that I ignore stride here: it seems like, for mpiuni, the important thing about type creation is to end up with a value that reflcts the size of the resulting type, and from some experimentation with this MPI function with openmpi, it seems like the MPI_Type_size of the result is independent of the stride. However, this function is the one that I'm least certain of, so would be happy to have a second set of eyes on this. |
||
| MPI_Datatype oldtype, MPI_Datatype *newtype) | ||
| { | ||
| // Note mpiuni's definition of each datatype as sizeof(raw-type). | ||
| // | ||
| // From some experimentation with a real MPI library, the MPI_Type_size of newtype is | ||
| // independent of the value of stride. Since the MPI_Datatype in mpiuni is just the size | ||
| // of the datatype, we ignore the possible complexity of stride in this implementation. | ||
| *newtype = count*blocklength*oldtype; | ||
| return MPI_SUCCESS; | ||
| } | ||
|
|
||
| int ESMC_MPI_Type_create_indexed_block(int count, int blocklength, | ||
| const int array_of_displacements[], | ||
|
There was a problem hiding this comment. Note that I ignore array_of_displacements here: it seems like, for mpiuni, the important thing about type creation is to end up with a value that reflects the size of the resulting type, and from some experimentation with this MPI function with openmpi, it seems like the MPI_Type_size of the result is independent of the displacements. (I originally had a check that these displacements were evenly spaced, but removed this check (in commit 59a996d) because it seems unnecessary if what we care about is the size of the result.) But I would be happy to have a second set of eyes on this. |
||
| MPI_Datatype oldtype, | ||
| MPI_Datatype *newtype) | ||
| { | ||
| // Note mpiuni's definition of each datatype as sizeof(raw-type). | ||
| // | ||
| // From some experimentation with a real MPI library, the MPI_Type_size of newtype is | ||
| // independent of the values in array_of_displacements. Since the MPI_Datatype in mpiuni | ||
| // is just the size of the datatype, we ignore the possible complexity of | ||
| // array_of_displacements in this implementation. | ||
| *newtype = count*blocklength*oldtype; | ||
| return MPI_SUCCESS; | ||
| } | ||
|
|
||
| int ESMC_MPI_Type_hvector(int count, int blocklength, MPI_Aint stride, | ||
| MPI_Datatype oldtype, MPI_Datatype *newtype) | ||
| { | ||
| // MPI_Type_hvector is a deprecated version of MPI_Type_create_hvector; the only | ||
| // difference is in how stride is specified (bytes vs. elements); since we ignore stride | ||
| // in our implementation of MPI_Type_create_hvector, we can use the same implementation | ||
| // for both. | ||
| return ESMC_MPI_Type_create_hvector(count, blocklength, stride, oldtype, newtype); | ||
| } | ||
|
|
||
| int ESMC_MPI_Type_size(MPI_Datatype datatype, int *size) | ||
| { | ||
| // Note that, conveniently, mpiuni defines each datatype as sizeof(raw-type) | ||
| *size = datatype; | ||
| return MPI_SUCCESS; | ||
| } | ||
|
|
||
| #if !defined (ESMF_OS_MinGW) | ||
| // POSIX version | ||
| double ESMC_MPI_Wtime(void) | ||
|
|
@@ -403,7 +501,7 @@ void MPIUNI_STDCALL mpi_allreduce(void *sendbuf,void *recvbuf,int *count,int *da | |
| *ierr = MPI_ERR_OP; | ||
| return; | ||
| } | ||
| MPIUNI_Memcpy(recvbuf,sendbuf,(*count)*MPIUNI_DATASIZE[*datatype],CHECK_FOR_MPI_IN_PLACE_SOURCE); | ||
| *ierr = MPI_SUCCESS; | ||
| } | ||
| void MPIUNI_STDCALL mpi_allreduce_(void *sendbuf,void *recvbuf,int *count,int *datatype,int *op,int *comm,int *ierr) | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note that I am setting PIO_ENABLE_TESTS=OFF.
This means that you can't run 'make tests', 'ctest' or 'make check' on the PIO installation. Is that a problem?
This was needed for the mpiuni build. I am leaving this at its default (ON) for the non-mpiuni build, though if people feel it's never used with the internal PIO build, we could just set it to OFF always.
I didn't spend much time digging into the problems I had building the tests. If people feel it's important to keep these tests enabled, I can dig into this more.