MsQuic + MPI

This is a proof-of-concept sandbox for using MsQuic with MPI.

• MsQuic API Documentation: link
• Borrowed code from sample.c (see License)

This is meant for a desktop application that can also be run via command line.

Build Notes

• Built successfully with VS2019. Should be able to restore nuget packages and build without problems.
• Requires MPI installed to test, installers here.
• Run mpiexec -n 2 QuicMPISandbox.exe through command line to test on machine.
• In main.cpp's main function:
  • Use RunQuicOnlyTest() to run without MPI involved.
• Added in .pem files for testing, these aren't used anywhere and is only for testing. Don't use in any production code.
• If running via debugging in VS, make sure that the Debugging->Working Directory setting is pointed to the output directory.

Notes

My goal is to pass all data to a single writer rank to write to a local SQLite database.

Reasons for exploring MsQuic instead of just using MPI to pass data between ranks:

• There are data limitations to message passing via MPI, I ran into a limit of 4mb per rank.
• The data per rank that needs to be passed to the writer will definitely exceed 4mb, minimum will be 5mb.
• The specs of the machine(s) running the application isn't under my control so this has to work regardless of hardware.

Data being sent via StreamSend must be allocated on the heap and freed only after QUIC_STREAM_EVENT_SEND_COMPLETE is received in the stream callback otherwise there will be race conditions with the data being freed before it's fully sent. I'm handling this by passing in a DataPacket pointer into the context which is the Event->SEND_COMPLETE.ClientContext accessible when that send complete event is raised.

TODO

• Figure out how to make the program wait until all other ranks are done with their Quic connection & streams.
• Set up the sending of test data
  • Currently sends but program appears to sometimes exit before the receiver rank can process it.
  • Need to figure out conversion from char* to uint8_t* and back again. Uncertain if something happens with the buffer during the sending process.
  • Test data consisting of various ints, floats, and chars injected into a std::stringstream.
• Figure out a better way to create testing SSL certificates other than adding to repository.
• Clean up artifacts (.dll & .pem files) on project/solution clean.
• Centralize the logging s.t. timestamp and rank # are added with each log.
• Add in an iterative MPI structure used in the actual simulations (see stack overflow question)
• Set up something to check whether listener connection has finished starting and streams can be created.
• Add storage to local txt file from the receiver rank as part of the incoming data processing, also include results from the receiver rank itself.
• Add timing mechanisms to check:
  • Time taken to send.
  • Time taken between sending and receiving.
  • Time taken to process the data.
  • Time taken to wait for all streams to close.

Testing

Each packet is 44 bytes, all times below are in ms.

	TimeToStartStream					TimeToReceive					TimeToProcess					TimeToSend
# of Packets	1	1000	5000	10000		1	1000	5000	10000		1	1000	5000	10000		1	1000	5000	10000
Average	1.43	1.49	1.72	1.65		2.47	3.19	8.49	9.31		0	0.11	2.06	2.45		1.54	1.53	2.44	2.6
Min	1	1	1	1		2	2	4	6		0	0	0	0		1	1	1	1
Max	3	3	3	3		4	7	23	16		0	1	6	6		3	3	5	4
Count	100	100	100	100		100	100	100	100		100	100	100	100		100	100	100	100
Sum	143	149	172	165		247	319	849	931		0	11	206	245		154	153	244	260

License

• Code from MsQuic is open sourced under the MIT license, see here for full license.