Note: The project that this work supported has concluded.
The purpose of this project was to write a network socket that would allow the transportation of TCP data at ~10Gb/s in order to support on-going research in distributed systems like Apache Spark, Apache Kafka, and Apache Mesos.
This work concluded with approximately: 8Gb/s throughput from source to sink, which is close enough to the goal to support the research (see Supported Research).
This socket supported the following research projects:
- High-Throughput Apache Spark Streaming
- On GitHub (https://github.com/LeStarch/hyper-spark)
- At Apache Con Europe (http://sched.co/400u)
- High-Throughput Kafka and Kafka in HPC https://github.com/LeStarch/kafka-benchmarking