Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a benchmark which uses Google benchmark library #1392

Merged
merged 94 commits into from
Nov 14, 2024
Merged
Show file tree
Hide file tree
Changes from 73 commits
Commits
Show all changes
94 commits
Select commit Hold shift + click to select a range
3eefad0
Add attestation steps for packaging
egecetin May 9, 2024
cc997aa
tab to spaces
egecetin May 9, 2024
8235841
update permissions
egecetin May 9, 2024
3fd4e01
Update README for verify procedure
egecetin May 9, 2024
202fb59
fix
egecetin May 9, 2024
ba8be94
version to commit hash
egecetin May 11, 2024
566566d
Use google benchmark
egecetin May 11, 2024
fe7d42d
add stats
egecetin May 11, 2024
bcd9d70
update comment
egecetin May 11, 2024
943f04d
change default opt
egecetin May 11, 2024
5d346a5
add workflow
egecetin May 11, 2024
e770341
test package install
egecetin May 11, 2024
345b5e7
forgot to build :)
egecetin May 11, 2024
8ed27cf
fix path
egecetin May 11, 2024
16fea77
fix path
egecetin May 11, 2024
82a8d93
fix path again
egecetin May 11, 2024
b03aabd
pre-commit fixes
egecetin May 11, 2024
50e76bb
Fix cmake format + disable pull requests
egecetin May 11, 2024
0b56c68
disable verbose
egecetin May 11, 2024
ad48e8a
add dev for testing
egecetin May 12, 2024
f0aab9d
add ignore files
egecetin May 12, 2024
9520bb0
upload as artifact for test
egecetin May 12, 2024
4be134f
upload results as artifact to generate charts manually
egecetin May 12, 2024
88f9f4a
Merge branch 'dev' into attestation
egecetin May 27, 2024
96e9c5d
Merge branch 'dev' into benchmark
egecetin May 27, 2024
ea31445
change action version format
egecetin May 27, 2024
dba91ea
Merge remote-tracking branch 'upstream/dev' into attestation
egecetin Jun 27, 2024
62d8105
update
egecetin Jun 27, 2024
dda4c8f
remove ubuntu 18.04
egecetin Jun 27, 2024
ebf5d75
move permissions
egecetin Jun 27, 2024
bb4b18d
should be package dir
egecetin Jun 27, 2024
d8f505b
remove wildcard
egecetin Jun 27, 2024
746a174
Merge remote-tracking branch 'upstream/dev' into attestation
egecetin Jul 9, 2024
44d309d
Merge pull request #11 from egecetin/attestation
egecetin Jul 9, 2024
bb959b2
bump version str
egecetin Jul 9, 2024
842680d
Merge remote-tracking branch 'upstream/dev' into benchmark
egecetin Jul 13, 2024
590be53
fix format issues
egecetin Jul 13, 2024
10169fa
move benchmark
egecetin Jul 13, 2024
17d5e8e
compile success
egecetin Jul 13, 2024
7bb2070
minor fixes
egecetin Jul 13, 2024
0642949
Merge remote-tracking branch 'origin/master' into benchmark
egecetin Aug 3, 2024
d76289a
Merge remote-tracking branch 'origin/master' into benchmark
egecetin Aug 30, 2024
4657572
remove unnecessary changes
egecetin Aug 30, 2024
d8db674
not related
egecetin Aug 30, 2024
802e4f7
test benchmark workflow is buildable
egecetin Aug 30, 2024
3564c8b
Add PRs temporarily
egecetin Aug 30, 2024
227a661
fix errors
egecetin Aug 30, 2024
b6575c9
fix
egecetin Aug 30, 2024
21e85d9
no need matrix
egecetin Aug 30, 2024
1022065
use bencher
egecetin Aug 30, 2024
915817d
remove bencher
egecetin Aug 30, 2024
2881040
minor update
egecetin Aug 30, 2024
4b0c896
Add write and crafting benchmarks
egecetin Aug 30, 2024
00997b6
update function names
egecetin Aug 31, 2024
7795cfc
add info to benchmarks
egecetin Aug 31, 2024
7107fd6
update path
egecetin Aug 31, 2024
ea982b7
update python
egecetin Aug 31, 2024
caf6c64
format fix
egecetin Aug 31, 2024
72f6bd2
fix typos
egecetin Aug 31, 2024
b6d7e98
simplify
egecetin Aug 31, 2024
5e8e056
use pps
egecetin Sep 1, 2024
27b4619
use FetchContent to get benchmark
egecetin Sep 1, 2024
95b6562
Rename benchmark in example because same name with lib
egecetin Sep 1, 2024
d6e90c0
format cmake
egecetin Sep 1, 2024
f9e4cfb
Merge remote-tracking branch 'upstream/dev' into benchmark
egecetin Oct 3, 2024
b9aad7d
simplify parsedPacket process
egecetin Oct 10, 2024
90faf97
move benchmarks to examples
egecetin Oct 10, 2024
66b3e14
set cross-compiling
egecetin Oct 11, 2024
1588009
Revert "set cross-compiling"
egecetin Oct 11, 2024
a8a2a46
change readme and enable benchmark for only linux
egecetin Oct 11, 2024
ed12ae4
revert unnecessary change
egecetin Oct 11, 2024
80152bf
Merge branch 'dev' into benchmark
egecetin Oct 12, 2024
6b861d7
Merge branch 'dev' into benchmark
seladb Oct 12, 2024
58701ed
improve readme
egecetin Oct 14, 2024
557839a
fix for comments
egecetin Oct 14, 2024
7ee9f25
table header
egecetin Oct 14, 2024
1ecf4d6
pre-commit fix
egecetin Oct 14, 2024
bc5be8a
remove PCAPPP_USE_GOOGLEBENCHMARK
egecetin Oct 14, 2024
bb0e73d
Merge remote-tracking branch 'upstream/dev' into benchmark
egecetin Oct 14, 2024
9dd6134
test cross-compile macos
egecetin Oct 14, 2024
f5d1a8b
use additional flag for macos13
egecetin Oct 14, 2024
de4ae9c
add same flag to tutorial build
egecetin Oct 14, 2024
b1f00ab
disable google bench lib install
egecetin Oct 15, 2024
7774cb6
add comment to cmake
egecetin Oct 15, 2024
10f40c2
test macos from CI
egecetin Oct 15, 2024
419c63e
Merge remote-tracking branch 'origin/master' into benchmark
egecetin Nov 3, 2024
a6a9cb1
auto detect C++14
egecetin Nov 3, 2024
fe12995
remove additional flags
egecetin Nov 3, 2024
9fc6e6e
fetchcontent_makeavailable requires 3.14
egecetin Nov 3, 2024
bc4ab17
fix macos arch difference
egecetin Nov 3, 2024
948d1cf
add warning message for C++14
egecetin Nov 3, 2024
602ddd9
fix logic
egecetin Nov 3, 2024
800f287
Merge branch 'dev' into benchmark
egecetin Nov 3, 2024
e8efae5
Merge branch 'dev' into benchmark
egecetin Nov 11, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/build_and_test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ jobs:
${{ matrix.image }}-ccache

- name: Configure PcapPlusPlus
run: cmake -DLIGHT_PCAPNG_ZSTD=${{ matrix.config-zstd }} -DPCAPPP_BUILD_COVERAGE=ON ${{ matrix.additional-flags }} -S . -B "$BUILD_DIR"
run: cmake -DPCAPPP_USE_GOOGLEBENCHMARK=ON -DLIGHT_PCAPNG_ZSTD=${{ matrix.config-zstd }} -DPCAPPP_BUILD_COVERAGE=ON ${{ matrix.additional-flags }} -S . -B "$BUILD_DIR"

- name: Build PcapPlusPlus
run: cmake --build "$BUILD_DIR" -j
Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
*.pcapng.zst
*.pcapng.zstd
*.cap
*.snoop

#Bin and Obj directories
**/Bin
Expand Down
1 change: 1 addition & 0 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,7 @@ option(PCAPPP_BUILD_TESTS "Build Tests" ${PCAPPP_MAIN_PROJECT})
option(PCAPPP_BUILD_COVERAGE "Generate Coverage Report" OFF)
option(PCAPPP_BUILD_FUZZERS "Build Fuzzers binaries" OFF)
option(PCAPPP_BUILD_REPRODUCIBLE "Build a reproducible version" OFF)
option(PCAPPP_USE_GOOGLEBENCHMARK "Build benchmarks using Google benchmark" OFF)

option(BUILD_SHARED_LIBS "Build using shared libraries" OFF)

Expand Down
35 changes: 31 additions & 4 deletions Examples/PcapPlusPlus-benchmark/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -1,12 +1,39 @@
add_executable(benchmark benchmark.cpp)
add_executable(BenchmarkExample benchmark.cpp)

target_link_libraries(benchmark PUBLIC PcapPlusPlus::Pcap++)
target_link_libraries(BenchmarkExample PUBLIC PcapPlusPlus::Pcap++)

set_target_properties(benchmark PROPERTIES RUNTIME_OUTPUT_DIRECTORY "${PCAPPP_BINARY_EXAMPLES_DIR}")
set_target_properties(BenchmarkExample PROPERTIES RUNTIME_OUTPUT_DIRECTORY "${PCAPPP_BINARY_EXAMPLES_DIR}")

if(PCAPPP_USE_GOOGLEBENCHMARK)
include(FetchContent)
egecetin marked this conversation as resolved.
Show resolved Hide resolved

# Fetch Google Benchmark
fetchcontent_declare(
benchmark
GIT_REPOSITORY https://github.com/google/benchmark.git
GIT_TAG v1.9.0)

# Disable testing for benchmark
set(BENCHMARK_ENABLE_TESTING OFF)
fetchcontent_makeavailable(benchmark)

add_executable(BenchmarkExampleGoogle benchmark-google.cpp)

target_link_libraries(BenchmarkExampleGoogle PUBLIC PcapPlusPlus::Pcap++ benchmark::benchmark)

set_target_properties(BenchmarkExampleGoogle PROPERTIES RUNTIME_OUTPUT_DIRECTORY "${PCAPPP_BINARY_EXAMPLES_DIR}")
endif()

if(PCAPPP_INSTALL)
install(
TARGETS benchmark
TARGETS BenchmarkExample
EXPORT PcapPlusPlusTargets
RUNTIME DESTINATION ${PCAPPP_INSTALL_BINDIR})

if(PCAPPP_USE_GOOGLEBENCHMARK)
install(
TARGETS BenchmarkExampleGoogle
EXPORT PcapPlusPlusTargets
RUNTIME DESTINATION ${PCAPPP_INSTALL_BINDIR})
endif()
endif()
9 changes: 9 additions & 0 deletions Examples/PcapPlusPlus-benchmark/README.md
seladb marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
Expand Up @@ -6,3 +6,12 @@ This is a benchmark application used for measuring PcapPlusPlus performance. It
See this page for more details: https://pcapplusplus.github.io/docs/benchmark

This application currently compiles on Linux only (where benchmark was running on)

Additionally, there is an integration of Google Benchmark library for benchmarking. For enabling, configure the library with `-DPCAPPP_USE_GOOGLEBENCHMARK=ON`. Each benchmark can be affected from different sources. You can check the table below for more information. If you have performance critical applications which will use PcapPlusPlus, please benchmark it in your environment to get better and more precise results. Also using different pcaps, both in terms of size (larger is better to prevent open/close overhead) and protocol/session variety, can also help to give idea about performance of PcapPlusPlus in your environment.
egecetin marked this conversation as resolved.
Show resolved Hide resolved

| Benchmark | Operation | Can be affected |
egecetin marked this conversation as resolved.
Show resolved Hide resolved
|:-----------------:|:-------------:|:-----------------:|
| BM_PcapFileRead | Read | CPU + Disk (Read) |
| BM_PcapFileWrite | Write | CPU + Disk (Write)|
| BM_PacketParsing | Read + Parse | CPU + Disk (Read) |
| BM_PacketCrafting | Craft | CPU |
263 changes: 263 additions & 0 deletions Examples/PcapPlusPlus-benchmark/benchmark-google.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,263 @@
#include <Packet.h>
#include <PcapFileDevice.h>
#include <PcapPlusPlusVersion.h>

#include <EthLayer.h>
#include <IPv4Layer.h>
#include <IPv6Layer.h>
#include <TcpLayer.h>
#include <UdpLayer.h>

#include <benchmark/benchmark.h>

#include <iostream>

static std::string pcapFileName = "";

static void BM_PcapFileRead(benchmark::State& state)
{
// Open the pcap file for reading
pcpp::PcapFileReaderDevice reader(pcapFileName);
if (!reader.open())
{
state.SkipWithError("Cannot open pcap file for reading");
return;
}

size_t totalBytes = 0;
size_t totalPackets = 0;
pcpp::RawPacket rawPacket;
for (auto _ : state)
{
if (!reader.getNextPacket(rawPacket))
{
// If the rawPacket is empty there should be an error
if (totalBytes == 0)
{
state.SkipWithError("Cannot read packet");
break;
seladb marked this conversation as resolved.
Show resolved Hide resolved
}
else // If the rawPacket is not empty, it means the file is over
{
state.PauseTiming();
reader.close();
reader.open();
seladb marked this conversation as resolved.
Show resolved Hide resolved
state.ResumeTiming();
}
}

++totalPackets;
totalBytes += rawPacket.getRawDataLen();
}

state.SetBytesProcessed(totalBytes);
state.SetItemsProcessed(totalPackets);
}
BENCHMARK(BM_PcapFileRead);

static void BM_PcapFileWrite(benchmark::State& state)
{
// Open the pcap file for writing
pcpp::PcapFileWriterDevice writer("benchmark-output.pcap");
if (!writer.open())
{
state.SkipWithError("Cannot open pcap file for writing");
return;
}

pcpp::Packet packet;
pcpp::EthLayer ethLayer(pcpp::MacAddress("00:00:00:00:00:00"), pcpp::MacAddress("00:00:00:00:00:00"));
pcpp::IPv4Layer ip4Layer(pcpp::IPv4Address("192.168.0.1"), pcpp::IPv4Address("192.168.0.2"));
pcpp::TcpLayer tcpLayer(12345, 80);

packet.addLayer(&ethLayer);
packet.addLayer(&ip4Layer);
packet.addLayer(&tcpLayer);
packet.computeCalculateFields();

size_t totalBytes = 0;
size_t totalPackets = 0;
for (auto _ : state)
{
// Write packet to file
writer.writePacket(*(packet.getRawPacket()));

// Count total bytes and packets
++totalPackets;
totalBytes += packet.getRawPacket()->getRawDataLen();
}

// Set statistics to the benchmark state
state.SetBytesProcessed(totalBytes);
state.SetItemsProcessed(totalPackets);
}
BENCHMARK(BM_PcapFileWrite);

static void BM_PacketParsing(benchmark::State& state)
{
// Open the pcap file for reading
size_t totalBytes = 0;
size_t totalPackets = 0;
pcpp::PcapFileReaderDevice reader(pcapFileName);
if (!reader.open())
{
state.SkipWithError("Cannot open pcap file for reading");
return;
}

pcpp::RawPacket rawPacket;
for (auto _ : state)
{
if (!reader.getNextPacket(rawPacket))
{
// If the rawPacket is empty there should be an error
if (totalBytes == 0)
{
state.SkipWithError("Cannot read packet");
break;
}
else // If the rawPacket is not empty, it means the file is over
{
state.PauseTiming();
reader.close();
reader.open();
state.ResumeTiming();
continue;
}
}
seladb marked this conversation as resolved.
Show resolved Hide resolved

// Parse packet
pcpp::Packet parsedPacket(&rawPacket);

// Use parsedPacket to prevent compiler optimizations
assert(parsedPacket.getFirstLayer());

// Count total bytes and packets
++totalPackets;
totalBytes += rawPacket.getRawDataLen();
}

// Set statistics to the benchmark state
state.SetBytesProcessed(totalBytes);
state.SetItemsProcessed(totalPackets);
}
BENCHMARK(BM_PacketParsing);

static void BM_PacketCrafting(benchmark::State& state)
{
size_t totalBytes = 0;
size_t totalPackets = 0;

for (auto _ : state)
{
uint8_t randNum = static_cast<uint8_t>(rand() % 256);

// Generate random MAC addresses
pcpp::MacAddress srcMac(randNum, randNum, randNum, randNum, randNum, randNum);
pcpp::MacAddress dstMac(randNum, randNum, randNum, randNum, randNum, randNum);

// Craft packet
std::unique_ptr<pcpp::EthLayer> ethLayer(new pcpp::EthLayer(srcMac, dstMac));

std::unique_ptr<pcpp::IPv4Layer> ipv4Layer(nullptr);
std::unique_ptr<pcpp::IPv6Layer> ipv6Layer(nullptr);

std::unique_ptr<pcpp::TcpLayer> tcpLayer(nullptr);
std::unique_ptr<pcpp::UdpLayer> udpLayer(nullptr);
seladb marked this conversation as resolved.
Show resolved Hide resolved

// Randomly choose between IPv4 and IPv6
if (randNum % 2)
{
ipv4Layer.reset(new pcpp::IPv4Layer(randNum, randNum));
}
else
{
std::array<uint8_t, 16> srcIP = { randNum, randNum, randNum, randNum, randNum, randNum, randNum, randNum,
randNum, randNum, randNum, randNum, randNum, randNum, randNum, randNum };
std::array<uint8_t, 16> dstIP = { randNum, randNum, randNum, randNum, randNum, randNum, randNum, randNum,
randNum, randNum, randNum, randNum, randNum, randNum, randNum, randNum };

ipv6Layer.reset(new pcpp::IPv6Layer(srcIP, dstIP));
}

// Randomly choose between TCP and UDP
if (randNum % 2)
{
tcpLayer.reset(new pcpp::TcpLayer(randNum % 65536, randNum % 65536));
}
else
{
udpLayer.reset(new pcpp::UdpLayer(randNum % 65536, randNum % 65536));
}

// Add layers to the packet
pcpp::Packet packet;
packet.addLayer(ethLayer.get());
if (ipv4Layer)
{
packet.addLayer(ipv4Layer.get());
}
else
{
packet.addLayer(ipv6Layer.get());
}

if (tcpLayer)
{
packet.addLayer(tcpLayer.get());
}
else
{
packet.addLayer(udpLayer.get());
}

packet.computeCalculateFields();

// Count total bytes and packets
++totalPackets;
totalBytes += packet.getRawPacket()->getRawDataLen();
}

// Set statistics to the benchmark state
state.SetBytesProcessed(totalBytes);
state.SetItemsProcessed(totalPackets);
}
BENCHMARK(BM_PacketCrafting);

int main(int argc, char** argv)
{
// Initialize the benchmark
benchmark::Initialize(&argc, argv);

// Parse command line arguments to find the pcap file name
for (int idx = 1; idx < argc; ++idx)
{
if (strcmp(argv[idx], "--pcap-file") == 0)
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since --pcap-file is a mandatory parameter (and the only param for now), maybe we don't need the param itself and can run the benchmark like this:

.\BenchmarkExampleGoogle.exe D:\my_packets.pcap

Copy link
Collaborator Author

@egecetin egecetin Nov 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@seladb The problem is arguments are also used by google benchmark. It supports some flags as default, for example --benchmark_filter or --benchmark_out_format etc. So --pcap-file actually not only parameter.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh ok, so we can keep it 👍

{
if (idx == argc - 1)
{
std::cerr << "Please provide a pcap file name after --pcap-file" << std::endl;
return 1;
}

pcapFileName = argv[idx + 1];
break;
}
}

if (pcapFileName.empty())
{
std::cerr << "Please provide a pcap file name using --pcap-file" << std::endl;
return 1;
}

benchmark::AddCustomContext("PcapPlusPlus version", pcpp::getPcapPlusPlusVersionFull());
benchmark::AddCustomContext("Build info", pcpp::getBuildDateTime());
benchmark::AddCustomContext("Git info", pcpp::getGitInfo());
benchmark::AddCustomContext("Pcap file", pcapFileName);

// Run the benchmarks
benchmark::RunSpecifiedBenchmarks();

return 0;
}
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ PcapPlusPlus enables decoding and forging capabilities for a large variety of ne
- [Vcpkg](#vcpkg)
- [Conan](#conan)
- [Build It Yourself](#build-it-yourself)
- [Verify your packages](#verify-your-packages)
- [Feature Overview](#feature-overview)
- [Getting Started](#getting-started)
- [API Documentation](#api-documentation)
Expand Down
Loading