./configure errors

[strongwazz]

seems like something is missing after running ./configure

git clone https://github.com/xdp-project/bpf-examples.git bpfex2
cd bpfex2/
./configure
clang: 13.0.0-2
libbpf support: submodule
Submodule 'lib/libbpf' (https://github.com/xdp-project/libbpf.git) registered for path 'lib/libbpf'
Submodule 'lib/xdp-tools' (https://github.com/xdp-project/xdp-tools) registered for path 'lib/xdp-tools'
Cloning into '/bpfex2/lib/libbpf'...
Cloning into '/bpfex2/lib/xdp-tools'...
Submodule path 'lib/libbpf': checked out '2cd2d03f63242c048a896179398c68d2dbefe3d6'
Submodule path 'lib/xdp-tools': checked out 'fe029c770733930a92014687d386c0b34ba13627'
ELF support: yes
zlib support: yes
libxdp support: submodule
Configuring libxdp to use our libbpf submodule
Found clang binary 'clang' with version 13 (from 'Ubuntu clang version 13.0.0-2')
libbpf support: custom v0.7.0
  perf_buffer__consume support: cp: cannot stat '/bpfex2/lib/libbpf/src/{bpf,btf,libbpf*}.h': No such file or directory
  btf__load_from_kernel_by_id support: cp: cannot stat '/bpfex2/lib/libbpf/src/{bpf,btf,libbpf*}.h': No such file or directory
  btf__type_cnt support: cp: cannot stat '/bpfex2/lib/libbpf/src/{bpf,btf,libbpf*}.h': No such file or directory
  bpf_object__next_map support: cp: cannot stat '/bpfex2/lib/libbpf/src/{bpf,btf,libbpf*}.h': No such file or directory
  bpf_object__next_program support: cp: cannot stat '/bpfex2/lib/libbpf/src/{bpf,btf,libbpf*}.h': No such file or directory
  bpf_program__insn_cnt support: cp: cannot stat '/bpfex2/lib/libbpf/src/{bpf,btf,libbpf*}.h': No such file or directory
  bpf_map_create support: cp: cannot stat '/bpfex2/lib/libbpf/src/{bpf,btf,libbpf*}.h': No such file or directory
  perf_buffer__new_raw support: cp: cannot stat '/bpfex2/lib/libbpf/src/{bpf,btf,libbpf*}.h': No such file or directory
zlib support: yes
ELF support: yes
pcap support: yes
secure_getenv support: yes

Everything is there in ./lib/libbpf/src/, not sure if this is permissions or something else?

If I then build pping it throws plenty of errors, but guessing its related to the above

platform is Ubuntu 21.10 on 5.13

I really appreciate any help you can provide

[simosund]

Hmm, I'm also running into this issue now, even if I go back to commits that I know have worked well in the past, so something strange is going on.

The issue seems to be with the configure script for libxdp (in ./lib/xdp-tools/configure). The {bpf,btf,libbpf*}.h part doesn't seem to be expanded correctly by sh. As a temporary workaround, try changing the shebang of ./lib/xdp-tools/configure from #!/bin/sh to #!/bin/bash, that seemed to work for me at least.

@tohojo, could you have a look at this and see if we can get a better long term fix?

[simosund]

And a small heads-up, while ePPing should work on 5.13 (just tested it myself), it will probably not work in XDP mode (which is the default), see issue#49. So you will probably have to run it in tc mode (by using arguments -I tc or --ingress-hook tc). Alternatively, if you really want to run it in XDP mode for some reason you could either update to a kernel which has this patch or check out commit #404a70c or earlier.

[strongwazz]

Hi @simosund ; thanks for the reply

Moving configure to run through bash hasn't made a change to the parameter expansion. I wonder if this is a bash version thing? I'm running 5.1.8, specifically

Ignore the previous, bit hasty this morning - bash does the trick to get configure ran without errors.

You're right about XDP mode, here's what I get if it's of use

Starting ePPing in json mode tracking TCP on ens1
libbpf: elf: skipping unrecognized data section(7) xdp_metadata
libbpf: elf: skipping unrecognized data section(7) xdp_metadata
libbpf: prog 'pping_xdp_ingress': BPF program load failed: Invalid argument
libbpf: prog 'pping_xdp_ingress': -- BEGIN PROG LOAD LOG --
Func#1 is safe for any args that match its prototype
Validating pping_xdp_ingress() func#0...
; int pping_xdp_ingress(struct xdp_md *ctx)
0: (bf) r6 = r1
1: (b7) r7 = 0
; struct packet_info p_info = { 0 };
2: (7b) *(u64 *)(r10 -8) = r7
3: (7b) *(u64 *)(r10 -16) = r7
4: (7b) *(u64 *)(r10 -24) = r7
5: (7b) *(u64 *)(r10 -32) = r7
6: (7b) *(u64 *)(r10 -40) = r7
7: (7b) *(u64 *)(r10 -48) = r7
8: (7b) *(u64 *)(r10 -56) = r7
9: (7b) *(u64 *)(r10 -64) = r7
10: (7b) *(u64 *)(r10 -72) = r7
11: (7b) *(u64 *)(r10 -80) = r7
12: (7b) *(u64 *)(r10 -88) = r7
13: (7b) *(u64 *)(r10 -96) = r7
14: (7b) *(u64 *)(r10 -104) = r7
15: (7b) *(u64 *)(r10 -112) = r7
16: (7b) *(u64 *)(r10 -120) = r7
17: (7b) *(u64 *)(r10 -128) = r7
18: (bf) r2 = r10
; 
19: (07) r2 += -128
; if (parse_packet_identifer_xdp(ctx, &p_info) < 0)
20: (85) call pc+13
R1 type=ctx expected=fp
Caller passes invalid args into func#1
processed 216304 insns (limit 1000000) max_states_per_insn 40 total_states 13719 peak_states 879 mark_read 34
-- END PROG LOAD LOG --
libbpf: failed to load program 'pping_xdp_ingress'
libbpf: failed to load object 'pping_kern.o'
Failed attaching ingress BPF program on interface ens1: Invalid argument
Failed loading and attaching BPF programs in pping_kern.o

But it does run with -I tc

#: ./pping -i ens1 -I tc -F json
Starting ePPing in json mode tracking TCP on ens1
libbpf: Kernel error message: Exclusivity flag on, cannot modify
libbpf: Kernel error message: Exclusivity flag on, cannot modify
packet_ts: cycle: 1, entries: 7, time: 23704, timeout: 0, tot timeout: 0, selfdel: 0, tot selfdel: 0
flow_state: cycle: 1, entries: 7, time: 46993, timeout: 0, tot timeout: 0, selfdel: 0, tot selfdel: 0
packet_ts: cycle: 2, entries: 3654, time: 1129987, timeout: 0, tot timeout: 0, selfdel: 0, tot selfdel: 0
flow_state: cycle: 2, entries: 2109, time: 304090, timeout: 0, tot timeout: 0, selfdel: 242, tot selfdel: 242
packet_ts: cycle: 3, entries: 7096, time: 2285823, timeout: 0, tot timeout: 0, selfdel: 0, tot selfdel: 0
flow_state: cycle: 3, entries: 3511, time: 549371, timeout: 0, tot timeout: 0, selfdel: 256, tot selfdel: 498
packet_ts: cycle: 4, entries: 10917, time: 3448209, timeout: 0, tot timeout: 0, selfdel: 0, tot selfdel: 0
flow_state: cycle: 4, entries: 4845, time: 748436, timeout: 0, tot timeout: 0, selfdel: 316, tot selfdel: 814
packet_ts: cycle: 5, entries: 14879, time: 4639299, timeout: 0, tot timeout: 0, selfdel: 0, tot selfdel: 0
flow_state: cycle: 5, entries: 5930, time: 890944, timeout: 0, tot timeout: 0, selfdel: 355, tot selfdel: 1169
10:28:28.763995812 Warning: Unable to create timestamp entry for flow 163.47.185.169:22+103.226.55.38:13424
packet_ts: cycle: 6, entries: 16384, time: 5099368, timeout: 0, tot timeout: 0, selfdel: 0, tot selfdel: 0
flow_state: cycle: 6, entries: 6914, time: 1100977, timeout: 0, tot timeout: 0, selfdel: 349, tot selfdel: 1518
10:28:29.764420757 Warning: Unable to create timestamp entry for flow 180.222.66.11:53536+31.13.78.17:443
packet_ts: cycle: 7, entries: 16384, time: 5236161, timeout: 0, tot timeout: 0, selfdel: 0, tot selfdel: 0
flow_state: cycle: 7, entries: 7701, time: 1285070, timeout: 0, tot timeout: 0, selfdel: 336, tot selfdel: 1854
10:28:30.764693259 Warning: Unable to create timestamp entry for flow 163.47.186.60:50948+77.234.41.215:80

Some warnings in the mix

With -F json, where would I be capturing that output?

tia :) :)

[simosund]

Hi, unfortunately I suspect we have quite some timezone differences (CEST for me) so my replies may not be that timely.

At this stage all output formats (including -F json) should just be dumped to stdout for now. Based on the debug output (all the lines starting with packet_ts: and flow_state) you definitely have a lot of traffic that ePPing should be able to get some RTTs for (as it's saving timestamps), but for some reason it doesn't seem to be getting any matches (the tot selfdel field for packet_ts stays at 0, which tells me it was never able to match a reply to one of its saved timestamps). Does the interface you're capturing on see traffic in both directions? If ePPing can't see replies it can't calculate RTTs as it needs to match packets going in one direction of the flow with ACKs in the reverse direction. If you run Kathie's pping in the same setup do you get any output? Because if you get output with Kathie's pping then you should be getting output with ePPing (they rely on the same principles).

[simosund]

As for the initial warnings:
libbpf: Kernel error message: Exclusivity flag on, cannot modify

They occur because the BPF tc programs that ePPing try to set up a clsact qdisc on the interface, but one already exists. This is not really a problem because then I just reuse the existing clsact qdisc on the interface then, but libbpf will throw out a warning because I tried creating one.

Another little heads-up though, the semantics for how ePPing cleans up that qdisc is probably not optimal at the moment. By default, ePPing does not even try to delete the clsact on shutdown to avoid risking tearing it out from underneath any other application that might be using the qdisc (for example any other tc BPF programs you have have running on the interface). So if you want to get rid of the clsact qdisc you have to manually delete it (tc qdisc del dev <iface> clsact). You can also pass the -f or --force argument ePPing in which case it will try to delete the qdisc on shutdown, but only if it was the process that created the qdisc in the first place (and as in this case you probably already have the qdisc there since a previous run of ePPing, it will still leave it around as the new instance of ePPing wasn't the one that created it, so you still have to delete it manually first). However, if you run any other programs that will use the clsact after ePPing created it, you then risk ePPing destroying the clsact for these other programs on shutdown with --force, so you should only use it if you're reasonably sure you're not running any other tc BPF programs on the interface (or anything else that uses the clsact qdisc, although as far as I know it's mainly used for tc BPF programs).

Hopefully libbpf will add some API that makes it convenient to simply remove the clsact qdisc if no other programs are attached to it (they did in fact have that at some point, but removed due to risks with race conditions).

[simosund]

As for the warnings of the type Warning: Unable to create timestamp entry for flow XXXX they simply mean that the timestamp map (used to store timestamps to later match against replies and calculate RTTs with) is full. The default map size is only 2^14 = 16384 entries, which is very conservative. I usually increase it to 2^16 = 65536 entries in my own tests, which eats up ~3.5 MB (each entry is 56 bytes). This should probably be configurable through a parameter, but right now it requires changing the source (specifically this line in pping_kern.c).

In this case, since it seems like it's not able to match any replies to the saved timestamps they will sit around and occupy space in the map for 10 seconds before they're timed out an removed by a periodical cleanup process (although this means that ePPing will typically not be able to detect RTTs above 10s, but I've assumed these are very rare). In a scenario where you actually see replies the map should be used a bit more efficiently, partly because the entries are automatically deleted after they've been matched, and partly because the period for the timing out entries that have not been matched is decreased to a multiple of the flows RTT, which should in most cases be much less than the 10 second default. While these values should probably also be configurable from the command line, for now they're simply a bunch of defines in pping_kern.c, have a look at these lines if you want to change them.