Spurious low performance when purging an old POSIX file system in esdm_mkfs()

[nhuebbe]

Every once in a while, when testing with a lot of fragments (a-many-fragments-stress-test -l 500 which creates 125250 fragments), a subsequent esdm_mkfs() call experiences extremely low performance. This may happen either during a repetition of the same test, or in another test that purges the file system created by the above test. The execution time may well exceed 10min on a task that usually takes only about 20s to 30s, which is roughly the time that it took to write the data.

When the low performance happens, the hard disk is working flat out, and the program execution is invariably to be found at

esdm_mkfs(), esdm/src/esdm.c:133
mkfs(), esdm/src/backends-data/posix/posix.c:142
posix_recursive_remove, esdm/src/utils/auxiliary.c:114, several frames
posix_recursive_remove, esdm/src/utils/auxiliary.c:105

The last line is a stat() call. The process is not fully hung up, it does still make progress, just way too slow.

I have no idea what causes this, other than that it appears to have a connection to the huge directory that the POSIX backend creates. Usually performance is satisfactory, but sometimes it takes almost forever. In these cases, it's usually quicker to just shoot the process and do the cleanup manually with rm -r.