PMF Ground Classification with LAScatalog Parallelization

[myerbro3]

For those interested,

I recently asked a question about parallelizing the PMF Algorithm (#746). With @Jean-Romain 's advice, I successfully ran the following script (edited for brevity):

RStudio v.2023.12.1, R v.4.2.2, lidR v4.1.1, future v.1.33.1

library ("lidR")
library ("future")

set_lidr_threads(0)

ctg <- readLAScatalog(datapath)
las_check(ctg)
summary(ctg)
plot(ctg, mapview=TRUE)

ws <- seq(3, 6, 3)
th <- seq(0.1, 1.5, length.out = length(ws))

plan(multisession, workers = 6)
opt_output_files(ctg) <- paste0(datapath, "/{*}_class")
ctg_class <- classify_ground(ctg, algorithm = pmf(ws = ws, th = th))

Within the LAScatalog datapath was a single folder containing 63 .las files averaging roughly 2 GB each, 144 GB in total. These are exported .las tiles from a single point cloud generated in Agisoft Metashape Pro from RGB UAS imagery totaling roughly 5 billion points.

RGB (left) and classified (right). Brown = ground, green = not ground (i.e. vegetation)

Processing took 12 days on a Windows 10 machine with an i9-13900K CPU and 64 GB RAM. According to the resource monitor, RAM usage was consistently 63 GB / 64 GB. CPU usage varied, but was generally between 30% - 50%.

To say I'm pleased with the results is an understatement. The algorithm performed exceedingly well and the LAScatalog method of parallelization was reasonably fast, given the amount of data. I'm posting this because I've seen an increase in interest in working with .las data as it becomes more available and I hope it will of use to others in the future.

[Jean-Romain]

Thank you for your feedback. Here is my own feedback to this:

• You used 64 GB out of 64 GB which means that you actually used more than that and that you fall back to swap memory or whatever it is called on Windows. Maybe using only 4 cores would have been faster. Anyway, you won't run the computation again to figure out if my prediction is true.
• 12 days, 64 GB RAM for 144 GB of data is not reasonably fast. It is 1.7 MB/min/cores. It is waaay too slow and inefficient in my opinion. I hope we will be able to do better in my new lasR package.

Anyway, I'm glad you are happy with the result and I appreciate your feedback. I hope it will be useful for someone else.

[myerbro3]

I was suspicious of the system using 63 GB / 64 GB and thought that it falling back to swap memory could be the case. Fortunately, I was in a position that 12 days of processing wasn't an issue as I had/have several other projects to keep me busy.

As it turns out, I have another dataset to run (and likely more in the future) so I'll test 4 cores this time around - thanks!