Resource utilization of Philosopher during protein inference

[magnusarntzen]

Hi,
I use Fragpipe for metaproteomics, i.e., with large multi-species databases typically having around 2M protein entries.

In general, this works fine and fast during an MSFragger search, but Philosopher is very slow even when I run this on a PC with 48 logical processors and 100 Gb RAM. Philosopher only uses a small fraction of this potential (it typically use only 9-13% of available CPU and about 7 Gb of RAM) in the step 'Processing protein inference'. On average, this step takes about 1.5-2h per raw file (timsTOF), and when having typically 50 raw files, this step is the most time consuming of the whole analysis.

I presume this is so slow due to my large databases, but would it be possible to make this step faster by accessing more CPUs through (better/more) parallellization?

Kind regards,
Magnus Arntzen
Norwegian University of Life Sciences

[magnusarntzen]

Btw, this is observed using Philosopher v4.8.1

[guoci]

Can you post the Philosopher logs? I would like to look at the timestamps.

[magnusarntzen]

Indeed, see attached. Here you will find that the step 'Processing protein inference' takes about 2 hours per raw file.
log_2023-11-09_23-55-48.txt

[guoci]

Can you upgrade to the latest version of Philosopher? That issue is fixed in 51cd256.
Upgrade Fragpipe and MSFragger as well, if they are not on the latest version.

[magnusarntzen]

Excellent, thank you. I will use newest version next time.