Performance

1. Chaput2016.csv^[1]

Data set: DUDE

Metric: BEDROC (alpha=80.5)

Software: GOLD，Glide, Surflex and FlexX

2. Cleves2020.csv^[2]

Data set: DUDE+

Metric: ROC AUC and ER 1%

Software: Dock, Glide and Surflex

3. Eberhardt2021.csv^[3]

Data set: DUDE

Metric: ROC AUC, BEDROC (alpha=20), EF at 1%, 5% and 10%

Software: AutoDock 1.2

4. Mysinger2012.csv^[4]

Data set: DUDE

Metric: ROC AUC, logAUC and EF at 1%

Software: DOCK

5. Wang2019.csv^[6]

Data set: DUDE

Metric: ROC AUC and BEDROC (alpha=80.5)

Software: GLIDE

6. Jiang2020.xlsx^[10]

Data set: DUDE

Metric: ROC AUC, BEDROC (alpha=20.0,80.5,321.0) and EF at 0.5%, 1%, 2%, 8%, 20%

Software: AutoPH4

7. Cleves2019.csv^[12]

Data set: DUDE

Metric: ROC AUC

Software: Surflex eSim(-pscreen), maximum AUC over the alternate methods

8. Koes2014.csv^[13]

Data set: DUDE

Metric: ROC AUC and BEDROC

Software: USR, ROCS and VAMS

9. Puertas-Martín2019.csv^[14]

Data set: DUDE

Metric: ROC AUC

Software: OptiPharm and WEGA

10. Shen2020.xlsx^[15]

Data set: DUDE, DEKOIS2.0, dataset III

Metric: ROC AUC, logAUC, BEDROC(alpha=80.5), EF at 0.1%，0.5%, 1%, 5%

Software: GLIDE, GOLD, LeDock

11. Jiang2021.xlsx^[16]

Data set: DUD-E, LIT-PCBA

Metric: ROC AUC, EF at 1%, 5%, 10%

Software: ROCS、Phase Shape、SHAFTS、WEGA、ShaEP、Shape-it、Align-it、LIGSIFT、LS-align

12. Jocelyn2021.xlsx^[17]

Data set: DUD-E, LIT-PCBA

Metric: ROC AUC, EF at 1%

Software: GNINA 1.0 with scoring function: Affinity,Pose,Affinity-dense,Pose-dense,Affinity-General,Pose-General,Vina,Vinardo,RFScore-VS,RFScore-4

Tools

1. metrics.py

metrics: ROC AUC, BEDAUC, enrichment_factor(EF) and logAUC

metrics.py can be available from oddt.

2. ROCKER^[9]

ROCKER is a visualization tool for ROC and semi-log ROC curve

ROCKER can be available from: http://www.medchem.fi/rocker

3. bootstrap_tldr.py^[11]

bootstrap_tldr.py is a visualization tool for ROC and semi-log ROC curve

bootstrap_tldr.py can be available from: https://dudez.docking.org

Reference

1. Chaput, L.; Martinez-Sanz, J.; Saettel, N.; Mouawad, L. Benchmark of Four Popular Virtual Screening Programs: Construction of the Active/Decoy Dataset Remains a Major Determinant of Measured Performance. J. Cheminform. 2016, 8 (1), 56. https://doi.org/10.1186/s13321-016-0167-x.
2. Cleves, A. E.; Jain, A. N. Structure- and Ligand-Based Virtual Screening on DUD-E + : Performance Dependence on Approximations to the Binding Pocket. J. Chem. Inf. Model. 2020, 60 (9), 4296–4310. https://doi.org/10.1021/acs.jcim.0c00115.

Download: https://www.jainlab.org/downloads/

3. Eberhardt, J.; Santos-Martins, D.; Tillack, A. F.; Forli, S. AutoDock Vina 1.2.0: New Docking Methods, Expanded Force Field, and Python Bindings. J. Chem. Inf. Model. 2021, acs.jcim.1c00203. https://doi.org/10.1021/acs.jcim.1c00203.
4. Mysinger, M. M.; Carchia, M.; Irwin, J. J.; Shoichet, B. K. Directory of Useful Decoys, Enhanced (DUD-E): Better Ligands and Decoys for Better Benchmarking. J. Med. Chem. 2012, 55 (14), 6582–6594. https://doi.org/10.1021/jm300687e.
5. Giangreco, I.; Mukhopadhyay, A.; C. Cole, J. Validation of a Field-Based Ligand Screener Using a Novel Benchmarking Data Set for Assessing 3D-Based Virtual Screening Methods. J. Chem. Inf. Model. 2021, 61 (12), 5841–5852. https://doi.org/10.1021/acs.jcim.1c00866.
6. Wang, D.; Cui, C.; Ding, X.; Xiong, Z.; Zheng, M.; Luo, X.; Jiang, H.; Chen, K. Improving the Virtual Screening Ability of Target-Specific Scoring Functions Using Deep Learning Methods. 2019, 10 (August), 1–11. https://doi.org/10.3389/fphar.2019.00924.
7. Imrie, F.; Bradley, A. R.; Van Der Schaar, M.; Deane, C. M. Protein Family-Specific Models Using Deep Neural Networks and Transfer Learning Improve Virtual Screening and Highlight the Need for More Data. J. Chem. Inf. Model. 2018, 58 (11), 2319–2330. https://doi.org/10.1021/acs.jcim.8b00350.
8. Scoring - Calculate rank statistics. http://www.rdkit.org/docs/source/rdkit.ML.Scoring.Scoring.html
9. Lätti, S.; Niinivehmas, S.; Pentikäinen, O. T. Rocker: Open Source, Easy-to-Use Tool for AUC and Enrichment Calculations and ROC Visualization. J. Cheminform. 2016, 8 (1), 45. https://doi.org/10.1186/s13321-016-0158-y.
10. Jiang, S.; Feher, M.; Williams, C.; Cole, B.; Shaw, D. E. AutoPH4: An Automated Method for Generating Pharmacophore Models from Protein Binding Pockets. J. Chem. Inf. Model. 2020, 60 (9), 4326–4338. https://doi.org/10.1021/acs.jcim.0c00121.
11. Stein, R. M.; Yang, Y.; Balius, T. E.; O’Meara, M. J.; Lyu, J.; Young, J.; Tang, K.; Shoichet, B. K.; Irwin, J. J. Property-Unmatched Decoys in Docking Benchmarks. J. Chem. Inf. Model. 2021, 61 (2), 699–714. https://doi.org/10.1021/acs.jcim.0c00598.
12. Cleves, A. E.; Johnson, S. R.; Jain, A. N. Electrostatic-Field and Surface-Shape Similarity for Virtual Screening and Pose Prediction. J. Comput. Aided. Mol. Des. 2019, 33 (10), 865–886. https://doi.org/10.1007/s10822-019-00236-6.
13. Koes, D. R.; Camacho, C. J. Shape-Based Virtual Screening with Volumetric Aligned Molecular Shapes. J. Comput. Chem. 2014, 35 (25), 1824–1834. https://doi.org/10.1002/jcc.23690.
14. Puertas-Martín, S.; Redondo, J. L.; Ortigosa, P. M.; Pérez-Sánchez, H. OptiPharm: An Evolutionary Algorithm to Compare Shape Similarity. Sci. Rep. 2019, 9 (1), 1–24. https://doi.org/10.1038/s41598-018-37908-6.
15. Shen, C.; Hu, Y.; Wang, Z.; Zhang, X.; Pang, J.; Wang, G.; Zhong, H.; Xu, L.; Cao, D.; Hou, T. Beware of the Generic Machine Learning-Based Scoring Functions in Structure-Based Virtual Screening. 2020, 00 (April), 1–22. https://doi.org/10.1093/bib/bbaa070.
16. Jiang, Z.; Xu, J.; Yan, A.; Wang, L. A Comprehensive Comparative Assessment of 3D Molecular Similarity Tools in Ligand-Based Virtual Screening. Brief. Bioinform. 2021, 22 (6), 1–17. https://doi.org/10.1093/bib/bbab231.
17. Sunseri, J.; Koes, D.R. Virtual Screening with Gnina 1.0. Molecules 2021, 26, 7369. https://doi.org/10.3390/molecules26237369