Large-scale comparison of machine learning methods for drug target prediction on ChEMBL

Abstract:

Deep learning is currently the most successful machine learning technique in a wide range of application areas and has recently been applied successfully in drug discovery research to predict potential drug targets and to screen for active molecules.

However, due to
(1) the lack of large-scale studies,
(2) the compound series bias that is characteristic of drug discovery datasets and
(3) the hyperparameter selection bias that comes with the high number of potential deep learning architectures,
it remains unclear whether deep learning can indeed outperform existing computational methods in drug discovery tasks.

We therefore assessed the performance of several deep learning methods on a large-scale drug discovery dataset and compared the results with those of other machine learning and target prediction methods. To avoid potential biases from hyperparameter selection or compound series, we used a nested cluster-cross-validation strategy.

We found
(1) that deep learning methods significantly outperform all competing methods and
(2) that the predictive performance of deep learning is in many cases comparable to that of tests performed in wet labs (i.e., in vitro assays).

BibTeX:

@Article{bib:Mayr2018, author="Mayr, Andreas and Klambauer, G{\"u}nter and Unterthiner, Thomas and Steijaert, Marvin and Wegner, J{\"o}rg K. and Ceulemans, Hugo and Clevert, Djork-Arn{\'e} and Hochreiter, Sepp", title={{Large-scale comparison of machine learning methods for drug target prediction on ChEMBL}}, journal="Chem. Sci.", year="2018", volume="9", issue="24", pages="5441-5451" }

Publication
Electronic Supplementary Information
Technical Appendix and Partial Reanalysis

Supplementary_Table_S10.csv
Supplementary_Table_S11.csv
Supplementary_Table_S12.csv
Supplementary_Table_S13.csv
Supplementary_Table_S14.csv
Supplementary_Table_S15.csv

Benchmark Dataset

Compounds

ChEMBL20-SDF
ChEMBL20-MySQL

Static Features (MD5: b8d29262e7d1069052fdf2078e44e6d8)
Semisparse Features (MD5: 0458b111e5bf47b11d5fe3edb44fcd7f)
ECFP6 Features (MD5: f9bac94a3738f8e3c20ad4248832a767)
DFS8 Features (MD5: c26a01d4dbd9db7477f41b87ebfb3361)
Toxicophore Features (MD5: f293b77f62ab6a2dc910cd570b2b4c46)

ECFP4 Features (MD5: d09fca4b2a05a4aeed64375d49124f04, used for clustering)

Training

Sample Map: SampleIndex to ChEMBLId (MD5: df7e773de4ce0272d8ed8207c4ef6a6f, index starting at 0)
Folds (MD5: 7187541a8706bdfbc32d53c6f936266c, columns: SampleIndex, Fold | Folds: 0, 1, 2)

SQL Query Result (MD5: 77789163dd0e75a0fb66c3b703bc0570, Extraction of Data from ChEMBL database with SQL query)
Application of Protocol for Activity Call (MD5: 7b2becf6f52d9f9798631e6529378fc1, columns: Activity, SampleIndex, Target | double entries may exist)

Meaning of Activity:
Activity 0: garbage measurement
Activity 1: inactive
Activity 2: indeterminate
Activity 3: active
Activity 10: weak garbage measurement
Activity 11: weakly inactive
Activity 13: weakely active

Used Assays

tocompute.info (MD5: e5ee2c1c729f1af7190e1d220241a780)

License of our codes: LICENSE

Preprocessed data:

Preprocessed data in C/C++ format and Python format is available here.

C/C++ and Python Pipeline:

The individual steps are given in commdands.sh. More details on these steps are described below.

Code to produce Benchmark Dataset:

available at GitHub
and at git.bioinf.jku.at

ECFP4, ECFP6, DFS8

created with jCompoundMapper (source code [r55]) with 2 files exchanged:

ECFPFeature.java: insert new line between lines 58 and 59
substructureHash=false;
ExporterHelper.java: insert new line between lines 35 and 36
newHash=seed;
restart jCompoundMapper if it stops at certain compounds: chemblScript1.sh
callChemblScript1.sh

Static (dense) Features:

based on ChemoPy
some adaptions, see: git.bioinf.jku.at

Semisparse features

MACCS:
- computed by these scripts
PubChem:
- PubChem substructural keys
- empty compounds (compounds without any FPF) and skipped compounds substituted with median)
- restart script for fingerprint creator: chemblScript3.sh
- callChemblScript3.sh
SHED, CATS2D:
- jCompoundMapper (provided binary version)
- skipped features with zero standard deviation
- skipped compounds substituted with median
- restart jCompoundMapper if it stops at certain compounds: chemblScript2.sh
- callChemblScript2.sh
RDKit Topological Fingerprints (daylight):
- RDKit
- skipped compounds substituted with median
- creation script: createRDKit.py

Training Labels

Selection from the ChEMBL20-MySQL database: extractChEMBL.sql
Assignment of labels to measurements, removing measurements, etc.: createTrainTable.R

Clustering ChEMBL

all ChEMBL compounds from ChEMBL20-SDF clustered using single linkage clustering based on ECFP4 Jaccard distances (=1-Tanimoto similarity)
Computation of different single linkage clusterings: clusterMinFull.cpp
Assignment of clusters to folds: createClusters.R

Considered Targets (=Assays)

Assays used in publication: tocompute.info
Selection of considered targets: readDataset.py

Code to reproduce results:

available at GitHub

C/C++ Pipeline

SVMs, KNN, NB, SEA (Part1, Part2, Part3) were used in C/C++.
binary sparse and dense data format used:
- genDirStructure.cpp for producing directory structures
- convertFPFBinary.cpp and convertPropBinary.cpp for converting fpf files or dense matrices (csv) to binary format
configuration files for invoking programs: configuration files
optimized with OpenMP to allow high parallelism

Python Pipeline

For Deep Learning methods (FNNs, GC, Weave, SmilesLSTM) Python was used.
Estimation of GPU size: estGPUSize.py
Hyperparameter selection inner loop: step1.py
Performance estimation outer loop: step2.py
pickle data format used, input is C/C++ binary format:
- readDataset.py for conversion of C/C++ format to pickle and other preprocessing
- reduceDataset.py for taking only the compounds used for training or evaluation
- readMoleculesDeepchem.py and reduceMoleculesDeepchem.py for preparing compounds for GC and Weave
- readMoleculesLSTM.py, reduceMoleculesDeepchem.py and genAdditional.py for preparing compounds for SmilesLSTM

Data analysis

Data analysis scripts are available here.

aucPerformancesPerMethod.zip

Contact: Andreas Mayr (mayr@bioinf.jku.at)
Last Update: July, 5th, 2018