Usage

This data is not intended to be a competition set, it is designed to be a resource for your own projects. To make it easy to use, we have made the test data publicly available. At the same time this means that anyone attempting to misuse the data could potentially overfit it with enough attempts. 

You can access the train/test data from the Releases section. The splits and target metrics are provided in separate JSON files, categorized into “ko” for gene essentiality and “drug” for drug sensitivity. 

For gene essentiality predictions, we have created the following splits based on DepMap data, (https://depmap.org/portal/,  https://www.nature.com/articles/ng.3984): 

• TRAIN: This is the training set. 
• RND: The random test set consists of cell lines and perturbations present in the training set, excluding the specific cell line-perturbation pairs. 
• CEX: The cell-exclusive test set enables prediction of the effects of training perturbations on new cell lines. 
• GEX: The gene expression test set allows prediction of the effects of new perturbations on the training cell lines. 
• AEX: The all-exclusive test set facilitates prediction of the effects of new perturbations on new cell lines. 

In our benchmark, we have utilized only a subset of gene essentiality data (a subset of genes) in order to keep the dataset more balanced. However, we provide the EXT_GEX and EXT_AEX splits to explore performance on the genome-wide DepMap data.  

Each JSON file contains the following information about the samples: 

• cell_line: The perturbed cell. 
• perturbation: The perturbed (CRISPR KOd) gene. 
• gene_effect: The target variable for predictions, representing the gene effect from DepMap. A gene effect of 0 indicates no fitness effect of the KO, while negative and positive gene effects represent negative and positive fitness effects, respectively. Typically, a gene effect of -0.5 indicates a “significant” viability reduction. 

Similarly, for drug sensitivity prediction, we have created the following splits based on GDSC2 data (https://www.cancerrxgene.org/,  https://www.cell.com/cell/fulltext/S0092-8674(16)30746-2): 

• TRAIN: This is the training set. 
• RND: The random test set consists of cell lines and perturbations present in the training set, excluding the specific cell line-perturbation pairs. 
• CEX: The cell-exclusive test set enables prediction of the effects of training perturbations on new cell lines. 
• DEX: The gene expression test set allows prediction of the effects of new perturbations (drugs) on the training cell lines. 
• AEX: The all-exclusive test set facilitates prediction of the effects of new perturbations on new cell lines. 

Each JSON file contains the following information about the samples: 

• cell_line: The perturbed cell. 
• perturbation: The PubChem ID of the drug. 

We have three different target metrics: 

• LN_IC50: The natural logarithm of the half inhibitory concentration (IC50). 
• z-score: The drug-wise normalized version of LN_IC50. 
• AUC: The area under the drug response curve. 

We did this splitting exercise 3 times in order to allow gauging the dependence of each model on a specific given test set. The file behind the “all train/test sets” button contains all 3 “split variants”. The primary test set is the test set of split 0.

Evaluation scripts: 

You can use our evaluation metrics and bias detector to correctly evaluate your predictions.

The downloadable zip file includes example files (example.json for targets and example.npy for predictions), precalculated cell and perturbation biases and two notebooks to run the evaluation scripts (eval_script.ipynb) and the bias detector (bias-detector.ipynb).  

FAQ

Q: I have a set of models trained separately for each drug / KO! Can I use this set?
A: Of course! Just leave the PEX and AEX splits out – the CEX results will still be valid.

Q: Should I run all split variants?
A: It makes sense to train/test all split variants once, so you can ensure you’re not overfitting any specific split. But generally, results on split 0 are fine on their own – the downloadable primary test set is actually just split 0’s test.

Q: Where are the rest of the genes?
A: We’ve only included genes for which we could generate node2vec embeddings from Omnipath data.

Roadmap

Planned for future versions:

• Harmonized drug and gene dependency train / test sets so a single model can use both without data leaks.
• RNAi tests
• Synthetic lethality and combination tests

License

The evaluation scripts and sample models are released under a CC-BY-SA 4.0 license. In a nutshell, feel free to use them in your projects – even commercial ones, as long as you don’t resell the datasets themselves.

If you publish results using or derived from the EFFECT benchmark, please cite the following article:
https://www.biorxiv.org/content/10.1101/2023.10.02.560281

The drug sensitivity dataset is based on GDSC data, so their license also applies (which are largely similar terms)

The gene dependency dataset is based on DepMap, which is published under CC-BY-4.0, don’t forget to attribute them as well!