The Innovative Methodologies and New Data for Predictive Oncology Model Evaluation (IMPROVE) project was created under the Cancer Moonshot℠ as part of the NCI-DOE Collaboration to help address the need in the AI modeling community for a robust, reproducible, fair, and transparent way to compare AI models at scale. 

IMPROVE is intended to help address challenges in data-driven modeling for predicting cancer drug response by establishing consistent methodologies for model comparison and evaluation, and by enhancing machine learning models through novel data integration. 

IMPROVE also generates new data to reduce bias and improve performance based on the deficiencies seen in the current AI models we are testing. These data will be available to the public to improve future drug response models. 
 

The IMPROVE project aims to construct a standardized comparison and evaluation framework for deep learning models of cancer drug response. This goal is facilitated by meticulously curated benchmark datasets that allow for the development of protocols to compare deep learning models for cancer drug response—and the identification of model attributes that enhance prediction performance, with the overarching goal of refining future models.

Additionally, IMPROVE is devising approaches for designing drug screening experiments to generate a specialized cancer drug response dataset explicitly intended for improving model performance in training and testing. 

In collaboration with our partners in the AI community, IMPROVE is researching and defining conventions for characterizing tumor-associated data features and molecular/drug representations to support the inclusion of multi-omics data, new drugs and therapies, and clinical data in future biomedically relevant AI models.
 

IMPROVE resources are available in the Dataset Finder, on GitHub, and through the NCI Predictive Oncology Model and Data Clearinghouse (MoDaC).

IMPROVE Curated Tumor Drug Response Model Collection

IMPROVE is jointly led by: 

• Rick Stevens, Argonne National Laboratory/University of Chicago
• Jeff Hildesheim, National Cancer Institute 
• Ryan Weil, Frederick National Laboratory for Cancer Research

Collaborative Core Modeling Group 

The Collaborative Core Modeling Group includes experts in cancer therapy and deep learning from:

• Texas Tech University, Lubbock Campus  
• Pacific Northwest National Laboratory
• Mayo Clinic, Rochester Minnesota

New Collaborators

As an open effort, IMPROVE welcomes additional extramural teams to contribute tumor drug response models, data sets and features to the IMPROVE framework. 

We also invite the broader community to use the state-of-the-art and formerly state-of-the-art models we have curated in your research. The IMPROVE team provides buildable code and Singularity containers to researchers who want to use the models curated by the IMPROVE team and collaborators.

To learn more about IMPROVE and how you can get involved, contact us at Improve-AI@nih.gov or computational-cancer-tech@nih.gov.