CANDLE/Supervisor: A Workflow Framework for Machine Learning Applied to Cancer Research

Background

Current multi-petaflop supercomputers are powerful systems, but present challenges when faced with problems requiring large machine learning workflows. Complex algorithms running at system scale, often with different patterns that require disparate software packages and complex data flows cause difficulties in assembling and managing large experiments on these machines.

Results

This paper presents a workflow system that makes progress on scaling machine learning ensembles, specifically in this first release, ensembles of deep neural networks that address problems in cancer research across the atomistic, molecular and population scales. The initial release of the application framework that the authors call CANDLE/Supervisor addresses the problem of hyper-parameter exploration of deep neural networks.

Conclusions

Initial results demonstrating CANDLE on Department of Energy systems at Oak Ridge National Laboratory, Argonne National Laboratory, and National Energy Research Scientific Computing Center (Titan, Theta and Cori, respectively) demonstrate both scaling and multi-platform execution.