Auto-Tune: Structural Optimization of Machine Learning Frameworks for Large Datasets

• Applicants: Professor Dr.-Ing. Thomas Brox; Professor Dr. Philipp Hennig; Professor Dr. Frank Hutter, Ph.D.
• Subject Area: Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing
• Term: from 2014 to 2021
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 260351709

We aim to automate the design of machine learning algorithms, in order to facilitate their use by non-experts and in autonomous systems. Although automated adaptation is a core idea of machine learning, most algorithms still require a choice of external design parameters by an expert, which limits their commercial success. Our approach formalizes the search for good algorithm configurations as an optimization problem over the combined space of different machine learning algorithms and develops novel Bayesian optimization algorithms for its solution.This projects follows in the steps of our recent Auto-WEKA framework, which demonstrated that modern Bayesian optimization methods can provide non-experts with an automated (albeit computationally very expensive) method to identify state-of-the-art instantiations of complex learning frameworks. The next step is to make this approach feasible under realistic budget constraints, which, for modern-day (big) datasets and learning frameworks (especially deep learning) often imply that we cannot evaluate more than a few full model instantiations.We take inspiration from the way human practitioners attack a new learning problem: compare the dataset to those previously encountered, and evaluate some promising methods on subsets of the data, before then only constructing one or a few models on the full dataset.We plan to integrate all these components into a probabilistic model, using our recent Bayesian optimization algorithm of Entropy Search on a design space covering these dimensions to automatically derive a strategy that resembles the design strategy of a human expert. We will validate our approaches by improving upon the existing Auto-WEKA system, and by implementing a first approach for learning an effective deep network for a new dataset at the push of a button.We propose two theoretical research projects: 1) General Probabilistic Models of Algorithm PerformanceThis thread involves finding structured models that capture the highly structured interdependences across the often very high-dimensional parameter spaces of machine learning algorithms.2) Budget-Thrifty Hyperparameter OptimizationIt is often feasible to run machine learning algorithms in a cost-reduced form, either by thinning the dataset or by "switching off" certain parts of an algorithm. We aim to encode this possibility in a cost-aware optimization algorithm, which should then be able to automatically control the progression from rough prototyping to fine tuning.These two theoretical advances will enable two applied projects, whichare: 1) Automatic Machine Learning2) Automated structural optimization in computer vision, especially deep learning

• DFG Programme: Priority Programmes
• Subproject of: SPP 1527: Autonomous Learning
• International Connection: Canada
• Major Instrumentation: GPU Cluster
• Instrumentation Group: 7030 Dedizierte, dezentrale Rechenanlagen, Prozeßrechner
• Participating Persons: Professor Dr. Holger Hoos; Professor Kevin Leyton-Brown, Ph.D.