Wild Refitting for Black box Prediction

MARTIN WAINWRIGHT – MASSACHUSETTS INSTITUTE OF TECHNOLOGY

ABSTRACT

Obtaining inferential guarantees on the performance of a prediction method is essential in practice. Modern predictive methods present barriers: (a) they are opaque, so that a statistician is limited to querying its predicted values only (with no further insight into the method’s properties); (b) a severely limited number of refits, due to computational expense; and (c) data can be heterogeneous. We describe a novel procedure for estimating the excess risk of any black box regression method that overcomes these challenges, and avoids any use of hold-out.  Inspired by the wild bootstrap, it uses Rademacher residual symmetrization to construct a synthetic dataset for refitting. Unlike the bootstrap, it requires only a single refit, and we give non-asymptotic guarantees on the risk estimate. We illustrate its behavior for non-rigid structure-from-motion, and plug-and-play image denoising using deep net priors.

BIOGRAPHY: Martin Wainwright is the Ford Professor in Electrical Engineering and Computer Science and Mathematics at MIT, and affiliated with the Laboratory for Information and Decision Systems and Statistics and Data Science Center. He is broadly interested in statistics, machine learning, information theory and algorithms. He has received a number of awards and recognition including the COPSS Presidents’ Award from the Joint Statistical Societies, a Section Lecturer with the International Congress of Mathematicians, a John Simon Guggenheim Fellowship, an Alfred P. Sloan Foundation Fellowship, and the Blackwell Lectureship and Award from the Institute of Mathematical Statistics.