Black-Box Model Confidence Sets Using Cross-Validation with Gaussian Comparison

JING LEI – CARNEGIE MELLON UNIVERSITY

ABSTRACT

We derive high-dimensional Gaussian comparison results for the standard V-fold cross-validated risk estimates. Our result combines a recent stability-based argument for the low-dimensional central limit theorem of cross-validation with the high-dimensional Gaussian comparison framework for sums of independent random variables.  These results give new insights into the joint sampling distribution of cross-validated risks in the context of model comparison and tuning parameter selection, where the number of candidate models and tuning parameters can be larger than the fitting sample size. As a consequence, our results provide theoretical support for a recent methodological development that constructs model confidence sets using cross-validation.