ABSTRACT

A major asset of the cross-validation approach to smoothing-parameter choice is its utilitarian character.  However, the bandwidths produced by cross-valdiation are relatively noisy, and this difficulty impedes good performance.  The stochastic variability of cross-validation can be reduced significantly by using bootstrap aggregation, or bagging, a method proposed by Breiman (1999).  The technique is very simple to use, and enjoys the utilitarian character of cross-validation.  For instance, it can be applied in practically all of the many settings where cross-validation is employed for bandwidth choice with the aim of optimising an $L_2$ measure of performance.  Arbitrarily large reductions in bandwidth variability are theoretically possible, although in practice bagging would likely be used relatively modestly.  In particular, half-sample bagging can reduce bandwidth variability by approximately 50%.  We shall also discuss a bagging-based approach to constrained inference, for example to parameter estimation when it is known that the true value of the parameter satisfies an inequality constraint.