Does data splitting improve prediction?

By Julian J. Faraway. 2013

51 citations recorded by the source corpus.

Data splitting divides data into two parts. One part is reserved for model selection. In some applications, the second part is used for model validation but we use this part for estimating the parameters of the chosen model. We focus on the problem of constructing reliable predictive distributions for future observed values. We judge the predictive performance using log scoring. We compare the full data strategy with the data splitting strategy for prediction. We show how the full data score can be decomposed into model selection, parameter estimation and data reuse costs. Data splitting is preferred when data reuse costs are high. We investigate the relative performance of the strategies in four simulation scenarios. We introduce a hybrid estimator that uses one part for model selection but both parts for estimation. We argue that a split data analysis is prefered to a full data analysis for prediction with some exceptions.

Subfields: Management Science and Operations Research, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Management Science and Operations Research

In Statistics and Probability

Citations in the corpus

Interpretable classification models for recidivism prediction (2015, background): “rent points on the ROC curve on most problems. SVM RBF usually achieved the highest AUC on most problems (e.g., arrest, 12There is little guidance on how much data to allocate for a test set (see e.g Faraway 2014, for a discussion.) 16 Method Acronym Software Free Parameters and Settings CART Decisi”
Decision support from financial disclosures with deep neural networks and transfer learning (2017, methodology): “4 We thus follow [37, 38] and split the training and test sets in chronological order.”
Using Regression Kernels to Forecast A Failure to Appear in Court (2014, background): “To begin, there is currently no clear statistical guidance on the relative sizes of split samples, even when there are only two (Faraway, 2014).”
Determinants of Default in P2P Lending (2015, background): “This practice has long been recognized as generating over-optimistic inference but practitioners frequently do little to address it [75].”