Cross-Validation¶

Concepts¶

Function
`sklearn.model_selection.cross_validate()`	k-fold cross-validation
`sklearn.model_selection.ShuffleSplit`	Shuffle-split cross-validation
`sklearn.model_selection.StratifiedShuffleSplit`	stratified Shuffle-split CV
`sklearn.utils.resample()`	Select sample from dataset

K-fold Cross-Validation¶

K-fold Cross-Validation divides the dataset into k portions called folds. Each fold serves as a test dataset once, and the rest as a training set. This results in a table of k scores for the training and test fold that allow to evaluate the robustness of the model.

With cross validation we can use all our training data for fitting the model as we don’t need to apply an additional train-validation split. On the other side, cross validation is computationally costly, especially for large datasets.

from sklearn.model_selection import cross_validate

model = LinearRegression()
cv = cross_validate(model, X_train_trans, y_train,
                    cv=5,
                    scoring='r2',
                    return_train_score=True
)
# convert the dictionary of lists into a DataFrame
cv = pd.DataFrame(cv)
print(cv)

Try 5-fold cross-validation with scoring=’neg_mean_squared_log_error’ on the Bike dataset.

How would you interpret the result?

Shuffle-Split¶

Shuffle-Split is a more flexible alternative, especially with large datasets. It allows you to select random samples with a specific size. The stratified variant takes care to select representative features.

Compare k-fold CV and Shuffle-split CV on the Titanic dataset. What impact does the size of the dataset have? What impact does stratification have?

Cross-Validation and Time Series Data¶

In time series, we try to predict the future from the past. Because of that, the test data always has to be after the training portion. In this case, cross-validation should look like this:

There is no scikit function for that, you need to slice the data folds yourself.

Bootstrapping¶

If you want to calculate a confidence score for your model (a probability range of the model quality), you have 3 options:

For some models (e.g. Linear Regression), a confidence interval can be calculated directly using the t distribution
Other models (e.g. Logistic Regression, SVMs) provide a method to calculate probabilities for predictions.
But for many models, neither of this works (in particular, Random Forests and Neural Networks). With these, we have to use bootstrapping.

The scores should be more or less normally distributed for the bootstrapping to be reliable.

Of course, retraining your model 1000 times is costly.

from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.utils import resample

X, y = load_iris(True)
eighty_percent_split = int(len(X)*0.8)

m = LogisticRegression()

boots = []
for i in range(1000):
    # Resample the original data to create a "new" dataset
    Xb, yb = resample(X, y)
    
    # Split the data into training and validation set
    Xb_train = Xb[:eighty_percent_split]
    yb_train = yb[:eighty_percent_split]
    Xb_validation = Xb[eighty_percent_split:]
    yb_validation = yb[eighty_percent_split:]

    # Fit the model and calculate the validation score
    m.fit(Xb_train, yb_train)
    score = m.score(Xb_validation, yb_validation)
    boots.append(score)
    print(i, score)

# get percentiles for 90% confidence
boots.sort()
ci80 = boots[100:-100]
print(f"80% confidence interval: {ci80[0]:5.2} -{ci80[-1]:5.2}")
ci90 = boots[50:-50]
print(f"90% confidence interval: {ci90[0]:5.2} -{ci90[-1]:5.2}")
ci95 = boots[25:-25]
print(f"95% confidence interval: {ci95[0]:5.2} -{ci95[-1]:5.2}")
ci99 = boots[5:-5]
print(f"99% confidence interval: {ci99[0]:5.2} -{ci99[-1]:5.2}")

What is a sampling bias?

What is the purpose of a validation set?

Why is a single validation set not sufficient?

What does it mean if your cross-validation scores fluctuate a lot?

What is stratification?