Feature Engineering¶

Exercise¶

Train linear regression models on the following data files:

Methods¶

Feature Engineering refers to all techniques that make your data easier for a model to understand. Here is an (incomplete) list of scaling/encoding techniques:

name	description
one-hot encoding	converts a category to binary columns
binning	bins scalar data to make it categorical
min-max scaling	normalize the data to the range 0.0 - 1.0
factorizing	converts categories to a single integers
target encoding	uses proportions of the target class
Log-Transform	converts exponential to linear

Feature Engineering means preprocessing your data before building a model. The goal of Feature Engineering is to derive a better representation of your input data.

The Transform Pattern¶

In scikit-learn, there is a generic transform() mechanism that applies to all feature engineering methods:

create a transformer object
call `.fit()` on the training data
call `.transform()` on the test/validation data

Warning

Never call `.fit()` on test/validation data. This ruins everything!

One-Hot Encoding¶

One-Hot encoding (dummy encoding) transforms a categorical variable into binary features.

Every distinct category becomes a binary column, so that every row has exactly one positive value.

category	apple	banana	cherry
apple	1	0	0
banana	0	1	0
banana	0	1	0
apple	1	0	0
cherry	0	0	1

from sklearn.preprocessing import OneHotEncoder

one_hot = OneHotEncoder(sparse=False, handle_unknown='ignore')

one_hot.fit(Xtrain[['column_name']])           # learn the categories
one_hot.transform(Xtrain[['column_name']])     # apply the transformation

Warning

The last column outputted from the OneHotEncoder is redundant, because the value of the last one-hot encoded column can logically be deduced from the others. If you take all columns, your data will be multicollinear, which creates trouble with some models.

You could remove the last column from the transformed data with the parameter drop='first'. However, in that case you can’t use handle_unknown='ignore' anymore.

Binning¶

Binning transforms a continuous, numeric variable to an ordinal, categorical variable, e.g. all ages –> [‘child’, ‘adolescent’, ‘middle-aged’, ‘elderly’].

Each “bin” is its own category.

The KBinsDiscretizer also one-hot-encodes the resulting categories straight away:

from sklearn.preprocessing import KBinsDiscretizer

binner = KBinsDiscretizer(n_bins=3, encode='onehot-dense', strategy='quantile')
binner.fit(Xtrain[['column_name']])
binner.transform(Xtrain[['column_name']])

The quantile strategy will place the same number of data points in each bin. An alternative, the uniform strategy will create bins of equal width.

Scaling¶

Scaling adjusts the range of the data or the mean and standard deviation to a defined range.

Many models require data to be scaled (e.g. SVM and Neural Networks). Logistic and Linear Regression are faster if the data is scaled.

Min-Max Scaling scales data to values in the range between 0 and 1.

\[X_i' = \frac{X_i - min(x)}{max(x) - min(x)}\]

You would use the MinMaxScaler class on the data after transforming the columns:

from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler()
scaler.fit(Xtrain[['column_name']])                    # learn the min and max
scaler.transform(Xtrain[['column_name']])              # apply the transformation

Alternatively, the StandardScaler scales to a normal distribution with mean 0 and standard deviation 1. The syntax is the same, only the import is different:

from sklearn.preprocessing import StandardScaler

Scaling is a special case of normalizing the data. There are many other normalization methods. We will see some of them later in the course.

Log-Transform to predict Count Data¶

Count data is often evaluated with the Root Mean Squared Log Error (RMSLE).

The purpose of this metric is to treat the error in relation to the absolute value. If the predicted value is 100, an error of 10 does not matter that much, but if the predicted value is only 1, the same error is huge. The logarithm fixes that.

To optimize your model against the RMSLE, you should take the logarithm of the target colum (y). Because 0 is a valid target value, use the log of \(y+1\) instead:

ylog = np.log1p(y)

Then train your model on the transformed column ylog. To bring back your log predictions to the original scale you have to apply the inverse transformation on the predictions:

ypred_log = m.predict(X)
ypred = np.exp(ypred_log)-1

You can then calculate the RMSLE score using sklearn:

from sklearn.metrics import mean_squared_log_error

np.sqrt(mean_squared_log_error(y, ypred))

Warning

The log transformation only makes sense if there is an absolute zero. This is the case with countable quantities (products sold, money, stocks, movies, electrons). It does not work with uncountable ones (Celsius temperature, timestamps).

pandas functions for Feature Engineering¶

Many pandas functions are very useful for feature engineering as well. They are a good starting point if you want to write your own FunctionTransformer.

function	description
`pandas.get_dummies()`	one-hot encoding
`pandas.cut()`	binning (bins of equal width)
`pandas.qcut()`	binning (quantile bins)
`pandas.Series.factorize()`	convert a category to an integer